t81_558_class07_cnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 7: Convolutional Neural Networks.**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), School of Engineering and Applied Science, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module Video Material\n",
    "\n",
    "Main video lecture:\n",
    "\n",
    "* [Part 7.1: Data Sets for Computer Vision](https://www.youtube.com/watch?v=u8xn393mDPM&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=21)\n",
    "* [Part 7.2: Convolution Neural Network](https://www.youtube.com/watch?v=cf6FDLFNWEk&index=22&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)\n",
    "* [Part 7.3: Using Convolutional Neural Networks (CNNs) in Keras and TensorFlo](https://www.youtube.com/watch?v=LSSH_NdXwhU&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN&index=23)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Helpful Functions\n",
    "\n",
    "You will see these at the top of every module.  These are simply a set of reusable functions that we will make use of.  Each of them will be explained as the semester progresses.  They are explained in greater detail as the course progresses.  Class 4 contains a complete overview of these functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import base64\n",
    "import os\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import requests\n",
    "from sklearn import preprocessing\n",
    "\n",
    "\n",
    "# Encode text values to dummy variables(i.e. [1,0,0],[0,1,0],[0,0,1] for red,green,blue)\n",
    "def encode_text_dummy(df, name):\n",
    "    dummies = pd.get_dummies(df[name])\n",
    "    for x in dummies.columns:\n",
    "        dummy_name = f\"{name}-{x}\"\n",
    "        df[dummy_name] = dummies[x]\n",
    "    df.drop(name, axis=1, inplace=True)\n",
    "\n",
    "\n",
    "# Encode text values to a single dummy variable.  The new columns (which do not replace the old) will have a 1\n",
    "# at every location where the original column (name) matches each of the target_values.  One column is added for\n",
    "# each target value.\n",
    "def encode_text_single_dummy(df, name, target_values):\n",
    "    for tv in target_values:\n",
    "        l = list(df[name].astype(str))\n",
    "        l = [1 if str(x) == str(tv) else 0 for x in l]\n",
    "        name2 = f\"{name}-{tv}\"\n",
    "        df[name2] = l\n",
    "\n",
    "\n",
    "# Encode text values to indexes(i.e. [1],[2],[3] for red,green,blue).\n",
    "def encode_text_index(df, name):\n",
    "    le = preprocessing.LabelEncoder()\n",
    "    df[name] = le.fit_transform(df[name])\n",
    "    return le.classes_\n",
    "\n",
    "\n",
    "# Encode a numeric column as zscores\n",
    "def encode_numeric_zscore(df, name, mean=None, sd=None):\n",
    "    if mean is None:\n",
    "        mean = df[name].mean()\n",
    "\n",
    "    if sd is None:\n",
    "        sd = df[name].std()\n",
    "\n",
    "    df[name] = (df[name] - mean) / sd\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the median\n",
    "def missing_median(df, name):\n",
    "    med = df[name].median()\n",
    "    df[name] = df[name].fillna(med)\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the default\n",
    "def missing_default(df, name, default_value):\n",
    "    df[name] = df[name].fillna(default_value)\n",
    "\n",
    "\n",
    "# Convert a Pandas dataframe to the x,y inputs that TensorFlow needs\n",
    "def to_xy(df, target):\n",
    "    result = []\n",
    "    for x in df.columns:\n",
    "        if x != target:\n",
    "            result.append(x)\n",
    "    # find out the type of the target column.  Is it really this hard? :(\n",
    "    target_type = df[target].dtypes\n",
    "    target_type = target_type[0] if hasattr(\n",
    "        target_type, '__iter__') else target_type\n",
    "    # Encode to int for classification, float otherwise. TensorFlow likes 32 bits.\n",
    "    if target_type in (np.int64, np.int32):\n",
    "        # Classification\n",
    "        dummies = pd.get_dummies(df[target])\n",
    "        return df[result].values.astype(np.float32), dummies.values.astype(np.float32)\n",
    "    # Regression\n",
    "    return df[result].values.astype(np.float32), df[[target]].values.astype(np.float32)\n",
    "\n",
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return f\"{h}:{m:>02}:{s:>05.2f}\"\n",
    "\n",
    "\n",
    "# Regression chart.\n",
    "def chart_regression(pred, y, sort=True):\n",
    "    t = pd.DataFrame({'pred': pred, 'y': y.flatten()})\n",
    "    if sort:\n",
    "        t.sort_values(by=['y'], inplace=True)\n",
    "    plt.plot(t['y'].tolist(), label='expected')\n",
    "    plt.plot(t['pred'].tolist(), label='prediction')\n",
    "    plt.ylabel('output')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "# Remove all rows where the specified column is +/- sd standard deviations\n",
    "def remove_outliers(df, name, sd):\n",
    "    drop_rows = df.index[(np.abs(df[name] - df[name].mean())\n",
    "                          >= (sd * df[name].std()))]\n",
    "    df.drop(drop_rows, axis=0, inplace=True)\n",
    "\n",
    "\n",
    "# Encode a column to a range between normalized_low and normalized_high.\n",
    "def encode_numeric_range(df, name, normalized_low=-1, normalized_high=1,\n",
    "                         data_low=None, data_high=None):\n",
    "    if data_low is None:\n",
    "        data_low = min(df[name])\n",
    "        data_high = max(df[name])\n",
    "\n",
    "    df[name] = ((df[name] - data_low) / (data_high - data_low)) \\\n",
    "        * (normalized_high - normalized_low) + normalized_low\n",
    "\n",
    "\n",
    "# This function submits an assignment.  You can submit an assignment as much as you like, only the final\n",
    "# submission counts.  The paramaters are as follows:\n",
    "# data - Pandas dataframe output.\n",
    "# key - Your student key that was emailed to you.\n",
    "# no - The assignment class number, should be 1 through 1.\n",
    "# source_file - The full path to your Python or IPYNB file.  This must have \"_class1\" as part of its name.  \n",
    "# .             The number must match your assignment number.  For example \"_class2\" for class assignment #2.\n",
    "def submit(data,key,no,source_file=None):\n",
    "    if source_file is None and '__file__' not in globals(): raise Exception('Must specify a filename when a Jupyter notebook.')\n",
    "    if source_file is None: source_file = __file__\n",
    "    suffix = '_class{}'.format(no)\n",
    "    if suffix not in source_file: raise Exception('{} must be part of the filename.'.format(suffix))\n",
    "    with open(source_file, \"rb\") as image_file:\n",
    "        encoded_python = base64.b64encode(image_file.read()).decode('ascii')\n",
    "    ext = os.path.splitext(source_file)[-1].lower()\n",
    "    if ext not in ['.ipynb','.py']: raise Exception(\"Source file is {} must be .py or .ipynb\".format(ext))\n",
    "    r = requests.post(\"https://api.heatonresearch.com/assignment-submit\",\n",
    "        headers={'x-api-key':key}, json={'csv':base64.b64encode(data.to_csv(index=False).encode('ascii')).decode(\"ascii\"),\n",
    "        'assignment': no, 'ext':ext, 'py':encoded_python})\n",
    "    if r.status_code == 200:\n",
    "        print(\"Success: {}\".format(r.text))\n",
    "    else: print(\"Failure: {}\".format(r.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Computer Vision\n",
    "\n",
    "This class will focus on computer vision.  There are some important differences and similarities with previous neural networks.\n",
    "\n",
    "* We will usually use classification, though regression is still an option.\n",
    "* The input to the neural network is now 3D (height, width, color)\n",
    "* Data are not transformed, no zscores or dummy variables.\n",
    "* Processing time is much longer.\n",
    "* We now have different layer times: dense layers (just like before), convolution layers and max pooling layers.\n",
    "* Data will no longer arrive as CSV files. TensorFlow provides some utilities for going directly from image to the input for a neural network.\n",
    "\n",
    "\n",
    "# Computer Vision Data Sets\n",
    "\n",
    "There are many data sets for computer vision.  Two of the most popular are the MNIST digits data set and the CIFAR image data sets.\n",
    "\n",
    "## MNIST Digits Data Set\n",
    "\n",
    "The [MNIST Digits Data Set](http://yann.lecun.com/exdb/mnist/) is very popular in the neural network research community.  A sample of it can be seen here:\n",
    "\n",
    "![MNIST Data Set](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_mnist.png \"MNIST Data Set\")\n",
    "\n",
    "This data set was generated from scanned forms.\n",
    "\n",
    "![Exam](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_exam.png \"Exam\")\n",
    "\n",
    "\n",
    "## CIFAR Data Set\n",
    "\n",
    "The [CIFAR-10 and CIFAR-100](https://www.cs.toronto.edu/~kriz/cifar.html) datasets are also frequently used by the neural network research community.\n",
    "\n",
    "![cifar-10](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_cifar.png \"cifar-10\")\n",
    "\n",
    "The CIFAR-10 data set contains low-rez images that are divided into 10 classes.  The CIFAR-100 data set contains 100 classes in a hierarchy. \n",
    "\n",
    "# Other Resources\n",
    "\n",
    "* [Imagenet:Large Scale Visual Recognition Challenge 2014](http://image-net.org/challenges/LSVRC/2014/index)\n",
    "* [Andrej Karpathy](http://cs.stanford.edu/people/karpathy/) - PhD student/instructor at Stanford.\n",
    "    * [CS231n Convolutional Neural Networks for Visual Recognition](http://cs231n.stanford.edu/) - Stanford course on computer vision/CNN's.\n",
    "        * [CS231n - GitHub](http://cs231n.github.io/)\n",
    "    * [ConvNetJS](http://cs.stanford.edu/people/karpathy/convnetjs/) - Javascript library for deep learning.\n",
    " \n",
    "\n",
    "# Convolutional Neural Networks (CNNs)\n",
    "\n",
    "The convolutional neural network (CNN) is a neural network technology that has profoundly impacted the area of computer vision (CV). Fukushima (1980) introduced the original concept of a convolutional neural network, and LeCun, Bottou, Bengio & Haffner (1998) greatly improved this work. From this research, Yan LeCun introduced the famous LeNet-5 neural network architecture. This class follows the LeNet-5 style of convolutional neural network.\n",
    "\n",
    "**A LeNET-5 Network (LeCun, 1998)**\n",
    "![LENET5](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_lenet5.png \"LENET5\")\n",
    "\n",
    "So far we have only seen one layer type (dense layers).  By the end of this course we will have seen:\n",
    "\n",
    "* **Dense Layers** - Fully connected layers.  (introduced previously)\n",
    "* **Convolution Layers** - Used to scan across images. (introduced this class)\n",
    "* **Max Pooling Layers** - Used to downsample images. (introduced this class)\n",
    "* **Dropout Layer** - Used to add regularization. (introduced next class)\n",
    "\n",
    "## Convolution Layers\n",
    "\n",
    "The first layer that we will examine is the convolutional layer. We will begin by looking at the hyper-parameters that you must specify for a convolutional layer in most neural network frameworks that support the CNN:\n",
    "\n",
    "* Number of filters\n",
    "* Filter Size\n",
    "* Stride\n",
    "* Padding\n",
    "* Activation Function/Non-Linearity\n",
    "\n",
    "The primary purpose for a convolutional layer is to detect features such as edges, lines, blobs of color, and other visual elements. The filters can detect these features. The more filters that we give to a convolutional layer, the more features it can detect.\n",
    "\n",
    "A filter is a square-shaped object that scans over the image. A grid can represent the individual pixels of a grid. You can think of the convolutional layer as a smaller grid that sweeps left to right over each row of the image. There is also a hyper parameter that specifies both the width and height of the square-shaped filter. Figure 10.1 shows this configuration in which you see the six convolutional filters sweeping over the image grid:\n",
    "\n",
    "A convolutional layer has weights between it and the previous layer or image grid. Each pixel on each convolutional layer is a weight. Therefore, the number of weights between a convolutional layer and its predecessor layer or image field is the following:\n",
    "\n",
    "```\n",
    "[FilterSize] * [FilterSize] * [# of Filters]\n",
    "```\n",
    "\n",
    "For example, if the filter size were 5 (5x4) for 10 filters, there would be 250 weights.\n",
    "\n",
    "You need to understand how the convolutional filters sweep across the previous layer’s output or image grid. Figure 10.2 illustrates the sweep:\n",
    "\n",
    "![CNN](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_cnn_grid.png \"CNN\")\n",
    "\n",
    "The above figure shows a convolutional filter with a size of 4 and a padding size of 1. The padding size is responsible for the boarder of zeros in the area that the filter sweeps. Even though the image is actually 8x7, the extra padding provides a virtual image size of 9x8 for the filter to sweep across. The stride specifies the number of positions at which the convolutional filters will stop. The convolutional filters move to the right, advancing by the number of cells specified in the stride. Once the far right is reached, the convolutional filter moves back to the far left, then it moves down by the stride amount and\n",
    "continues to the right again.\n",
    "\n",
    "Some constraints exist in relation to the size of the stride. Obviously, the stride cannot be 0. The convolutional filter would never move if the stride were set to 0. Furthermore, neither the stride, nor the convolutional filter size can be larger than the previous grid. There are additional constraints on the stride (s), padding (p) and the filter width (f) for an image of width (w). Specifically, the convolutional filter must be able to start at the far left or top boarder, move a certain number of strides, and land on the far right or bottom boarder. The following equation shows the number of steps a convolutional operator\n",
    "must take to cross the image:\n",
    "\n",
    "$ steps = \\frac{w - f + 2p}{s+1} $\n",
    "\n",
    "The number of steps must be an integer. In other words, it cannot have decimal places. The purpose of the padding (p) is to be adjusted to make this equation become an integer value.\n",
    "\n",
    "## Max Pooling Layers\n",
    "\n",
    "Max-pool layers downsample a 3D box to a new one with smaller dimensions. Typically, you can always place a max-pool layer immediately following convolutional layer. The LENET shows the max-pool layer immediately after layers C1 and C3. These max-pool layers progressively decrease the size of the dimensions of the 3D boxes passing through them. This technique can avoid overfitting (Krizhevsky, Sutskever & Hinton, 2012).\n",
    "\n",
    "A pooling layer has the following hyper-parameters:\n",
    "\n",
    "* Spatial Extent (f )\n",
    "* Stride (s)\n",
    "\n",
    "Unlike convolutional layers, max-pool layers do not use padding. Additionally, max-pool layers have no weights, so training does not affect them. These layers simply downsample their 3D box input. The 3D box output by a max-pool layer will have a width equal to this equation:\n",
    "\n",
    "$ w_2 = \\frac{w_1 - f}{s + 1} $\n",
    "\n",
    "The height of the 3D box produced by the max-pool layer is calculated similarly with this equation:\n",
    "\n",
    "$ h_2 = \\frac{h_1 - f}{s + 1} $\n",
    "\n",
    "The depth of the 3D box produced by the max-pool layer is equal to the depth the 3D box received as input. The most common setting for the hyper-parameters of a max-pool layer are f =2 and s=2. The spatial extent (f) specifies that boxes of 2x2 will be scaled down to single pixels. Of these four pixels, the pixel with the maximum value will represent the 2x2 pixel in the new grid. Because squares of size 4 are replaced with size 1, 75% of the pixel information is lost. The following figure shows this transformation as a 6x6 grid becomes a 3x3:\n",
    "\n",
    "![MaxPool](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_8_conv_maxpool.png \"MaxPool\")\n",
    "\n",
    "Of course, the above diagram shows each pixel as a single number. A grayscale image would have this characteristic. For an RGB image, we usually take the average of the three numbers to determine which pixel has the maximum value.\n",
    "\n",
    "[More information on CNN's](http://cs231n.github.io/convolutional-networks/)\n",
    "\n",
    "# TensorFlow with CNNs\n",
    "\n",
    "The following sections describe how to use TensorFlow/Keras with CNNs.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Access to Data Sets\n",
    "\n",
    "Keras provides built in access classes for MNIST.  It is important to note that MNIST data arrives already separated into two sets:\n",
    "\n",
    "* **train** - Neural network will be trained with this.\n",
    "* **test** - Used for validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of x_train: (60000, 28, 28)\n",
      "Shape of y_train: (60000,)\n",
      "\n",
      "Shape of x_test: (10000, 28, 28)\n",
      "Shape of y_test: (10000,)\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "from keras.callbacks import EarlyStopping\n",
    "from keras.layers import Dense, Dropout\n",
    "from keras import regularizers\n",
    "from keras.datasets import mnist\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "print(\"Shape of x_train: {}\".format(x_train.shape))\n",
    "print(\"Shape of y_train: {}\".format(y_train.shape))\n",
    "print()\n",
    "print(\"Shape of x_test: {}\".format(x_test.shape))\n",
    "print(\"Shape of y_test: {}\".format(y_test.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Display the Digits \n",
    "\n",
    "The following code shows what the MNIST files contain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape for dataset: (60000, 28, 28)\n",
      "Labels: [5 0 4 ... 5 6 8]\n",
      "Shape for single: (28, 28)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>175</td>\n",
       "      <td>26</td>\n",
       "      <td>166</td>\n",
       "      <td>255</td>\n",
       "      <td>247</td>\n",
       "      <td>127</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>36</td>\n",
       "      <td>...</td>\n",
       "      <td>225</td>\n",
       "      <td>172</td>\n",
       "      <td>253</td>\n",
       "      <td>242</td>\n",
       "      <td>195</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>49</td>\n",
       "      <td>238</td>\n",
       "      <td>253</td>\n",
       "      <td>...</td>\n",
       "      <td>93</td>\n",
       "      <td>82</td>\n",
       "      <td>82</td>\n",
       "      <td>56</td>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "      <td>219</td>\n",
       "      <td>253</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>80</td>\n",
       "      <td>156</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>150</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>253</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>253</td>\n",
       "      <td>249</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>253</td>\n",
       "      <td>207</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>250</td>\n",
       "      <td>182</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>78</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23</td>\n",
       "      <td>66</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "      <td>171</td>\n",
       "      <td>219</td>\n",
       "      <td>253</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>55</td>\n",
       "      <td>172</td>\n",
       "      <td>226</td>\n",
       "      <td>253</td>\n",
       "      <td>253</td>\n",
       "      <td>253</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>136</td>\n",
       "      <td>253</td>\n",
       "      <td>253</td>\n",
       "      <td>253</td>\n",
       "      <td>212</td>\n",
       "      <td>135</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>28 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    0   1   2   3    4    5    6    7    8    9  ...   18   19   20   21   22  \\\n",
       "0    0   0   0   0    0    0    0    0    0    0 ...    0    0    0    0    0   \n",
       "1    0   0   0   0    0    0    0    0    0    0 ...    0    0    0    0    0   \n",
       "2    0   0   0   0    0    0    0    0    0    0 ...    0    0    0    0    0   \n",
       "3    0   0   0   0    0    0    0    0    0    0 ...    0    0    0    0    0   \n",
       "4    0   0   0   0    0    0    0    0    0    0 ...    0    0    0    0    0   \n",
       "5    0   0   0   0    0    0    0    0    0    0 ...  175   26  166  255  247   \n",
       "6    0   0   0   0    0    0    0    0   30   36 ...  225  172  253  242  195   \n",
       "7    0   0   0   0    0    0    0   49  238  253 ...   93   82   82   56   39   \n",
       "8    0   0   0   0    0    0    0   18  219  253 ...    0    0    0    0    0   \n",
       "9    0   0   0   0    0    0    0    0   80  156 ...    0    0    0    0    0   \n",
       "10   0   0   0   0    0    0    0    0    0   14 ...    0    0    0    0    0   \n",
       "11   0   0   0   0    0    0    0    0    0    0 ...    0    0    0    0    0   \n",
       "12   0   0   0   0    0    0    0    0    0    0 ...    0    0    0    0    0   \n",
       "13   0   0   0   0    0    0    0    0    0    0 ...    0    0    0    0    0   \n",
       "14   0   0   0   0    0    0    0    0    0    0 ...   25    0    0    0    0   \n",
       "15   0   0   0   0    0    0    0    0    0    0 ...  150   27    0    0    0   \n",
       "16   0   0   0   0    0    0    0    0    0    0 ...  253  187    0    0    0   \n",
       "17   0   0   0   0    0    0    0    0    0    0 ...  253  249   64    0    0   \n",
       "18   0   0   0   0    0    0    0    0    0    0 ...  253  207    2    0    0   \n",
       "19   0   0   0   0    0    0    0    0    0    0 ...  250  182    0    0    0   \n",
       "20   0   0   0   0    0    0    0    0    0    0 ...   78    0    0    0    0   \n",
       "21   0   0   0   0    0    0    0    0   23   66 ...    0    0    0    0    0   \n",
       "22   0   0   0   0    0    0   18  171  219  253 ...    0    0    0    0    0   \n",
       "23   0   0   0   0   55  172  226  253  253  253 ...    0    0    0    0    0   \n",
       "24   0   0   0   0  136  253  253  253  212  135 ...    0    0    0    0    0   \n",
       "25   0   0   0   0    0    0    0    0    0    0 ...    0    0    0    0    0   \n",
       "26   0   0   0   0    0    0    0    0    0    0 ...    0    0    0    0    0   \n",
       "27   0   0   0   0    0    0    0    0    0    0 ...    0    0    0    0    0   \n",
       "\n",
       "     23  24  25  26  27  \n",
       "0     0   0   0   0   0  \n",
       "1     0   0   0   0   0  \n",
       "2     0   0   0   0   0  \n",
       "3     0   0   0   0   0  \n",
       "4     0   0   0   0   0  \n",
       "5   127   0   0   0   0  \n",
       "6    64   0   0   0   0  \n",
       "7     0   0   0   0   0  \n",
       "8     0   0   0   0   0  \n",
       "9     0   0   0   0   0  \n",
       "10    0   0   0   0   0  \n",
       "11    0   0   0   0   0  \n",
       "12    0   0   0   0   0  \n",
       "13    0   0   0   0   0  \n",
       "14    0   0   0   0   0  \n",
       "15    0   0   0   0   0  \n",
       "16    0   0   0   0   0  \n",
       "17    0   0   0   0   0  \n",
       "18    0   0   0   0   0  \n",
       "19    0   0   0   0   0  \n",
       "20    0   0   0   0   0  \n",
       "21    0   0   0   0   0  \n",
       "22    0   0   0   0   0  \n",
       "23    0   0   0   0   0  \n",
       "24    0   0   0   0   0  \n",
       "25    0   0   0   0   0  \n",
       "26    0   0   0   0   0  \n",
       "27    0   0   0   0   0  \n",
       "\n",
       "[28 rows x 28 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display as text\n",
    "from IPython.display import display\n",
    "import pandas as pd\n",
    "\n",
    "print(\"Shape for dataset: {}\".format(x_train.shape))\n",
    "print(\"Labels: {}\".format(y_train))\n",
    "\n",
    "# Single MNIST digit\n",
    "single = x_train[0]\n",
    "print(\"Shape for single: {}\".format(single.shape))\n",
    "\n",
    "display(pd.DataFrame(single.reshape(28,28)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image (#105): Which is digit '1'\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADFFJREFUeJzt3X/oXfV9x/Hne2nyh6Z/6Ioh2rDUKkLxDzuCDKaj2lmcFGL/kQaZmQtL/4i6yAITB04ZA5k2sr8CqcakIzMd+CuUZbELY65Yg1Gcv7JULSlN/CaZWGmKSmd874/vSfmq33vu1+/9ce437+cDLt97z+fe83lzyCufc+4593wiM5FUz+90XYCkbhh+qSjDLxVl+KWiDL9UlOGXijL8UlGGXyrK8EtFfW6cnUWElxNKI5aZMZf3DTTyR8S1EXEoIt6IiDsGWZek8Yr5XtsfEYuAnwLXAEeA54A1mflay2cc+aURG8fIfznwRmb+LDN/A+wCVg+wPkljNEj4LwB+MeP1kWbZx0TE+og4EBEHBuhL0pCN/Au/zNwKbAV3+6VJMsjIfxRYMeP1F5tlkhaAQcL/HHBxRHwpIpYA3wZ2D6csSaM2793+zPwwIm4B9gKLgG2Z+erQKpM0UvM+1Tevzjzml0ZuLBf5SFq4DL9UlOGXijL8UlGGXyrK8EtFGX6pKMMvFWX4paIMv1SU4ZeKMvxSUYZfKsrwS0UZfqkowy8VZfilogy/VJThl4oy/FJRhl8qaqxTdKueSy65pGfbwYMHWz8b0X4T2n53nt68eXPPtk2bNrV+tgJHfqkowy8VZfilogy/VJThl4oy/FJRhl8qaqDz/BFxGDgJnAI+zMxVwyhKZ46bb765Z1u/8/T92t96663W9j179rS2VzeMi3yuysy3h7AeSWPkbr9U1KDhT+CpiHg+ItYPoyBJ4zHobv8VmXk0Is4DfhQR/5OZT898Q/Ofgv8xSBNmoJE/M482f08AjwOXz/KerZm5yi8Dpcky7/BHxNkR8fnTz4FvAK8MqzBJozXIbv8y4PHmZ5efA/45M/9tKFVJGrnody51qJ1FjK8zjcWSJUta2998882ebeeff/5AfV955ZWt7c8888xA61+oMrP9RggNT/VJRRl+qSjDLxVl+KWiDL9UlOGXivLW3Wq1YsWK1vaHH364tX3Q03lt+t36W+0c+aWiDL9UlOGXijL8UlGGXyrK8EtFGX6pKM/zq9XVV1/d2n7VVVeNrO/t27e3tr///vsj67sCR36pKMMvFWX4paIMv1SU4ZeKMvxSUYZfKsrz/MUtWrSotX3p0qUj6/uDDz5obX/iiScG+rzaOfJLRRl+qSjDLxVl+KWiDL9UlOGXijL8UlF9p+iOiG3AN4ETmXlps+xc4AfASuAwcENm/rJvZ07RPXH63Zf/8OHDA63/5MmTPds2bdrU+tkHH3xwoL6rGuYU3duBaz+x7A5gX2ZeDOxrXktaQPqGPzOfBt75xOLVwI7m+Q7g+iHXJWnE5nvMvywzp5rnx4BlQ6pH0pgMfG1/ZmbbsXxErAfWD9qPpOGa78h/PCKWAzR/T/R6Y2ZuzcxVmblqnn1JGoH5hn83sLZ5vhZ4cjjlSBqXvuGPiEeAnwCXRMSRiFgH3AtcExGvA3/cvJa0gPQ95s/MNT2avj7kWtSBjRs3jnT9U1NTPds8j98tr/CTijL8UlGGXyrK8EtFGX6pKMMvFeWtu89w99xzT2v7hg0bxlSJJo0jv1SU4ZeKMvxSUYZfKsrwS0UZfqkowy8V5Xn+M9x5553X2r548eKR9r93796Rrl/z58gvFWX4paIMv1SU4ZeKMvxSUYZfKsrwS0V5nl8jdd9993Vdgnpw5JeKMvxSUYZfKsrwS0UZfqkowy8VZfilovqe54+IbcA3gROZeWmz7G7gL4D/bd52Z2b+66iKrG7RokWt7evWrevZduONNw67nI+5/fbbW9uPHTs20v41f3MZ+bcD186y/IHMvKx5GHxpgekb/sx8GnhnDLVIGqNBjvlviYiXImJbRJwztIokjcV8w78F+DJwGTAFfLfXGyNifUQciIgD8+xL0gjMK/yZeTwzT2XmR8D3gMtb3rs1M1dl5qr5Filp+OYV/ohYPuPlt4BXhlOOpHGZy6m+R4CvAV+IiCPA3wJfi4jLgAQOA98ZYY2SRqBv+DNzzSyLHxpBLerhpptuam3fsmXLyPo+dOhQa/v+/ftb20+dOjXMcjREXuEnFWX4paIMv1SU4ZeKMvxSUYZfKspbdy8At912W2d9P/vss63t/U71aXI58ktFGX6pKMMvFWX4paIMv1SU4ZeKMvxSUZ7nXwA2bdrU2v7UU0+NqRKdSRz5paIMv1SU4ZeKMvxSUYZfKsrwS0UZfqkoz/MvAPfff//I1n3y5MnWdq8hOHM58ktFGX6pKMMvFWX4paIMv1SU4ZeKMvxSUX3P80fECuD7wDIgga2Z+Y8RcS7wA2AlcBi4ITN/ObpSNQrvvvtua/uuXbvGVInGbS4j/4fAX2XmV4A/ADZExFeAO4B9mXkxsK95LWmB6Bv+zJzKzBea5yeBg8AFwGpgR/O2HcD1oypS0vB9pmP+iFgJfBXYDyzLzKmm6RjThwWSFog5X9sfEUuBR4GNmfmriPhtW2ZmRGSPz60H1g9aqKThmtPIHxGLmQ7+zsx8rFl8PCKWN+3LgROzfTYzt2bmqsxcNYyCJQ1H3/DH9BD/EHAwMzfPaNoNrG2erwWeHH55kkZlLrv9fwj8KfByRLzYLLsTuBf4l4hYB/wcuGE0JZ75Vq5c2dp+1llnjacQldI3/Jn5YyB6NH99uOVIGhev8JOKMvxSUYZfKsrwS0UZfqkowy8V5a27J8Ctt97a2n7RRReNrO+dO3eObN2abI78UlGGXyrK8EtFGX6pKMMvFWX4paIMv1RUZM56963RdNbjVl/V9TuPv2fPntb2Cy+8sGfbAw880PrZu+66q7X9vffea23X5MnMXj/B/xhHfqkowy8VZfilogy/VJThl4oy/FJRhl8qyvP80hnG8/ySWhl+qSjDLxVl+KWiDL9UlOGXijL8UlF9wx8RKyLiPyLitYh4NSL+sll+d0QcjYgXm8d1oy9X0rD0vcgnIpYDyzPzhYj4PPA8cD1wA/DrzLx/zp15kY80cnO9yKfvjD2ZOQVMNc9PRsRB4ILBypPUtc90zB8RK4GvAvubRbdExEsRsS0izunxmfURcSAiDgxUqaShmvO1/RGxFPhP4O8z87GIWAa8DSTwd0wfGvx5n3W42y+N2Fx3++cU/ohYDPwQ2JuZm2dpXwn8MDMv7bMewy+N2NB+2BMRATwEHJwZ/OaLwNO+BbzyWYuU1J25fNt/BfBfwMvAR83iO4E1wGVM7/YfBr7TfDnYti5HfmnEhrrbPyyGXxo9f88vqZXhl4oy/FJRhl8qyvBLRRl+qSjDLxVl+KWiDL9UlOGXijL8UlGGXyrK8EtFGX6pqL438Byyt4Gfz3j9hWbZJJrU2ia1LrC2+Rpmbb831zeO9ff8n+o84kBmruqsgBaTWtuk1gXWNl9d1eZuv1SU4ZeK6jr8Wzvuv82k1japdYG1zVcntXV6zC+pO12P/JI60kn4I+LaiDgUEW9ExB1d1NBLRByOiJebmYc7nWKsmQbtRES8MmPZuRHxo4h4vfk76zRpHdU2ETM3t8ws3em2m7QZr8e+2x8Ri4CfAtcAR4DngDWZ+dpYC+khIg4DqzKz83PCEfFHwK+B75+eDSki/gF4JzPvbf7jPCcz/3pCarubzzhz84hq6zWz9J/R4bYb5ozXw9DFyH858EZm/iwzfwPsAlZ3UMfEy8yngXc+sXg1sKN5voPpfzxj16O2iZCZU5n5QvP8JHB6ZulOt11LXZ3oIvwXAL+Y8foIkzXldwJPRcTzEbG+62JmsWzGzEjHgGVdFjOLvjM3j9MnZpaemG03nxmvh80v/D7tisz8feBPgA3N7u1Eyuljtkk6XbMF+DLT07hNAd/tsphmZulHgY2Z+auZbV1uu1nq6mS7dRH+o8CKGa+/2CybCJl5tPl7Anic6cOUSXL89CSpzd8THdfzW5l5PDNPZeZHwPfocNs1M0s/CuzMzMeaxZ1vu9nq6mq7dRH+54CLI+JLEbEE+Dawu4M6PiUizm6+iCEizga+weTNPrwbWNs8Xws82WEtHzMpMzf3mlmajrfdxM14nZljfwDXMf2N/5vA33RRQ4+6LgT+u3m82nVtwCNM7wb+H9PfjawDfhfYB7wO/Dtw7gTV9k9Mz+b8EtNBW95RbVcwvUv/EvBi87iu623XUlcn280r/KSi/MJPKsrwS0UZfqkowy8VZfilogy/VJThl4oy/FJR/w+1WvPYf6D8mQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display as image\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "digit = 105 # Change to choose new digit\n",
    "a = x_train[digit]\n",
    "plt.imshow(a, cmap='gray', interpolation='nearest')\n",
    "print(\"Image (#{}): Which is digit '{}'\".format(digit,y_train[digit]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Define CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (60000, 28, 28, 1)\n",
      "Training samples: 60000\n",
      "Test samples: 10000\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "from keras import backend as K\n",
    "batch_size = 128\n",
    "num_classes = 10\n",
    "epochs = 12\n",
    "# input image dimensions\n",
    "img_rows, img_cols = 28, 28\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(\"Training samples: {}\".format(x_train.shape[0]))\n",
    "print(\"Test samples: {}\".format(x_test.shape[0]))\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "model = Sequential()\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training/Fitting CNN\n",
    "\n",
    "The following code will train the CNN for 20,000 steps.  This can take awhile, you might want to scale the step count back. GPU training can help.  My results:\n",
    "\n",
    "* CPU Training Time: Elapsed time: 1:50:13.10\n",
    "* GPU Training Time: Elapsed time: 0:13:43.06"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/12\n",
      " - 100s - loss: 0.2629 - acc: 0.9193 - val_loss: 0.0532 - val_acc: 0.9822\n",
      "Epoch 2/12\n",
      " - 92s - loss: 0.0877 - acc: 0.9732 - val_loss: 0.0446 - val_acc: 0.9855\n",
      "Epoch 3/12\n",
      " - 101s - loss: 0.0648 - acc: 0.9807 - val_loss: 0.0333 - val_acc: 0.9873\n",
      "Epoch 4/12\n",
      " - 93s - loss: 0.0538 - acc: 0.9833 - val_loss: 0.0304 - val_acc: 0.9900\n",
      "Epoch 5/12\n",
      " - 101s - loss: 0.0459 - acc: 0.9858 - val_loss: 0.0310 - val_acc: 0.9895\n",
      "Epoch 6/12\n",
      " - 98s - loss: 0.0424 - acc: 0.9872 - val_loss: 0.0266 - val_acc: 0.9908\n",
      "Epoch 7/12\n",
      " - 95s - loss: 0.0374 - acc: 0.9883 - val_loss: 0.0311 - val_acc: 0.9896\n",
      "Epoch 8/12\n",
      " - 97s - loss: 0.0335 - acc: 0.9901 - val_loss: 0.0273 - val_acc: 0.9912\n",
      "Epoch 9/12\n",
      " - 204s - loss: 0.0324 - acc: 0.9900 - val_loss: 0.0279 - val_acc: 0.9906\n",
      "Epoch 10/12\n",
      " - 535s - loss: 0.0296 - acc: 0.9904 - val_loss: 0.0299 - val_acc: 0.9908\n",
      "Epoch 11/12\n",
      " - 113s - loss: 0.0278 - acc: 0.9912 - val_loss: 0.0277 - val_acc: 0.9906\n",
      "Epoch 12/12\n",
      " - 102s - loss: 0.0260 - acc: 0.9920 - val_loss: 0.0249 - val_acc: 0.9922\n",
      "Test loss: 0.024928450847776548\n",
      "Test accuracy: 0.9922\n",
      "Elapsed time: 0:28:58.25\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import time\n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=2,\n",
    "          validation_data=(x_test, y_test))\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss: {}'.format(score[0]))\n",
    "print('Test accuracy: {}'.format(score[1]))\n",
    "\n",
    "elapsed_time = time.time() - start_time\n",
    "print(\"Elapsed time: {}\".format(hms_string(elapsed_time)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate Accuracy\n",
    "\n",
    "Note, if you are using a GPU you might get the **ResourceExhaustedError**.  This occurs because the GPU might not have enough ram to predict the entire data set at once.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.024928450847776548\n",
      "Test accuracy: 0.9922\n"
     ]
    }
   ],
   "source": [
    "# Predict using either GPU or CPU, send the entire dataset.  This might not work on the GPU.\n",
    "# Set the desired TensorFlow output level for this example\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss: {}'.format(score[0]))\n",
    "print('Test accuracy: {}'.format(score[1]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GPUs are most often used for training rather than prediction.  For prediction either disable the GPU or just predict on a smaller sample.  If your GPU has enough memory, the above prediction code may work just fine.  If not, just prediction on a sample with the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 1.0\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "# For GPU just grab the first 100 images\n",
    "small_x = x_test[1:100]\n",
    "small_y = y_test[1:100]\n",
    "small_y2 = np.argmax(small_y,axis=1)\n",
    "pred = model.predict(small_x)\n",
    "pred = np.argmax(pred,axis=1)\n",
    "score = metrics.accuracy_score(small_y2, pred)\n",
    "print('Accuracy: {}'.format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Latest Advances in CNN's\n",
    "\n",
    "He, K., Zhang, X., Ren, S., & Sun, J. (2016). [Deep residual learning for image recognition](https://arxiv.org/abs/1512.03385). In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition* (pp. 770-778).\n",
    "\n",
    "* [Caffee ResNet GitHub](https://github.com/KaimingHe/deep-residual-networks)\n",
    "* [ResNet in TensorFlow](https://github.com/tensorflow/models/blob/master/resnet/resnet_model.py)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Geoffrey Hinton is suggesting that we might be [doing computer vision wrong](https://www.wired.com/story/googles-ai-wizard-unveils-a-new-twist-on-neural-networks/) and has introduced two new papers about an old technique called Capsule Neural Networks:\n",
    "    \n",
    "* Sabour, S., Frosst, N., & Hinton, G. E. (2017). [Dynamic Routing Between Capsules](https://arxiv.org/abs/1710.09829). arXiv preprint arXiv:1710.09829.\n",
    "* [Matrix capsules with EM routing](https://openreview.net/forum?id=HJWLfGWRb&noteId=HJWLfGWRb) - ICLR 2018 Blind Submission\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Module 7 Assignment\n",
    "\n",
    "You can find the first assignmeht here: [assignment 7](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class7.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (wustl)",
   "language": "python",
   "name": "wustl"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}