01A_TensorFlow_Basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: If running this notebook directly, make sure you are running your Jupyter kernel in an environment with TensorFlow installed. Some useful packages to install/import first, if you didn't in notebook 00A:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fetching package metadata .......\n",
      "Solving package specifications: ..........\n",
      "\n",
      "# All requested packages already installed.\n",
      "# packages in environment at C:\\Users\\kevin_000\\Anaconda3\\envs\\tensorflow:\n",
      "#\n",
      "matplotlib                2.0.2               np113py36_0  \n",
      "Requirement already up-to-date: tqdm in c:\\users\\kevin_000\\anaconda3\\envs\\tensorflow\\lib\\site-packages\n"
     ]
    }
   ],
   "source": [
    "# Beginning a line with \"!\" allows you to execute a command in your terminal\n",
    "# You'll need to install these packages for certain lines of this notebook if you don't already have them.\n",
    "!conda install -y matplotlib\n",
    "!pip install -U tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from tqdm import trange         "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic TensorFlow test from the installation instructions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'Hello, TensorFlow!'\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "hello = tf.constant('Hello, TensorFlow!')\n",
    "sess = tf.Session()\n",
    "print(sess.run(hello))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define a simple TensorFlow graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The example from the TensorFlow Introduction slides:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7.0\n"
     ]
    }
   ],
   "source": [
    "a = tf.constant(3.0, dtype=tf.float32)\n",
    "b = tf.constant(4.0, dtype=tf.float32)\n",
    "sum_a_b = tf.add(a, b)\n",
    "\n",
    "# Using the Python \"with\" as a context manager\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(sum_a_b)) # Prints “7.0” to the screen "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous graph only produces a constant output. Not particularly interesting. To create a similar graph that can acccept variable inputs: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1st Result: 7.0\n",
      "2nd Result: 4035.0\n"
     ]
    }
   ],
   "source": [
    "a = tf.placeholder(tf.float32)\n",
    "b = tf.placeholder(tf.float32)\n",
    "sum_a_b = tf.add(a, b)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    feed_dict = {a: 3.0, b: 4.0}\n",
    "    print('1st Result: {0}'.format(sess.run(sum_a_b, feed_dict=feed_dict)))\n",
    "    \n",
    "    feed_dict = {a: 2015, b: 2020}\n",
    "    print('2nd Result: {0}'.format(sess.run(sum_a_b, feed_dict=feed_dict)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST dataset is very popular machine learning dataset, consisting of 70000 grayscale images of handwritten digits, of dimensions 28x28. We'll be using it as our example for this section of the tutorial, with the goal being to predict which the digit is in each image.\n",
    "\n",
    "![mnist](Figures/mnist.PNG)\n",
    "\n",
    "Since it's such a common (and small) dataset, TensorFlow has commands for downloading and formatting the dataset conveniently baked in already:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at how the data is organized:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training image data: (55000, 784)\n",
      "Validation image data: (5000, 784)\n",
      "Testing image data: (10000, 784)\n",
      "28 x 28 = 784\n",
      "\n",
      "Test Labels: (10000, 10)\n",
      "Label distribution:[(0, 980), (1, 1135), (2, 1032), (3, 1010), (4, 982), (5, 892), (6, 958), (7, 1028), (8, 974), (9, 1009)]\n",
      "\n",
      "Train image 1 is labelled one-hot as [ 0.  0.  0.  1.  0.  0.  0.  0.  0.  0.]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x1e8803f1eb8>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADhBJREFUeJzt3V2MVPUZx/HfU7E36IWydCWKiyYGo16gWUkvkGisKMYE\nuDG+xNBUWWOsKdqL4kusCYqmqVa4QddIxMa3BthIDNYoaZAmDeHNKu6CWoMCQRbERI0XVvfpxRya\nVff8zzBzZs4sz/eTbHbmPHNmHo/748yZ/5zzN3cXgHh+VnUDAKpB+IGgCD8QFOEHgiL8QFCEHwiK\n8ANBEX4gKMIPBDWhnS9mZnydEGgxd7d6HtfUnt/MrjGzPWb2kZktaea5ALSXNfrdfjM7SdIHkq6S\ntF/SVkk3uvtgYh32/ECLtWPPP1PSR+7+sbt/K+llSfOaeD4AbdRM+M+UtG/U/f3Zsh8wsz4z22Zm\n25p4LQAla/kHfu7eL6lf4m0/0Ema2fMfkDR11P2zsmUAxoFmwr9V0nlmdo6Z/VzSDZLWl9MWgFZr\n+G2/u39nZr+V9IakkyStcvf3S+sMQEs1PNTX0ItxzA+0XFu+5ANg/CL8QFCEHwiK8ANBEX4gKMIP\nBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjC\nDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gqIan6JYkM9sr6StJ30v6zt17y2gK7dPT05Os33bb\nbcn6/fffn6ynZoE2S08mOzQ0lKw/8MADyfrAwECyHl1T4c9c4e5HSngeAG3E234gqGbD75LeMrPt\nZtZXRkMA2qPZt/2z3P2Amf1C0ptmttvd3x79gOwfBf5hADpMU3t+dz+Q/R6WNCBp5hiP6Xf3Xj4M\nBDpLw+E3s4lmduqx25LmSNpVVmMAWquZt/3dkgay4ZoJkl5097+X0hWAlrPUOGzpL2bWvhcLZPLk\nybm1e++9N7nuzTffnKxPmjQpWS8aq29mnL/ob3Pfvn3J+qWXXppbO3LkxB2ddvf0hs0w1AcERfiB\noAg/EBThB4Ii/EBQhB8IiqG+caDotNmlS5fm1or+/7Z6uO3w4cPJekpXV1eyPm3atGR9cHAwt3bh\nhRc20tK4wFAfgCTCDwRF+IGgCD8QFOEHgiL8QFCEHwiKcf5xYOvWrcn6JZdckltrdpw/NVYuSVdc\ncUWy3syps7NmzUrWN23alKyn/tsnTCjjwtWdiXF+AEmEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4/wd\n4Pzzz0/Wi8b5P//889xa0fn0RePwd999d7K+ePHiZH3ZsmW5tU8//TS5bpGiv92RkZHc2h133JFc\nt7+/v6GeOgHj/ACSCD8QFOEHgiL8QFCEHwiK8ANBEX4gqMJxfjNbJek6ScPuflG27HRJr0iaJmmv\npOvd/YvCF2OcvyFF3wNIjdU3OxV1X19fsr5y5cpkPTVN9o4dO5LrLliwIFlfs2ZNsp762z7jjDOS\n647nKbzLHOd/TtI1P1q2RNJGdz9P0sbsPoBxpDD87v62pKM/WjxP0urs9mpJ80vuC0CLNXrM3+3u\nB7Pbn0nqLqkfAG3S9IXM3N1Tx/Jm1icpfeAIoO0a3fMfMrMpkpT9Hs57oLv3u3uvu/c2+FoAWqDR\n8K+XtDC7vVDSq+W0A6BdCsNvZi9J+pek6Wa238xulfSYpKvM7ENJv8ruAxhHCo/53f3GnNKVJfeC\nHLt3767stYuuB7Bnz55kPXWtgaJrBSxZkh5BLppzoJXffzgR8A0/ICjCDwRF+IGgCD8QFOEHgiL8\nQFAn7jzFgcyePTu3VnQ6cNFQ3tDQULI+ffr0ZH3Lli25tcmTJyfXLTrdvKj3uXPnJuvRsecHgiL8\nQFCEHwiK8ANBEX4gKMIPBEX4gaAY5z8B3HTTTbm1RYsWJdctOi22jku7J+upsfxmTsmVpBUrViTr\nRZcGj449PxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ExTj/Ca5onL7K9Tdv3pxc95577knWGcdvDnt+\nICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiqcJzfzFZJuk7SsLtflC17SNIiSccunH6fu29oVZNIe/HF\nF3NrPT09yXW7urqS9aLr/k+cODFZT3nwwQeTdcbxW6uePf9zkq4ZY/lf3H1G9kPwgXGmMPzu/rak\no23oBUAbNXPMf5eZvWtmq8zstNI6AtAWjYZ/paRzJc2QdFDS43kPNLM+M9tmZtsafC0ALdBQ+N39\nkLt/7+4jkp6RNDPx2H5373X33kabBFC+hsJvZlNG3V0gaVc57QBol3qG+l6SdLmkLjPbL+mPki43\nsxmSXNJeSbe3sEcALWDNnq99XC9m1r4XQymKxvkffvjhZH3+/Pm5tZ07dybXnTt3brJedF3/qNw9\nPSFChm/4AUERfiAowg8ERfiBoAg/EBThB4JiqK9OqammDx8+nFuL7vXXX8+tXX311cl1iy7d/eST\nTzbU04mOoT4ASYQfCIrwA0ERfiAowg8ERfiBoAg/EBRTdGdmz56drD/+eO6VyrR79+7kurfccktD\nPZ0IHnnkkdzanDlzkutOnz697HYwCnt+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwgqzDh/6nx8SXrq\nqaeS9eHh4dxa5HH8oim6n3766dyaWV2nnaNF2PMDQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCF4/xm\nNlXS85K6JbmkfndfbmanS3pF0jRJeyVd7+5ftK7V5ixYsCBZLzp3fNOmTWW2M24UTdG9du3aZD21\nXYvmjCi6TgKaU8+e/ztJv3f3CyT9UtKdZnaBpCWSNrr7eZI2ZvcBjBOF4Xf3g+6+I7v9laQhSWdK\nmidpdfaw1ZLmt6pJAOU7rmN+M5sm6WJJWyR1u/vBrPSZaocFAMaJur/bb2anSForabG7fzn6e9nu\n7nnz8JlZn6S+ZhsFUK669vxmdrJqwX/B3ddliw+Z2ZSsPkXSmGe+uHu/u/e6e28ZDQMoR2H4rbaL\nf1bSkLs/Maq0XtLC7PZCSa+W3x6AVimcotvMZknaLOk9SSPZ4vtUO+7/m6SzJX2i2lDf0YLnqmyK\n7qIhq6GhoWR9cHAwt/boo4829dzbt29P1ov09PTk1i677LLkukVDoPPnpz/HLTotN/X3tXz58uS6\nRVN0Y2z1TtFdeMzv7v+UlPdkVx5PUwA6B9/wA4Ii/EBQhB8IivADQRF+ICjCDwRVOM5f6otVOM5f\nZM2aNcl6ary7mbFuSdq5c2eyXuTss8/OrU2aNCm5brO9F62fmqJ7xYoVyXWPHDmSrGNs9Y7zs+cH\ngiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAY588UTeG9YcOG3Fpvb/oiRSMjI8l6K8fai9b95ptvkvWi\ny2cvW7YsWR8YGEjWUT7G+QEkEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIzz16mrqyu3tnTp0qaeu68v\nPZvZunXrkvVmznsvunY+02SPP4zzA0gi/EBQhB8IivADQRF+ICjCDwRF+IGgCsf5zWyqpOcldUty\nSf3uvtzMHpK0SNLh7KH3uXv+Se8a3+P8wHhR7zh/PeGfImmKu+8ws1MlbZc0X9L1kr529z/X2xTh\nB1qv3vBPqOOJDko6mN3+ysyGJJ3ZXHsAqnZcx/xmNk3SxZK2ZIvuMrN3zWyVmZ2Ws06fmW0zs21N\ndQqgVHV/t9/MTpG0SdIj7r7OzLolHVHtc4Clqh0a/KbgOXjbD7RYacf8kmRmJ0t6TdIb7v7EGPVp\nkl5z94sKnofwAy1W2ok9Vrs07LOShkYHP/sg8JgFknYdb5MAqlPPp/2zJG2W9J6kY9egvk/SjZJm\nqPa2f6+k27MPB1PPxZ4faLFS3/aXhfADrcf5/ACSCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4\ngaAIPxAU4QeCIvxAUIQfCIrwA0EVXsCzZEckfTLqfle2rBN1am+d2pdEb40qs7eeeh/Y1vP5f/Li\nZtvcvbeyBhI6tbdO7Uuit0ZV1Rtv+4GgCD8QVNXh76/49VM6tbdO7Uuit0ZV0lulx/wAqlP1nh9A\nRSoJv5ldY2Z7zOwjM1tSRQ95zGyvmb1nZu9UPcVYNg3asJntGrXsdDN708w+zH6POU1aRb09ZGYH\nsm33jpldW1FvU83sH2Y2aGbvm9nvsuWVbrtEX5Vst7a/7TezkyR9IOkqSfslbZV0o7sPtrWRHGa2\nV1Kvu1c+JmxmsyV9Len5Y7MhmdmfJB1198eyfzhPc/c/dEhvD+k4Z25uUW95M0v/WhVuuzJnvC5D\nFXv+mZI+cveP3f1bSS9LmldBHx3P3d+WdPRHi+dJWp3dXq3aH0/b5fTWEdz9oLvvyG5/JenYzNKV\nbrtEX5WoIvxnSto36v5+ddaU3y7pLTPbbmZ9VTczhu5RMyN9Jqm7ymbGUDhzczv9aGbpjtl2jcx4\nXTY+8PupWe4+Q9JcSXdmb287kteO2TppuGalpHNVm8btoKTHq2wmm1l6raTF7v7l6FqV226MvirZ\nblWE/4CkqaPun5Ut6wjufiD7PSxpQLXDlE5y6Ngkqdnv4Yr7+T93P+Tu37v7iKRnVOG2y2aWXivp\nBXdfly2ufNuN1VdV262K8G+VdJ6ZnWNmP5d0g6T1FfTxE2Y2MfsgRmY2UdIcdd7sw+slLcxuL5T0\naoW9/ECnzNycN7O0Kt52HTfjtbu3/UfStap94v8fSfdX0UNOX+dK+nf2837VvUl6SbW3gf9V7bOR\nWyVNkrRR0oeS3pJ0egf19lfVZnN+V7WgTamot1mqvaV/V9I72c+1VW+7RF+VbDe+4QcExQd+QFCE\nHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeC+h9PPuXddgFbfgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e8fcd37f98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Dataset statistics\n",
    "print('Training image data: {0}'.format(mnist.train.images.shape))\n",
    "print('Validation image data: {0}'.format(mnist.validation.images.shape))\n",
    "print('Testing image data: {0}'.format(mnist.test.images.shape))\n",
    "print('28 x 28 = {0}'.format(28*28))\n",
    "\n",
    "print('\\nTest Labels: {0}'.format(mnist.test.labels.shape))\n",
    "labels = np.arange(10)\n",
    "num_labels = np.sum(mnist.test.labels, axis=0, dtype=np.int)\n",
    "print('Label distribution:{0}'.format(list(zip(labels, num_labels))))\n",
    "\n",
    "# Example image\n",
    "print('\\nTrain image 1 is labelled one-hot as {0}'.format(mnist.train.labels[1,:]))\n",
    "image = np.reshape(mnist.train.images[1,:],[28,28])\n",
    "plt.imshow(image, cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic Regression Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the graph input: this is where we feed in our training images into the model. Since MNIST digits are pretty small and the model we're using is very simple, we'll feed them in as flat vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define input placeholder\n",
    "x = tf.placeholder(tf.float32, [None, 784])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get our predicted probabilities of each digit, let's first start with the probability of a digit being a 3 like the image above. For our simple model, we start by applying a linear transformation. That is, we multiply each value of the input vector by a weight, sum them all together, and then add a bias. In equation form:\n",
    "\n",
    "\\begin{align}\n",
    "y_3 = \\sum_i w_{i,3} x_i + b_3\n",
    "\\end{align}\n",
    "\n",
    "The magnitude of this result $y_3$, we'll take as being correlated to our belief in how likely we think the input digit was a 3. The higher the value of $y_3$, the more likely we think the input image $x$ was a 3 (ie, we'd hope we'd get a relatively large value for $y_3$ for the above image). Remember though, our original goal was to identify all 10 digits, so we also have:\n",
    "\n",
    "\\begin{align*}\n",
    "y_0 =& \\sum_i w_{i,0} x_i + b_0 \\\\\n",
    "&\\vdots \\\\\n",
    "y_9 =& \\sum_i w_{i,9} x_i + b_9\n",
    "\\end{align*}\n",
    "\n",
    "We can express this in matrix form as:\n",
    "\n",
    "\\begin{align}\n",
    "y = W x + b \n",
    "\\end{align}\n",
    "\n",
    "To put this into our graph in TensorFlow, we need to define some Variables to hold the weights and biases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define linear transformation\n",
    "W = tf.Variable(tf.zeros([784, 10]))\n",
    "b = tf.Variable(tf.zeros([10]))\n",
    "y = tf.matmul(x, W) + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can interpret these values (aka logits) $y$ as probabilities if we normalize them to be positive and add up to 1. In logistic regression, we do this with a softmax:\n",
    "\n",
    "\\begin{align}\n",
    "p(y_i) = \\text{softmax}(y_i) = \\frac{\\text{exp}(y_i)}{\\sum_j\\text{exp}(y_j)}\n",
    "\\end{align}\n",
    "\n",
    "Notice that because the range of the exponential function is always non-negative, and since we're normalizing by the sum, the softmax achieves the desired property of producing values between 0 and 1 that sum to 1. If we look at the case with only 2 classes, we see that the softmax is the multi-class extension of the binary sigmoid function: \n",
    "<img src=\"Figures/Logistic-curve.png\", width=\"500\">\n",
    "\n",
    "Computing a softmax in TensorFlow is pretty easy, sort of*:\n",
    "\n",
    "*&#42;More on this later*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Softmax to probabilities\n",
    "py = tf.nn.softmax(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That defines our forward pass of our model! We now have a graph that performs a forward pass: given an input image, the graph returns the probabilities the model thinks the input is each of the 10 classes. Are we done?\n",
    "\n",
    "Not quite. We don't know the values of $W$ and $b$ yet. We're going to learn those by defining a loss and using gradient descent to do backpropagation. Essentially, we'll be taking the derivative with respect to each of the elements in $W$ and $b$ and wiggling them in a direction that reduces our loss.\n",
    "\n",
    "The loss we commonly use in classification is cross-entropy. Cross-entropy is a concept from information theory:\n",
    "\n",
    "\\begin{align}\n",
    "H_{y'}(y)=-\\sum_i y'_i \\text{log}(y_i)\n",
    "\\end{align}\n",
    "\n",
    "Cross-entropy not only captures how *correct* (max probability corresponds to the right answer) the model's answers are, it also accounts for how *confident* (high confidence in correct answers) they are. This encourages the model to produce very high probabilities for correct answers while driving down the probabilities for the wrong answers, instead of merely be satisfied with it being the argmax. \n",
    "\n",
    "In supervised models, we need labels to learn, so we create a placeholder for the labels in our training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define labels placeholder\n",
    "y_ = tf.placeholder(tf.float32, [None, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cross-entropy loss is pretty easy to implement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Loss\n",
    "cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(py), reduction_indices=[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the old days, we would have to go through and derive all the gradients ourselves, then code them into our program. Nowadays, we have libraries to compute all the gradients automatically. Not only that, but TensorFlow comes with a whole suite of optimizers implementing various optimization algorithms. I'm not going to go into the details of why you should appreciate that right now, because I know that Prof David Carlson has an entire day's worth of material on optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Optimizer\n",
    "train_step = tf.train.GradientDescentOptimizer(0.05).minimize(cross_entropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train, we simply call the optimizer op we defined above. First though, we need to start a session and initialize our variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a session object and initialize all graph variables\n",
    "sess = tf.Session()\n",
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are much cleverer ways to design a training regimen that stop training once the model is converged and before it starts overfitting, but for this demo, we'll keep it simple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████| 1000/1000 [00:01<00:00, 571.94it/s]\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "# trange is a tqdm function. It's the same as range, but adds a pretty progress bar\n",
    "for _ in trange(1000): \n",
    "    batch_xs, batch_ys = mnist.train.next_batch(100)\n",
    "    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice, because of the way the dependency links are connected in our graph, running the optimizer requires an input to both the training image placeholder `x` and the training label placeholder `y_` (as it should). The values of all variables (`W` and `b`) are updated in place automatically by the optimizer.\n",
    "\n",
    "Now let's see how we did! For every image in our test set, we run the data through the model, and take the digit in which we have the highest confidence as our answer. We then compute an accuracy by seeing how many we got correct:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.902899980545044\n"
     ]
    }
   ],
   "source": [
    "# Test trained model\n",
    "correct_prediction = tf.equal(tf.argmax(py, 1), tf.argmax(y_, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "print('Test accuracy: {0}'.format(sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels})))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not bad for a simple model and a few lines of code.  Before we close the session, there's one more interesting thing we can do. Normally, it can be difficult to inspect exactly what the filters in a model are doing, but since this model is so simple, and the weights transform the data directly to their logits, we can actually visualize what the model's learning by simply plotting the weights. The results look pretty reasonable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABIEAAACBCAYAAABXearSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnVvMJld1ptfG2ObQxnYf3O52H+w2PtBxEkDOHMRolJCJ\nQJEQd6NEyYgLJG5mooCQCJm5zUVmJCKuLQU5FxGjkRIpIEWKGJQITTSxYiNgsLG73d3G3Xa72yfA\nhxgwqbnw35WnXv69ur5zVX3vIyHq66q/atdee+29v8/rXas0TRPGGGOMMcYYY4wxZtq8bdMNMMYY\nY4wxxhhjjDGrxz8CGWOMMcYYY4wxxmwB/hHIGGOMMcYYY4wxZgvwj0DGGGOMMcYYY4wxW4B/BDLG\nGGOMMcYYY4zZAvwjkDHGGGOMMcYYY8wW4B+BjDHGGGOMMcYYY7aAhX4EKqV8tJTyRCnlyVLK55fV\nKLNebMfxYxtOA9tx/NiG08B2HD+24TSwHcePbTgNbMdpUZqmme8PS7kmIk5FxG9ExIWI+MeI+O2m\naR5bXvPMqrEdx49tOA1sx/FjG04D23H82IbTwHYcP7bhNLAdp8fbF/jbfxURTzZNczYiopTyPyPi\n4xFRHQw33HBDc+DAgQUeaebl3LlzLzRNs1vnz2TH66+/vtmzZ8/qGmqqvPTSS0uxYUTEnj17mr17\n966moSbl/PnzS7Gjbbg5lmXDiIh3v/vdtuMGeOmll+K1114rldMz2fGd73xnc+ONN66moSbl0qVL\nS/PFd73rXc1NN920moaaKj/4wQ/i9ddfX4ovej7dHBcuXFiaL77jHe/wd40N8Oqrr8Ybb7yxFF+0\nDTfHiy++WPPFDov8CHRbRJzH5wsR8a+zPzhw4ED80R/90QKPNPPyO7/zO9+vnJrJjnv27ImPfOQj\ny2ya6cmXv/zlpdgwImLv3r3xuc99bllNMzPwe7/3e0uxo224OZZlw4i37PjpT396WU0zPfniF7+Y\nnZ7JjjfeeGP87u/+7pJaZmbhC1/4wtJ88aabbopPfepTy2rayinlX76rzRvVPwQeeOCB7PTM6+Jn\nPvOZJbXMzMJnP/vZpfninj174mMf+9iymmZ68tWvfjU7PfP3RdtwMzz44IM1X+yw8sTQpZRPlVIe\nLqU8/Morr6z6cWYF0IZvvPHGpptj5oR2fPXVVzfdHDMHtuE0oB1fe+21TTfHzAFt+Prrr2+6OWZO\nbMfx4/l0Gvi7xvixDcfFIpFAz0TEUXw+svNvHZqmeSAiHoiIOHHixHj/U8V0uaodacN9+/bZhsNj\nZl88duyY7Tg8ZvJF23CQzOyLR48eXakd54kQyP6G0Qfzsox7rJiZfPHWW2+1Lw6PmX3x8OHDS7Vj\nNs77nsuumzf6p+/fLfu6OZnJF1c9n5q5mNkX9+/fvzE7DjGqbiBr5ky+uEkb9oW2XsZcOxA79WaR\nSKB/jIi7Sil3lFKui4jfioivLKdZZo3YjuPHNpwGtuP4sQ2nge04fmzDaWA7jh/bcBrYjhNj7kig\npmneLKX8l4j4m4i4JiK+1DTNo0trmVkLtuP4sQ2nge04fmzDaWA7jh/bcBrYjuPHNpwGtuP0WEQO\nFk3T/HVE/PWS2mI2hO04fmzDaWA7jh/bcBrYjuPHNpwGtuP4sQ2nge04LRb6EWhM/PM//3Ov66j7\nqx3v9pnUNIH67311329728rzd4+CeXS68+oza7af5X5j04YOmXlzE/Az5wC9jj6mdsvOGTN0Mt/h\nuZ/97Ge7Hl/tHiTzj2uuuaY9znyq77poX/x5Ft3nRKy3z7dtb1PzAT133XXX7frvERHXX399e/z2\nt7+9el1mY/XvK7z55pudzz/5yU92PdZreaxjcIj5VZbFqnMy9WVev9zmOXSePWW2v6ztNWchmw/7\nrpkZU7b3Mr4jzHO/iPm+8/e9x6rZrhXYGGOMMcYYY4wxZkvxj0DGGGOMMcYYY4wxW8Co5WCzhNwx\n/DX7O17HcDANk81C/2phZBrude2117bHDOvd7XONqYVTZ6GX7OfasZL1D+2RPSuTg/H++qy+MqIp\nhGhmoe0c5xER73znO9tjhrYzBF4/s49+/OMfd66jb/70pz/tnHv11Vfb41deeaU91tB2+n1m4yyc\nf4x2VAlBX/iu7Ae1YW0e0zHB/lcb8lxNghDR9VmVO9TaO0WyMGbtF/ZhZgN+7jvfaj9zLGQyluzc\nGH1sGWR9XtuLZPuSvnunbF7vu7Zm95x37R4y88r8Oe45P3KNjOjOse94xzva43e/+929rovorsFs\nxz/90z91rvvBD36w63FExI9+9KP2+LXXXmuPdX2u7anHyjzyoL7XzUJtnM0rr639zZjpK4XW62ry\nRt1z8DP3lNl1mfy2tkZGdH1Y/Znz8rx7ujEyj49lUvds/eTfZXv/bI3M9kd9/XnZjHOVNcYYY4wx\nxhhjjDEz4R+BjDHGGGOMMcYYY7YA/whkjDHGGGOMMcYYswWMLidQVuY5027Wyt+q3o46TOouNYcF\nz6kGuqYN7avrz/4u03uOUUOf2VDzUtT6T7Wz1Ma/613vao/V1rz/G2+8UW0Xbb9nz57qs7W9vCfH\nSJZfakxa7FpJ24huzoH3vOc9nXMHDhxoj2+++eZdj/UeWT4L9rPmNLh48WJ7fOHChfb40qVLneuY\nO0jzBdVy3vTNf7NpMu1xlpeMuSjoR/o506pzrqUPMD+TflYbMt8EbZ/lhlJq+TYybf0UyUpF1+ar\niO7cluUH4tjSvtXcJlfQPs/GZM0+2Xo/ZGr7mcxOWT6zLKdBrdy39jF9W9c72pDzivY37aT3qOUE\n0vfiGByLPSP653vpm2eSa1NE18e4Bukelfugffv2dc7V5sAsN6Xev2/OwzGS2a1v3pAsVyE/c9zP\nkiOrlj9G7cTPOn8uI8fX0Ohbtj3LL0ib0FbZPEc/0jmP6P6GubVef/31Xdse0V2f++YLmsL+Zl5f\nrNlX15navkevUx8mtfk02wOpn9b8uW/uoHkZ368GxhhjjDHGGGOMMWZm/COQMcYYY4wxxhhjzBYw\nTP2CUAv50tC8WrizXsuQOEpOIrphfPv372+Pb7zxxs51/DsNseOzXn755fZYS2xSApGFizJELZNs\naDjcUOVhWbm+2nvruUx2QBuqZIdwjGRysL6lVnXM1eSJ+s68bkxht5kfMfz84MGDnXO33HJLe3zT\nTTdV70EbZGWFGfbOcNqI7phh2K36oobck5qkSW3F/ti0HbNQfY5n+orK9m644YZdjyO6kpFsDmLo\nM20zS9h7LVw3k7uoP/Oe9GHtG47BTdtwFjJ7Z3LTmqQnC4/PZHdZaWuOLx7rHN03ZJr2p29H5GH1\nmyQr257JR7Kwd/YJ+1L3JfQJ9Q+SSUToO7U5IKJrXy1dzms5rl588cXOdc8//3x7rPP6kOVhWUl0\n2i6TD/M6ymH1nuw/tQHnshdeeKFzjjJp2lTby3VRZSy1uTgr1Txkau3MbJjtX9lfakPOV/wekO1D\ns70/ba3S7excba7NZLhDWxf7pgXJ0jXouVo5cJVVHj16tD3mvpYpDyK63x+1/+hjZ8+ebY+fffbZ\nznWcH3U+JHwXXQPGIvmryfh078HP+n2R/leT90V0fZPHmrKA5/T7AvuVe2X93aC2B4ro+iaP9XsR\n1/hMKtaXYf5KYIwxxhhjjDHGGGOWin8EMsYYY4wxxhhjjNkCRiEHI1loWBbyVQurzKoaZaHVDJ3U\ncOea3CLL2q9hoAxlY+hoJnPIpE+bpm94H0P6NOSR4Xi8n0q0aiGCGrpLSZCG/tWys2fVj9Q2tYpT\n2g7eX0NbhxSiGdFtK/tIQxYZEqnjkn3x3HPPtcfqs7WqGVnlHA0J5ZihNFOvIxpiWassoCHTbMeQ\nKoVl1X8ox9u7d2/nOs5dWu2CfkDbqM+yL3kPnTOze7C99FOV9HH8qD/z2qwCBMfqkGy4G5wbMjlY\nVk2iJrlU/2B/MixdZSb0AcqpIyKOHTvWHlMiqtfRBpnEhfZWucVYqMkV+la6i+j6B8PPdd6tjXtd\nj2rV+BTOAZxHIrrzv84dbCPfRecfjlXKlyJ+Xpq0bjLJU60SrZJJGzLpHm3Cfsikreqn2R6M1CpJ\nRtQrZarP9q10s26yZ/eVU+v7cBxk6xHJ0ljQNn0r6eo+iu3V8cg5oa9sT78LbcKGWQW+2thWP+I5\nHdu1Com6VvE69qXuTWgD9bdaWhCdU9lGnf9q83RW0UrZpGwzs2Em8+d3LN0D1NKHaGoD7kUOHz68\n679HdNc7fRbXJ8qYeRzR3dtmEnZKxbRvOOfouLUczBhjjDHGGGOMMcbsin8EMsYYY4wxxhhjjNkC\n/COQMcYYY4wxxhhjzBYwyKQHqm+slUdVbWWWN6RWTlH1y9TbUYuoORKoc9Y8MdTpUX+YlTzPdH9Z\nrqOsHOmQqGnmszwwqnvlZ/al6q1rfaJ6Sd5Pxxx1z3yW6jj5bG1vLaeN6sgzjfnQqOUEUh/gu6su\nljbPcibU+i/TtaumvlZeXOcOjg3NpcF3pr31WTqXrJNs3Og56tiZm0dzjdTKu0d0+7KW+0rhGNHS\nmWyH5hzgHMqxc+jQoc51HBdabrqWPyZbQ8ZELedaRHds63xLO7JffvjDH3auu3z5cnt84cKF9vjc\nuXOd63iPI0eOVNvBNVLnZdpEx2Qtb5HmCuBYyHIkrRtdo2tru15H39F8WswdwWPNF8C1i32n6xbz\nWWT9n5WN5udsj8Wyypqrge3g+Bsiffc3Gbw22/PV8gWprehHum9hfjzO33oP5qaYpdwxqeUu2wRs\nS1Y+u/Z9QT/rOfW5K6iP0R60hc67RMu7c5+S7XM5frJ5MttjsW+GlNcpIs8nw3NqmyxvGe3Pv9Nn\nMdcW/UjtyDlb+5a+zpxD2s+1eV6vpT04tiK6c+qm849mexaOWe49tTQ7/UjnyVq+Od2XvO9972uP\nf+mXfqk9/uAHP9i57sCBA+3xpUuXOuceeuih9vib3/xme3zq1KnOdcyBqmOklg9X5xiOn9p8MwvD\n/sZpjDHGGGOMMcYYY5aCfwQyxhhjjDHGGGOM2QIGKQdTaiU3Nawuk3TwM0OoNOSuVs5Sn8VQW5UU\nMCSeIacausUQRL1/LdRTwzmHKgHL2pWFnWb2rZV+1zBG9hH78eabb+5cx/A+lZTVbK8hfHyWSmH4\nnrR9VkJ1TLCPspD1TCZXKzWu96ANVHpG+2dSmEySl4XC1iRCOj43KSXKnq3vVrObSqg41l966aXO\nOYYVc77T0pl81r59+9rjO+64o3Pd0aNHq+3lZw2DrqHjke+SlcAe2nyalcJlv9TkKEpWgpjHun7S\n53icrZ8aYs+QbJ7L5GAqY+FcwndRuWIWMr3usPeanD2iLsfIJMMqm6JMhPsNvUettHhWqlbhPTI5\nHvtYxwHfmWuhzuuUGG1aRqSoL/bdo87zHipboo/R9npvznksYRwR8cwzz7THlFio1PDWW2+t3p/t\nyOyYpWlYtS/q/dmXug+rScAy+aW+K/uEz9L3ru1ldS+YtYP9yvVY13Gu3ZQvRdSlhUPa28wK+53H\nKquhHfVc7btZVoac/f7973+/cx3nVB0zNand8ePHO9dxfOo+iGMj+07COUF9Y9V7H70/P+t3aPoE\n+zjb36sklXMj5zHuQyO6Pkf7Pvroo9VnqTz57Nmz7bGup4Q21HWc92ebVj1HDmtlNcYYY4wxxhhj\njDErwT8CGWOMMcYYY4wxxmwB/hHIGGOMMcYYY4wxZgsYTDKSTDdcy4uQaaUzTT11eZr7gDpCaje1\nvDvLXmu+DGr9spKt/KzlNwn1qqpJpdZ006Ua50HtRJuqPrqmt1Z4HfMA3XvvvZ3r2K+ZZp46atV7\nai4EwrGVaTyHlu8go5aHRH2ApSmpV9d70MdUv6xj/QrqKyyrqWVUazmBMrtleWJ4vyGVns7Ici3x\nWPNiZaXAn3322fY48w/qrz/wgQ+0x6p3p03Vtzl+NL8UyXKN1caczv/ZvLJp+o7LWcpSc16iDTSP\ni+rXr6B+z/68//77O+fuu+++9pg2yPL56BzAPuB8ofbm2rqMMqqrgnM/bZHlx9GcBrWS1ZpDhPev\nlSOP6M4JWpK35os65rL8jLX31PZyz5XluRoafXM76Tn2O+2o705fZF/OsketlYrO9ia6H+M5zpvr\nzjUyC2yzjsuaH+l7M3+J7iNq/ZDtcznuNa8J96+676HfZnMc26jzem1PNLQ9avadUKG9spyg3Dfq\nnMp+ol/p/obfG9h/mjuUeyndox48eLA95tjSvQjtz/x6Ed3vtNk+lPfc9Jya2ZDnaE/NhcTPukep\n9Zf6ANexf/iHf9j13yO640DXxdrvC7qOcwxm3x94rL69bF+86t1KKV8qpVwupXwX/7a3lPK1Usrp\nnf+/ObuHGQS3246jxzacBrbj+LENJ4D3N5PAvjgB7IuTwL44AeyL20Ofn5QejIiPyr99PiK+3jTN\nXRHx9Z3PZti8ELbj2LENp4HtOH5sw2nwYNiOY8e+OA0eDNtx7NgXp8GDYTtuBVeVgzVN841Syu3y\nzx+PiF/dOf6ziPi7iPiDJbarSt8wY4UhVQyJ0/BLXsdQdMofIiIuXbrUHmso9C233NIe33nnne0x\nS5JHdMMCNTSMYWRsh4b1aih3wqsR8ZL820bsyHC2rISnhsHRvjzW8oI1+1KOENENH9RQ6lrJP5Un\nMGxRQ/9qZZV13M4Q3rd2G2rbONZrZbcjuiUyKQ2L6Iah0nYafsmQS7bj2LFjnetYXvzw4cOdc+z3\nWunQiO5Y07HA9mbjcwYp0VrtqDIijmf2+XPPPde57syZM+3x448/3jlXs6+ObYbr3n777e2xSjNp\nt0z2VCtzHdGVOOiY47W10N2I4drwanCMapg130ntw/mxNr9GdH2R/axh6UeOHGmPf/EXf7Fz7q67\n7mqPOXfo+slnZaW4ORbm9cV17290vaM9eE6lUVyrNOydIebcU2Th8Zl8hH2s8ofz58+3x5Q/6Lji\nXodraUR3f8Q5gWV8I7rjQKULMi4G5Ys1mVREdz3StbWvnJg2Zr9rP9PHtP84FtgmlVtQqqL24T0o\nn5pXvrAKX5xF1lQ7p3tsfs72nplEl+fop7qP4rP6zmn6ztm8XvvOlEn/rjJO1+6Lukaw7RyLWgae\n41eldjU5mPoR5176B+fJiJ9PiUDof5kkm9fpuZqUKBv/mRxs09/7Oa/xXXVN41yoexGOA/6dSjiZ\n6uCJJ55oj3V/mUkLOW9SZqjrIp+tZeZre6J514m+zCsuO9g0zRUx5HMRcTC72AwW23H82IbTwHYc\nP7bhNLAdx49tOA1sx/FjG04D23GCLJxhqHnrZ9hqhqdSyqdKKQ+XUh7WxKNmOGR2pA2zhLpms8zi\ni5rYzAyHvr5oGw6XWXyxlgDdbJ6+vqjJrc1wmMUXbcfh0tcXPZ8Ol1l80d81hou/L06HeauDXSql\nHGqa5mIp5VBEXK5d2DTNAxHxQETEiRMnepUJyDLBMyRSw/sY+pdVh2FomIZYMiSLIX0qh3jqqafa\nYw31pDyF5/RZDLXVcES2ke+lYeJZmGEPetmRNty3b9/CpR5KUjUhC3XjOMgqNDCMj/2lEhTa4+//\n/u875zgOuDHUsZlVN+E5yk4WCLXdjbl88dixY73sqNIc9julAhr+Sv9QCWOt+oxWVOB19AfdqNOO\nKu+pZepXW3HM6DvXJJeZXHEOZvbFeW3I96FsihLXiG5FC1YAi+jOawyFpfw1IuLDH/5we/yhD32o\nPaYMJKI7/+l4qVUw0TZRzqbV/ui3DCHWOXnBUNu5fPHo0aNVO2aVe2rnNASZ415DplkVhWHMOmae\nfPLJXc+pfIR2PXnyZPVZDLHv628R9b1A5ourmlNpw1tvvbWXL2ZSDb6DVpDhPKbvynP0I5XvsO84\nRnQPwblb52TOEZw71GZsP+Vf+pkh8Srz5Rw9g+y9beo8vnj48OGF9ze0T1aBSscC7UifyGQs/A+r\naiv22aFDh6rP4v01PQLHkN6DMg3uYfRHGK4V6/DFbD4lOu+wnTXZqZ7LxiX3FCot4T1r1awiur6i\n/9GHcy/bof+xvfZeEd39HG2o+/IFKxLN5Yv79+/Pfiza9TiiLt9XaAOV9WUpOMiJEyfaY9oqSw2h\n3xe57nKfq/egn2p7ec/sPw5mNu5RLWxmX8xsSDLZLMd5JsNSiRz7iMea2uX06dPtMdc0vR9TFtx2\n222dc5S6057ax7y/vjM/cy7P5P3LkIbN69lfiYhP7Bx/IiL+auGWmE1gO44f23Aa2I7jxzacBrbj\n+LENp4HtOH5sw2lgO06QPiXivxwR/zci7imlXCilfDIi/jgifqOUcjoi/sPOZzNs7gjbcezYhtPA\ndhw/tuEE8P5mEtgXJ4B9cRLYFyeAfXF76FMd7Lcrp359yW0xq+Vc0zT37/LvtuN4sA2nge04fmzD\nCeD9zSSwL04A++IksC9OAPvi9rBQEotNQK2gloujtlbzxFBPSR2hanWZZ+Kxxx5rjx955JHOdcw/\noZph6klrpeMiurp51WKzvdRqqk6U+kDVDm6STKuY5QTKtMe13CCqgWYugVqJ6ohumevnn3++c446\nYD4r0+nqOeq7aadVl/xbJjqmqK1lLolnnnmmcx11t6pRZk4fnlOfZW4F9p/qoZmbSMucU9fLfCia\nZ4H3VJ1/Lbmd5kPJfFHniHWi/VV7V31vjlPNJcN8ZtTC33fffZ3rfu3Xfq09vueee9pjzYFGH1YN\nPn2TOUmYsyiiO3drXgr1zSsM2fci+s+jHHu6LtJ2mieGtmNZb80vwjwx9J0sxxCP9e+y+TArX054\njyXnWVsZ2hZ+5jton2T9RWhD5iaM6M5jzJmleyD6jvpRrUwuczxF5PkZOefTZzWfG99lSDa8Gmxr\nlhukb54VzanE+3MO1Fx5zGHBvozo9nXWXj5b82DUcpS8+OKLneuYB2Pde9RsHdZxybwq/Dsd21lO\nIL4r9yK6ptGveA/NBcb1VM+xHWy7zt1cW3WvzHfL8uxk59ZBlg9P/Yg25tyme3zaQHMx8R7c5773\nve/tXPcrv/Ir7TH3mrpnpH+on9b6Vu+R2ZFt5Pyqc3uW62rd1PYvEd3vcNwr6vdk+kT2Ptl3Ffbz\nkSNH2uPjx493ruP8p/sorrXcf2n/0266Z2Ef8FjnqQXzc/0cy72bMcYYY4wxxhhjjBkk/hHIGGOM\nMcYYY4wxZgsYjByMIU5ZSTSGf6mkgOFVGjrJUD2GKmvo6tmzZ9vj73znO+0xpUMReak3hqwxDI1/\nE9ENpdeQ31poWxamOSQ52Lxk71ALqcxKkjKET8PvGIqu9mXpWo5NLS2eheaNKYS9LwxNZKhjJiVS\nX+RnSka0b+lH9F+VctH/Mhkoz2lZZIZFa7guxxPD6NX2HF+bDrXNqLVNQ6LZ/2pDygYoDaPkK6Jb\nOpN/8/TTT3euo6xB28fwZo4zLbVKf9MxQjjHjGnOzMqL0z4aqszSzioLof9xPc38mdfp2sf76Zih\nTejPKqvk32Vh9Rwn2bq4adh3OrZr64f2Pz9rn3BO4rHKcDnn1Uq962f1D/o6w951HNA3VU7Dccu9\nmNqMfbXsEPhFycZX7f30XCZhZN9qqWJKi2gr9SPOA1pauTae1Gfpzyr95HyerZ8cd5uWg3H/ojJp\nwnPqb3xXleVwHeOeUiUo7BPuo3Tupr+pzIv9z/aqv9HW+s61OXTI8+nV4Dtm0tZsL1GTyanElt/b\n6M8qyb733nvbY0pxIyLOnTvXHtOfOUdH/Pw8QPjOHOOzpNtYtW/2lbZHdNcx3ZcS2lBlU7Qh50xd\ng7k/4vGxY8c613EupNQ2ojv3sk0qQaRN9bcHprVg29VmtNMmS8QbY4wxxhhjjDHGmBHhH4GMMcYY\nY4wxxhhjtoDByMEyapVPNFM4wyM1tI3nGCat4Vpnzpxpj8+fP98ea+hWFkLLZ/WtBqHh3wwz5Tl9\nrzHJGXYje5+sMpWG/hFKHhjep+HxDMtUORjtzTBofS5DC7NKFFlFtDFRq2ajoeIMXc0qqtGftZoQ\n/ZtzAMMmI7qhnrUqUNreTPKVyYxqFd92+1y7/yrIKmRklYYy6QJto+Oe19Ke6gP0I4bEa7UU2jQb\nLxwTGgrLOVND4hlKPyYJbRbuy5BpSrRUWsw5UCXU7GuuaSqB4FigZEGvyyoS1dZglcVkEmo+L1sP\nlh0yvSrop5yTNGSd51TSQakP1zSVoNDnnnzyyV3/JqIrm9A5gZJdSlU0ZJ/9n1WoyeafofsmyfYB\nhHOv9lmtIo76B+3Dsa3VT+mn2re0OdcHbRPnXj3He+p8S7L5dt2+yedn0iiOUZ3jatXtIrpyj0z6\nwbWKz9I+5ryrNqTduBZoH7P9en/2/6r3KKsik/zRV3Seo300PQDnLK0IRmhH2kDlYLS/ytJYRY7f\nP3VNq1U91nO0sUrI6M9Dkvxl+1eufdonWdVM2pDzk/YJ/Yr30N8GOM649umzmFLm8ccf71xHW2cp\nKfhe2Xcay8GMMcYYY4wxxhhjTC/8I5AxxhhjjDHGGGPMFuAfgYwxxhhjjDHGGGO2gEHmBFKNZ03j\nqGWAmZtH9bO8B/NPqE6UJR6Zj0BziFCLp3kWqMVmfgbVcWY5SnguK/84Jt38Faj31NwHNV22Qv2t\n5oaihpdjQnMCMS+ClmOkPjprL9uYvUuWL2bI+Su0PfS52267rT3WfD78u6xUNHMJZTmB2M/M1aV/\np35Pra6WWCW0Vaabz3KS8e80j8C680Dxefps+g5tozp2voPql3mOc9Wzzz7buY524xyneZ1INp/S\n1jpeOE+qnpttZNt1fI/JFwnHfVZ6WqEdOD9q3gL6LHMM6Rp88uTJXa/Te/LvOB4jujbQObXvdXzn\nodmR0D+YX0Lz6HBsaz4t+uapU6faY31vzn8sS8w9jz5b51Puv5iLSMcc1089R2gnfdaQ7Zbl/8ty\nVfGdshxs7FvdG/L+XD/379/fuY79rvN3La+Q7lO4puk8wnmUbdT8llmex3WTzRn8zH225srhe+ue\nkvfgHKfDJTfxAAAcP0lEQVS2YQ4XXvcLv/ALnetOnDjRHuv6zL/jc3XvQZvqmMv22GTI66JS6wvt\nlyyPFb+r0XbqH/RFPlf3N9yPaH4o5iPl/knHHfdFms+K0O+zMa7+vEm7ajs5h/Bdtf+5d9DcOTVf\nzOaxp59+etd/j4i499572+M77rijc45r5qOPPtoecz2O6M4XOv5qeYCyXHnLyLHmSCBjjDHGGGOM\nMcaYLcA/AhljjDHGGGOMMcZsAaOQg9XCpDQUnWFTDLeM6IY9MnRLwzlrZdpU0sLPx48f75y78847\n22OWzNVwToZuaQgtwwnZJg0NzsL7hkQtDFdDHvlZQx4Z5kiZnYbaHjlyZNe/0XKeLOWn7ajJtzRs\nMQvNq53LQqI3XUJV0VD+WoizSjrob/oOlBQwxFVLSvNcJpWgX2lIaK00uPoR5wv1dYbE81k6xwyp\n3GpW9pdjmz6mNuTf6ftQXkC/Uh/jeOE9dL6jre+5557OOcrBKMHT+ZT31zHHc5TT6HsNOexd5wa2\nPZPa0a467ulXtKnOvXw2JShHjx7tXEf5gvYf78/jy5cvV69TWRrXwr6h7UOaU7WdNTmxyjQoIXjq\nqac65x577LH2mLZWf+Z7M9Rdbc35WedCzvm8TtdLjk09V5OqZnbJ5H6bQMcU+5DHKimgzEvXqkxq\nTDgv0z5qR87FKtNlf1K2qc+l7dQXOYdzndXrMunKuum7X8v2obxO90f0CY5nTQVBG3Lfo3Iw7mXV\nj9jPnNc5p0fk83ptPs18dujrIudO2lHnjOw7BG3C/cj73ve+znX0529/+9vt8SOPPNK5jmucpiXg\nGGK/az9zPs/kqLxftt5seg4l+j6cN9lmHb/cg3NfElFPI6PvTXtwnc2kdLrH4t9xrlXpH8ejrvF8\nl3XKZoczCowxxhhjjDHGGGPMyvCPQMYYY4wxxhhjjDFbgH8EMsYYY4wxxhhjjNkCRpETiFAnqZpJ\n6ikzLSTvr9pXaqJr+veIbp6KX/7lX+6cu/vuu9tj6hS1TdQHZnpr5i3SfChDJdNbZyXvVUNPaANq\nrA8ePNi5jhp39pfq4qnr1PxStbKEOl766qP7atGHprfW8oTUQHNsa/6JmtZcr6UOVvMR1PLJqPa3\nVp43ojueeH+1Nz9rrh9+zkpSvvzyy7u2abfPq4bjSNvJzxx7mktA/45Q486cAzrHUZfNZ2meImrt\nNdcP9fmZL3KMaNuZOyPLLzUknbyiGvJa2XAda9S86/tx3dE1iNAHmOcuy8FFf4iIOH36dHvMsuQX\nL17sXEefzfyIY0jnqVpOt02j78N2cl5jvoqIru9oTgmWH+a+Qec49lc273Jt1bmW1/IeOq7YDj1X\ny1+R5bwaml/2zUOiOWOyHGy01759+9pjHds8x2dpGXjudzT35W233dYeM68X1/eIetl0vSfHpD6L\nfaP9sUmyvCrZe/OzjkvOh1yDs33Ue9/73vZYS0/TZ9W+XLuynHDZXoxtzHLJDG1fSvSdajmo9Dsc\nvzfcd999nXMnT55sj+lvmhOI/cL17cyZM53raLtsP8Y26R6Jexhd3zh3cKwNbd6soTbkvJblr8ty\nd9X6RPd8up5eIVv79Lvkc8891x5zL6ZzTC3/k8JxtWobjmOEGGOMMcYYY4wxxpiF8I9AxhhjjDHG\nGGOMMVvAcGIzQRbex9BGDdPU8DnCkCqG32lpQEqJsvBIhtNqSWOGevLvtL21sGj9O4bX6j3GQk2y\no6WieU77hDIR9rHakH/39NNPt8davprPUgkKw/EYtj2L7IDvPK/EcROwrTruGZLKcEn1Pb6Hykx4\nT0oPNEyWfc1zGkbP8GeVzPAeDOdUu/G9tJxrrWSyShc5vrQd6w7L5fNU3sZ+YL/qvMsQcw2lZj/w\nOvVnht5mbaLki3NrRLd8PP1UfSULoeU5tlHtNGR0DuF70Me0LCnnQ31f/h1lcjrPcV3kOGE4tj5L\n59tz5861x5QTZhJn9VO2i8/KxsKm59RMCspxyrFNiU5E12fVNuxXyg50/aQfce7W+Y6+mZUxJ9rH\n2V6sJn/Wsalja0hkMsVsPNPf9P1qpd91vTt+/Hh7zL2hyoVoY5UZ8R7cP+n8nUln+Ty2Q9dFjtdN\n+2JGTdKX7VEz36b/qd8wnQTLwOs6yz6+dOlS5xz3PVyDtf/ZRl0Xa2XHdd4dcsoCnTdoL4497dtj\nx461x/od7sSJE7v+nfbtt771rfb4oYceao9VDpbJkThOuPfROZQ2yUql87q+3zs2TV85mPoifUD3\nG7X0MPodmr7DeZcywIjuHKeS15okO0upoOOW46CvNHMZNnQkkDHGGGOMMcYYY8wW4B+BjDHGGGOM\nMcYYY7aAQcrBFIZNMXRLq19kmbcZWsfQaobFRnTD0FjdREOrMzkSQywz6RPREG/+HcMHNWxuLNnf\nGZaYyaTYz2rfmixHQ+IYNstwQa2aQDQkvtamTLantqidG1IY5m6wfVk4Ke2oIev0Nw1rZR/WKlVF\ndP2PoZ7MxB/RDbHXMFnOA9nYov3Vnznu+M4afp/597qrg3G86btSFsLjrHphJs2gv2mFMUKfZTh8\nRDf8WqsysP85T6qtOX40bJu+z3DgoftiRk2CotUuGLqs0kzaizamrfTv2O/qA1m1OX7OKn5wHOp8\nS3tl1UCHatdsHuAcqqHo3LOof1BOoiHxtftn4ea1OVPJ+p/o3oawP8YkzVQ43vhOmWxAq7JREks/\n4hwd0bU357W+1cb0HOcObS8/s+pNRHd9pj8PeU/at0orx2JWlVH9uVYBU6VI/P5AH+P3m4jueNF9\nD6+lDTM/yvZifeXUQ6BmKz1Xq0QbUU/bEdEd66xk+dRTT3WuO3XqVHv88MMPt8cqyeacrfsxtrEm\n1Y/ozre6H9N7XmHd+8556VupT6V09AHd2/Bctn/luKAMW30xexb3NrXUM4rue+iL9NNsD7QMhjtb\nG2OMMcYYY4wxxpilcdUfgUopR0spf1tKeayU8mgp5fd3/n1vKeVrpZTTO/9/89XuZTbDzq/Sd9uG\no+da++K42fmvp/bF8WNfHDlvvvlm2IaTwL44cn72s5/ZF6eBfXHk2Be3iz6RQG9GxGebpjkZEf8m\nIv5zKeVkRHw+Ir7eNM1dEfH1nc9mgOyEp12wDSeBfXHE2BcnhX1xxOyEVduG08B2HD+24TSwHUeM\n18Xt4qo5gZqmuRgRF3eOXymlfC8ibouIj0fEr+5c9mcR8XcR8QfLaJRq3mqaQNXl1f4moluikfpc\nzQlEPR81zwo1mNpe6g/Z3iz/jWoW+Zma16wkeY0d7evrO21Yiw0V9hHfW3W6vE7P0W68h+rYqcfN\ncvbwWaq757VZ/2e5fmrn9LoZNJ4/bZrmmxHrs6OOWfYFc+Do+KUWNtPFUnutdqQGntp4ze1EX1dt\nNP2eGnAeR3THmuZDoR3ZRs29Ql1wTZe/M85W7otZSdeabVR7zLGuvsi8SdTMZ7Zh3iXNCXT48OH2\nWG2o2ukrqD6ceSlYgjyiO844VtWfh+yLfdum/cUxq2ua5k66guYXoa8zB5Tq5pn/RfuWnznuNE8F\n/U3zyfBz9qw+fXXNNdfEpudTjsUs5wD9TfO7sKww1z61DfM3ca7S3E1so87d9M1a7sOI7vynNqR9\ns1wmM+SWWbsvZvn/OPbUv7im6VhgP3Gu1D0q84u8//3vb491PqRNtCwyxwnnhGeffbZzHdddXe84\nJ9D/Mr+v+eUQfJHjLxt7tXwlCn1H1zTaOivvTtuofflZfZ1kOS1rdlsg78jG96iktubouWeeeaZz\n7vTp0+0x581z5851ruM5ztnZdw21cW0/xr2rkn2HUF8nfb4/vu1tb1u7LyqlkvtPbc1xr+sYbcM+\nyfYU9GfNgZZ936nlR83mQm0Hz9Xy90VsOCdQKeX2iPhARDwUEQd3fiCKiHguIg5W/uZTpZSHSykP\nZ0lDzXpY1Ia1Lw1mvSxqR/3Bxawf23AaLGrH7D9mmPWwqA2zL2FmfdiO48fz6TTwd43xYxtOn94/\nApVS9kTEX0TEp5um6fzn3uatn+d2/Tm2aZoHmqa5v2ma+7NfNs3qWYYNs6gOsx6WYUetVmHWi204\nDZZhR42yMutlGTbU6EGzfmzH8eP5dBr4u8b4sQ23g14l4ksp18Zbg+HPm6b5y51/vlRKOdQ0zcVS\nyqGIuFy/w9VhOJWG0jEEjCGQGibF6zRMk6FXXCRUlsBn80crLf/M0FH9tZP/hZ5hY9om3jMLaczK\nuc4QMl1ixTbMYKhb9uU1K6/He7DvNGyP/c++09K6maSP7aB9M0lf3/Lxi5RQXYcvEn1f/lc2vq+G\n8vO/qKq0hPek7SgzieiGzvM6/S99bIeWd6fsi3OASlDom/pfgxlWz/DsLFw08+dYgy/y+VmpS/oY\nJQgR3XGqG2v+Hc9pyXDeg1+wtDwyn619R3+mn85STrdWwnORUNt1+6JSCy3OSkWrbKcWupzZm76i\n6yLbofMcn5XNvZmUse+z+rJuG2Z7G/rO008/3bmOttF1jOtp1nf0AZ7TdZa2Vz/lOdpen0Vbqz9z\nPc3KbfOeV5lPN27Hmv+phKe2N4zoShZ43T333NO5jvIw2l59m/OhyrxqUmtd07L/ks/35Hqayf8y\nhjSf1qQZEd1+ztYP+orKwdivtLtKeTIfy2SzhO3Xd6HdliQH27gv1qREKs3JpP20A22lUlzuLynL\n1T0q76d+ylQUBw/+S3CN2pRzoPol75mtAX3tum4bZmObdlN/42e1by3Vge796ZvsV91fci5X2/D+\nvF/Wpuy7PO+3bPmX0qc6WImIP42I7zVN8yc49ZWI+MTO8Sci4q+W3zyzDHYmheNhG04B++KIsS9O\nCvviiNnxRdtwGtiOI8a+OClsxxFjX9wu+kQCfSgi/lNE/L9Syrd2/u2/RsQfR8T/KqV8MiK+HxH/\ncTVNNIty6tSpiIh9EfFh23DU7An74qg5e/ZshH1xCtgXR87Of2W3DcePfXHk7ERF2Ybjx744cnai\nimzDLaFPdbD/E2/JF3bj15fbHLMKdsKJH2ma5v5dTtuG4+HVpmnsiyPmzjvvjLAvTgH74si5/vrr\nwzacBPbFkXPdddfZF6eBfXHkXHvttfbFLaJXTqB1QD2c6r+pd6RmLyv1ptrXWm4ezX3AUsXUf6pW\nmvp91XaTrKwcUa1jTas7Vmq5frIyeXqOfcn+17LH1PdSM6+5FJijRBNCsv9Z9lr1vNRrZtrkrJT8\nqjWfi6C5fmo5DbTyH8dz5ou0FcvWRkS88MILu55TPXRfPXwtr0lE1956jlpgtlfLoXN8ZqVjV4E+\nL8txxHegnTQnEH1Hx8GRI0faY+ridRyw77K8WGyv6vO5HlBbn5XAVk0+x0iWx2vIvqhtq/mY5lxj\n//XNb6HX1XIHaT4ZzqN6rqaH1/WN84OOO55bRu6DdaPt4rtzbGsOF773Lbfc0jlHv81yK3L9o8/q\n3ME2af4EtpHjJduzaV4czpO6no4FtWMt56Tu+bhm6J6SOfGefPLJ9vgb3/hG5zruS+lT6ivs56x8\nPN9F11Lmt1B/5rUcM9k+aEhkvpjl7mCfaJ/zWvqOrjOaS63Ps5QsVwqhnbLvTNkedWjU8v5EdPuC\n/ax7gr79x73O3Xff3Tl37Nix9pj9p3vZ8+fPt8f6fYXvwnbo/pLt13fme2bfF4dq4yw/Do91X5Il\n8q8liNc1rfa9QL9nZGXmeQ/mEtb2sU1ZrjSup7pGXi0/3qzMn6HWGGOMMcYYY4wxxowG/whkjDHG\nGGOMMcYYswUMRg5GsjBNhkap9IDnslKLDI1l6JY+m9dpOC2frWWRKZnRdpBMllAr17hIefF10jfU\nMAvlzELMM2kdQ/A0fLD2LO1X3j8rRcz3zCQJQwq9nAXt55ocQ0MWOe419JmfKenRcF36GH1KxwVD\nplXawHBt+rCWPs5KTfIembxz3RKwjCyslWHG2RyU+VGtPK2Gr9NuHCNZqLPKvGrlMnX+z0Kia++Z\nzaeZZG0TZG2tyVEiuu+r4cl8pyzEntfxnNqbz9Yxw5BphmSrpIL317HLa2ttHxq0W1YGPZOkUiKZ\nlVznfkbD3lmKmPfXuZv3y+bkrHx4bc7U9md2W3bY+zLJpJkc5ypBzyRC3EdSKkYpSUTXXlmZcLZD\n/ZTjhONCr6v5rLZ/WeXF10lmQ76rzqfsZ527antFva6vDfl3+l2C/ke/1GfV5piIcdpNycqGcy/x\n/PPPd66jv+3fv79zjvvDTDJPeditt97aHjOVQUTEhQsX2uPLl7uV1SkDpd/r/MrPOjfWZF5jmkMJ\nx2X2HY7vTYlzRHeNy+R+3MdzvGjf8R76rAMHDrTHnPN1PuU9dI1nO9YpmR7HLwrGGGOMMcYYY4wx\nZiH8I5AxxhhjjDHGGGPMFjBIOVjfrNkaZsxQLg3Hq4WOZ9IzPiuTxWThWlmoZ626gv7d1MgkNZmk\nj3Iehr+qVK8mGdBwUI6XTD6SySTmkXyNNew2otu3WQUY9m0W1ppJCihdYUh2VlFOQ0fpw5RUUKYU\n0Q2T1pDpmi8OuWpfNk/Wqt3pdawypX3Cz7ST2oZ9R3/O/F6fxbHVN1xXw3+zymQ1Ni3/uhq1qiL6\nfjWpSkS337PQ8Vp1Te0jjiGVJWTVNUht/s7aqP8+1Dk2kxj2lX2rXJLXcj5VWzM0nc/SOZjzpEou\n1aZXmMVXanusMVPb52W+qFIBSk2yNZI+W5Np6rOyan88ziRf2buMRfqetY3vk1Ua5rqoPlZbgzL/\nyHy9Jp2P6K7jtXX2as+qyYiGbMOIvK20F/cm2n/0I61Iyj7M9khnzpxpjzkuMjm1+jPlYXyu2pHt\n13b0/X47Fth/2f6ec6b2ec2HMxkWbaPfaXg/VmjUzxxzur7xWfodZFOVMqf7S4MxxhhjjDHGGGOM\nafGPQMYYY4wxxhhjjDFbgH8EMsYYY4wxxhhjjNkCBpkTSKmVWFV9KzWBWc6JWn4D/ZxpK/nsrLz7\nvHkqxlgWvi/MK6Cayewc+5W6S9XW11DNaFaqttb/UywDPwt9cx8wt4DqlzMbk1rfZn6f2Ydk99A2\n1eaEWn6MoaHtrOVrUl08+yTLZ5bljajlJlK9NT9nc3c2n2a+OEXfnGeN0PWuluNL8xbQdnyu2pFl\nd7P21nKSRORltMeYhyQjKx9PanbSz9m+hNAvNfcE29E3Z8827V+uRt8cgtovtbwV2R61L1nOyWx/\nw7/LbDwFONb75iXTuYs5grL1szaPKVwLX3755c652pygtuY9MhuOdT7VtrKvmadHbcW167XXXuuc\n4zp28eLF9vixxx6rtoPfQ2644Ybquey7ZJZnMrPJ2POR9i0Xn33P0O+BzCOa5WuiL9KnsnylvHdE\n1+85P+g9sn1uLVfuqvPmbdfqbIwxxhhjjDHGGLOl+EcgY4wxxhhjjDHGmC1gFHIwkoVPM6xLQ74Y\nNsbwqizUqm8YVhbeN29Y9JRDqPuW6cxsSDmYUrPbLFKuWhuHGk65LvpKtLKy1KSvL9baMMu1ywin\nHQt9JQPscw1PpU01XLr2LLVhzb6ZrectRTw1G85C5oucNzMZJO2opaJrEk61YyYtqc3tWXh2JqeZ\nmr37SsOUWmlZleGSzIbzzJnZHih79tSZRV5V65dV9Nc8+5up+ZtSk4BlJaW17HitLLXKwdiXOtfW\n2qSyTZ7LJEVZ2oMpkM09Ndl6RHdf+p73vKdzrmb/bH+TtWkeG8ybemIKNuY7ZL7ItU/9gyXYX3jh\nhfY4218StW1Ndp21XW2Ryc14/3WukdP9pcEYY4wxxhhjjDHGtPhHIGOMMcYYY4wxxpgtwD8CGWOM\nMcYYY4wxxmwBo8sJRGbJm5OV3iPzlN/sy5Tz/MzL1LWt28qUc3dMiWy+6zsXel4bFlkekizP2jzM\nol0fexnbdbJqn1r2PsdzwOw4p9lw4bymObf4WfN69M21VMvhlvlR5rN9c3xtA/PsPft+P9wk22TH\neXLiZH7a997LzsWj/kwbDiU3nlduY4wxxhhjjDHGmC3APwIZY4wxxhhjjDHGbAFlnSFJpZTnI+L7\nEbE/Il64yuWrZghtiFhfO443TXNg0ZsMzIYR29WOpdgwYnB2HEIbIuyLi7JN7bAvrhbbcDG2qR1T\nteMQ2hBhGy7KNrVjqnYcQhsibMNF2aZ29LLjWn8Eah9aysNN09y/9gcPrA1DasesDKXdbsdiDKHd\nQ2jDkNoxK0Npt9uxGENo9xDaMKR2zMpQ2u12LMYQ2j2ENgypHbMylHa7HYsxhHYPoQ1DasesDKXd\nbsfPYzmYMcYYY4wxxhhjzBbgH4GMMcYYY4wxxhhjtoBN/Qj0wIaeS4bQhojhtGNWhtJut2MxhtDu\nIbQhYjjtmJWhtNvtWIwhtHsIbYgYTjtmZSjtdjsWYwjtHkIbIobTjlkZSrvdjsUYQruH0IaI4bRj\nVobSbrdD2EhOIGOMMcYYY4wxxhizXiwHM8YYY4wxxhhjjNkC1vojUCnlo6WUJ0opT5ZSPr/G536p\nlHK5lPJd/NveUsrXSimnd/7/5jW042gp5W9LKY+VUh4tpfz+ptqyCNtsR9tw4edu3IY7z7QdF3vu\nxu1oGy783I3bcOeZtuNiz924HW3DhZ+7cRvuPNN2XOy5G7ejbbjwczduw51n2o6LPXfjdhyFDZum\nWcv/IuKaiDgTESci4rqI+HZEnFzTs/99RHwwIr6Lf/sfEfH5nePPR8R/X0M7DkXEB3eOb4iIUxFx\nchNtsR1tw221oe04DTvahuO3oe04DTvahuO3oe04DTvahuO3oe04DTuOwYbrHAz/NiL+Bp//MCL+\ncI3Pv10GwxMRcQiGemLtnR/xVxHxG0Noi+1oG26rDW3HadjRNhy/DW3HadjRNhy/DW3HadjRNhy/\nDW3HadhxiDZcpxzstog4j88Xdv5tUxxsmubizvFzEXFwnQ8vpdweER+IiIc23ZYZsR13sA2Xhn1x\nPmzHHWzDpWFfnA/bcQfbcGnYF+fDdtzBNlwa9sX5sB13GKoNnRg6Ipq3fo5r1vW8UsqeiPiLiPh0\n0zQ/2mRbpsQ6+842XA32xWlgXxw/9sVpYF8cP/bFaWBfHD/2xWlgX3yLdf4I9ExEHMXnIzv/tiku\nlVIORUTs/P/ldTy0lHJtvDUY/rxpmr/cZFvmZOvtaBsuHfvifGy9HW3DpWNfnI+tt6NtuHTsi/Ox\n9Xa0DZeOfXE+tt6OQ7fhOn8E+seIuKuUckcp5bqI+K2I+Moan698JSI+sXP8iXhLq7dSSiklIv40\nIr7XNM2fbLItC7DVdrQNV4J9cT622o624UqwL87HVtvRNlwJ9sX52Go72oYrwb44H1ttx1HYcJ0J\niCLiN+Ot7NhnIuK/rfG5X46IixHx03hLk/jJiNgXEV+PiNMR8b8jYu8a2vHv4q2wr+9ExLd2/veb\nm2iL7WgbbqsNbcdp2NE2HL8Nbcdp2NE2HL8Nbcdp2NE2HL8Nbcdp2HEMNiw7DTXGGGOMMcYYY4wx\nE8aJoY0xxhhjjDHGGGO2AP8IZIwxxhhjjDHGGLMF+EcgY4wxxhhjjDHGmC3APwIZY4wxxhhjjDHG\nbAH+EcgYY4wxxhhjjDFmC/CPQMYYY4wxxhhjjDFbgH8EMsYYY4wxxhhjjNkC/COQMcYYY4wxxhhj\nzBbw/wHDJ7r3Gw3GGgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e8805cdf98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get weights\n",
    "weights = sess.run(W)\n",
    "\n",
    "fig, ax = plt.subplots(1, 10, figsize=(20, 2))\n",
    "\n",
    "for digit in range(10):\n",
    "    ax[digit].imshow(weights[:,digit].reshape(28,28), cmap='gray')\n",
    "\n",
    "# Close session to finish\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The entire model, with the complete model definition, training, and evaluation (but minus the weights visualization), is below. Note the slight difference when calculating the softmax; this is done for numerical stability purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████| 1000/1000 [00:01<00:00, 572.50it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.9190000295639038\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tqdm import trange\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "# Import data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)\n",
    "\n",
    "# Create the model\n",
    "x = tf.placeholder(tf.float32, [None, 784])\n",
    "W = tf.Variable(tf.zeros([784, 10]))\n",
    "b = tf.Variable(tf.zeros([10]))\n",
    "y = tf.matmul(x, W) + b\n",
    "\n",
    "# Define loss and optimizer\n",
    "y_ = tf.placeholder(tf.float32, [None, 10])\n",
    "cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y))\n",
    "train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)\n",
    "\n",
    "# Create a Session object, initialize all variables\n",
    "sess = tf.Session()\n",
    "sess.run(tf.global_variables_initializer())\n",
    "\n",
    "# Train\n",
    "for _ in trange(1000):\n",
    "    batch_xs, batch_ys = mnist.train.next_batch(100)\n",
    "    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})\n",
    "\n",
    "# Test trained model\n",
    "correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "print('Test accuracy: {0}'.format(sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels})))\n",
    "\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: The accuracy from the full version directly above might return a slightly different test accuracy from the step-by-step version we first went through. This is because mnist.train.next_batch by default shuffles the order of the training data, so we're seeing the data in a different order.\n",
    "\n",
    "*Acknowledgment: Material adapted from the TensorFlow tutorial: https://www.tensorflow.org/get_started/*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}