02A_CNNs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Convolutional Neural Networks (CNNs) in TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Representing images digitally\n",
    "\n",
    "While convolutional neural networks (CNNs) see a wide variety of uses, they were originally designed for images, and CNNs are still most commonly used for vision-related tasks.\n",
    "For today, we'll primarily be focusing on CNNs used on images.\n",
    "Before we dive into convolutions and neural networks, it's worth prefacing with how images are represented by a computer, as this understanding will inform some of our design choices.\n",
    "\n",
    "In yesterday's lecture, we saw an example of a digitized MNIST handwritten digit.\n",
    "Specifically, we represent it as an $H \\times W$ table, with the value of each element storing the intensity of the corresponding pixel.\n",
    "\n",
    "<img src=\"Figures/mnist_digital.PNG\" alt=\"mnist_digital\" style=\"width: 600px;\"/>\n",
    "\n",
    "With a 2D representation as above, we for the most part can only efficiently represent grayscale images.\n",
    "What if we want color?\n",
    "There are many schemes for color, but one of the most common ones is the [RGB color model](https://en.wikipedia.org/wiki/RGB_color_model).\n",
    "In such a system, we store 3 tables of pixel intensities (each called a *channel*), one each for the colors red, green, and blue (hence RGB), resulting in an $H \\times W \\times 3$ tensor.\n",
    "Pixel values for a particular channel indicate how much of the corresponding color the image has at a particular location.\n",
    "\n",
    "Let's load an image and see this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You may need to install matplotlib and imageio first:\n",
    "#\n",
    "# pip install matplotlib\n",
    "# pip install imageio\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the image we loaded has height and width of $620 \\times 1175$, with 3 channels corresponding to RGB.\n",
    "\n",
    "We can easily slice out and view individual color channels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Red channel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Blue channel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While we have so far considered only 3 channel RGB images, there are many settings in which we may consider a different number of channels.\n",
    "For example, [hyperspectral imaging](https://en.wikipedia.org/wiki/Hyperspectral_imaging) uses a wide range of the electromagnetic spectrum to characterize a scene.\n",
    "Such modalities may have hundreds of channels or more.\n",
    "Additionally, we'll soon see that certain intermediate represenations in a CNN can be considered images with many channels.\n",
    "However, we'll focus much of today's discussion to 1 channel grayscale and 3 channel RGB images as our inputs, as is most commonly the case in computer vision."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolutions\n",
    "Convolutional neural networks are a class of neural networks that have convolutional layers.\n",
    "CNNs are particularly effective for data that have spatial structures and correlations (e.g. images).\n",
    "We'll focus on CNNs applied to images in this tutorial.\n",
    "Recall from yesterday, a multilayer perceptron (MLP) is entirely composed of fully connected layers, which are each a matrix multiply operation (and addition of a bias) followed by a non-linearity (e.g. sigmoid, ReLU). \n",
    "A convolutional layer is similar, except the matrix multiply operation is replaced with a convolution operation (in practice a cross-correlation). \n",
    "Note that a CNN need not be entirely composed of convolutional layers; in fact, many popular CNN architectures end in fully connected layers.\n",
    "\n",
    "As before, since we're building neural networks, let's start by loading TensorFlow. We'll find NumPy useful as well, so we'll also import that here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Review: Fully connected layer\n",
    "In a fully connected layer, the input $x \\in \\mathbb R^{M \\times C_{in}}$ is a vector (or, rather a batch of vectors), where $M$ is the minibatch size and $C_{in}$ is the dimensionality of the input. \n",
    "We first matrix multiply the input $x$ by a weight matrix $W$.\n",
    "This weight matrix has dimensions $W \\in \\mathbb R^{C_{in} \\times C_{out}}$, where $C_{out}$ is the number of output units.\n",
    "We then add a bias for each output, which we do by adding $b \\in \\mathbb{R}^{C_{out}}$.\n",
    "The output $y \\in \\mathbb{R}^{M \\times C_{out}}$ of the fully connected layer then:\n",
    "\n",
    "\\begin{align*}\n",
    "y = \\text{ReLU}(x W + b)\n",
    "\\end{align*}\n",
    "\n",
    "Remember, the values of $W$ and $b$ are variables that we are trying to learn for our model. \n",
    "Below we have a visualization of what the matrix operation looks like (bias term and activation function omitted).\n",
    "\n",
    "<img src=\"Figures/fully_connected.png\" alt=\"fully_connected\" style=\"width: 700px;\"/>\n",
    "<center>Figure adapted from *[A Multithreaded CGRA for Convolutional Neural Network Processing](https://www.researchgate.net/publication/318025612_A_Multithreaded_CGRA_for_Convolutional_Neural_Network_Processing?_sg=MwYs2n43RZGhaLyzNQwxOhIkb4VfQVEavAttULBtZr3Ax4vsebFlihBGwom6soQAHqEb5kwS2Q)*</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create flat input vector\n",
    "\n",
    "\n",
    "# Create weight matrix variable\n",
    "\n",
    "\n",
    "# Create bias variable\n",
    "\n",
    "\n",
    "# Apply fully connected layer\n",
    "\n",
    "\n",
    "# Print output shape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convolutional layer\n",
    "In a convolutional layer, we convolve with a convolutional kernel (aka filter), which we also call $W$. \n",
    "Unlike the input of a fully connected layer, which is $x \\in \\mathbb R^{M\\times C_{in}}$, the dimensionality of an image input is 4D: $x \\in \\mathbb R^{M \\times H_{in} \\times W_{in} \\times C_{in}}$.\n",
    "Another difference is that also unlike the 2-D weight matrix for fully connected layers, the kernel is 4-D with dimensions $W \\in \\mathbb R^{H_K \\times W_K \\times C_{in} \\times C_{out}}$, where $H_K$ and $W_K$ are the kernel height and weight, respectively.\n",
    "Common choices for $H_K$ and $W_K$ are $H_K = W_K = 3$ or $H_K = W_K = 5$.\n",
    "Convolving the input with the kernel and adding a bias then gives an output $y \\in \\mathbb R^{M \\times H_{in} \\times W_{in} \\times C_{out}}$:\n",
    "\n",
    "\\begin{align*}\n",
    "y = \\text{ReLU}(x*W + b)\n",
    "\\end{align*}\n",
    "\n",
    "In the context of CNNs, the output $y$ is often referred to as feature maps. Again, the goal is to learn $W$ and $b$ for our model.\n",
    "\n",
    "If you're having trouble visualizing this operation in 4D, it's easier to think about for a single member of the minibatch, one convolutional kernel at a time. \n",
    "Consider a stack of $C_{out}$ kernels, each of which are 3D ($H_K \\times W_K \\times C_{in}$). \n",
    "This 3D volume is then slid across the input (which is also 3D: $H_{in} \\times W_{in} \\times C_{in}$) in the two spatial dimensions (along $H_{in}$ and $W_{in}$). \n",
    "The outputs of the multiplication of the kernel and the input at every location creates a single feature map that is $H_{in} \\times W_{in}$ (assuming no pooling or striding; we'll get to that later). \n",
    "Stacking the feature maps generated by each kernel gives the 3D output $H_{in} \\times W_{in} \\times C_{out}$.\n",
    "Repeat the process for all $M$ inputs, and we get a 4D output $M \\times H_{in} \\times W_{in} \\times C_{out}$.\n",
    "\n",
    "<img src=\"Figures/single_W_conv.png\" alt=\"single_W_conv\" style=\"width: 400px;\"/>\n",
    "<center>Figure from *[Convolutional Neural Network. Brilliant.org.](https://brilliant.org/wiki/convolutional-neural-network/)*</center>\n",
    "\n",
    "\n",
    "A couple more things to consider for convolutions:\n",
    "- An additional argument for the convolution is the *stride*, which controls the spacing at which we multiply the input with the kernel during the convolution operation. \n",
    "The convolutional operator, from its signal processing roots, by default considers a stride length of 1 in all dimensions, but in some situations we would like to consider strides more than 1 (or even less than 1). \n",
    "More on this later.\n",
    "- In the context of signal processing, convolutions usually result in outputs that are larger than the input size, which results from when the kernel \"hangs off the edge\" of the input on both sides. \n",
    "This might not always be desirable.\n",
    "We can control this by controlling the padding of the input.\n",
    "Typically, we use \"SAME\" padding, which ensures the output has the same spatial dimensions as the input (assuming stride of 1); this makes it easier for us to keep track of what the size of our model is.\n",
    "\n",
    "Let's implement this convolution operator in code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create image input placeholder\n",
    "\n",
    "\n",
    "# Create convolutional kernel variable\n",
    "\n",
    "\n",
    "# Create bias variable\n",
    "\n",
    "\n",
    "# Apply convolutional layer\n",
    "\n",
    "\n",
    "# Print output shape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like in a MLP, we can stack multiple of these convolutional layers. \n",
    "In the *Representing Images Digitally* section, we briefly mentioned considering images with channels more than 3.\n",
    "Observe that the input to the second layer (i.e. the output of the first layer) can be viewed as an \"image\" with $C_{out}$ channels.\n",
    "Instead of each channel representing a color content though, each channel effectively represents how much the original input image activated a particular convolutional kernel.\n",
    "Given $C_{out}$ kernels that are each $H_K \\times W_K \\times C_{in}$, this results in $C_{out}$ channels for the output of the convolution.\n",
    "\n",
    "Note that we need to change the dimensions of the convolutional kernel such that its input channels matches the number of output channels of the previous layer.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2nd layer variables\n",
    "\n",
    "\n",
    "# Apply 2nd convolutional layer\n",
    "\n",
    "\n",
    "# Print output shape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, we typically perform these convolution operations many times. Popular architectures for image analysis today are commonly 100+ layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reshaping\n",
    "\n",
    "You'll commonly finding yourself needing to reshape tensors while building CNNs.\n",
    "The TensorFlow function for doing so is `tf.reshape()`. \n",
    "Anyone familiar with `numpy` will find it very similar to `np.reshape()`.\n",
    "Importantly, it must be possible to rearrange the input into the shape of the output (ie the number of values must be the same).\n",
    "You can optionally leave out one of the dimensions by using a `-1`, which tells `reshape` to infer the missing dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get an idea of why reshaping is need in a CNN, let's look at a diagram of a simple CNN.\n",
    "\n",
    "<img src=\"Figures/mnist_cnn_ex.png\" alt=\"mnist_cnn_ex\" style=\"width: 700px;\"/>\n",
    "<center>Figure from [*Getting started with PyTorch for Deep Learning (Part 3: Neural Network basics). Code to Light.*](https://codetolight.wordpress.com/2017/11/29/getting-started-with-pytorch-for-deep-learning-part-3-neural-network-basics/)</center>\n",
    "\n",
    "First of all, the CNN expects a 4D input, with the dimensions corresponding to `[batch, height, width, channel]`.\n",
    "Your data probably won't come in this format, so you'll likely have to reshape it yourself.\n",
    "For example, remember MNIST comes in flat vectors (`[,784]`). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reshape flat input image into a 4D batched image input\n",
    "\n",
    "\n",
    "# Print input shape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CNN architectures also commonly contain fully connected layers or a softmax, as we're often interested in classification.\n",
    "Both of these expect 2D inputs with dimensions `[batch, dim]`, so you have to \"flatten\" a CNN's 4D output to 2D.\n",
    "For example, to flatten the convolutional feature maps we created earlier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Flatten convolutional feature maps into a vector\n",
    "\n",
    "# Print output shape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pooling and striding\n",
    "\n",
    "Almost all CNN architectures incorporate either pooling or striding. This is done for a number of reasons, including:\n",
    "- Dimensionality reduction: pooling and striding operations reduces computational complexity by shrinking the number of values passed to the next layer.\n",
    "For example, a 2x2 maxpool reduces the size of the feature maps by a factor of 4.\n",
    "- Translational invariance: Oftentimes in computer vision, it is desirable feature of our model that shifting the input by a few pixels doesn't change the output. Pooling and striding reduces sensitivity to exact pixel locations.\n",
    "- Increasing receptive field: by summarizing a window with a single value, subsequent convolutional kernels are seeing a wider swath of the original input image. For example, a max pool on some input followed by a 3x3 convolution results in a kernel \"seeing\" 6x6 instead of 3x3.\n",
    "\n",
    "#### Pooling\n",
    "The two most common forms of pooling are max pooling and average pooling. \n",
    "Both reduce values within a window to a single value, on a per feature map basis.\n",
    "Max pooling takes the maximum value of the window as the output value; average pooling takes the mean.\n",
    "\n",
    "<img src=\"Figures/avg_vs_max_pool.JPG\" alt=\"avg_vs_max\" style=\"width: 400px;\"/>\n",
    "<center>Figure from [*A deeper understanding of NNets (Part 1) — CNNs. Towards Data Science.*](https://towardsdatascience.com/a-deeper-understanding-of-nnets-part-1-cnns-263a6e3ac61)</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Taking the output we've been working with so far, first print its current size\n",
    "print(\"Shape of conv2 feature maps before pooling: \")\n",
    "\n",
    "# Max pool and then print new shape\n",
    "print(\"Shape of conv2 feature maps after max pooling: \")\n",
    "\n",
    "# Average pool and then print new shape\n",
    "print(\"Shape of conv2 feature maps after avg pooling: \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can confirm that the two types of pooling behave how we'd expect:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Recreate values in pooling figure and make it 4D\n",
    "feature_map_fig = tf.constant(np.array([[1,3,2,4],\n",
    "                                        [1,3,2,4],\n",
    "                                        [5,7,6,8],\n",
    "                                        [5,7,6,8]]), dtype=tf.float32)\n",
    "print(\"Feature map shape pre-pooling: \")\n",
    "\n",
    "# Maxpool\n",
    "print(\"Feature map shape post-max pooling: \")\n",
    "\n",
    "# Avgpool\n",
    "print(\"Feature map shape post-avg pooling: \")\n",
    "\n",
    "# Check that values match figure\n",
    "print(\"\\nMax pool\")\n",
    "print(\"\\nAvg pool\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Striding\n",
    "One might expect that pixels in an image have high correlation with neighboring pixels, so we can save computation by skipping positions while sliding the convolutional kernel. \n",
    "By default, a CNN slides across the input one pixel at a time, which we call a stride of 1.\n",
    "By instead striding by 2, we skip calculating 75% of the values of the output feature map, which yields a feature map that's half the size in each spatial direction.\n",
    "Note, while pooling is an operation done after the convolution, striding is part of the convolution operation itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Since striding is part of the convolution operation, we'll start with the feature maps before the 2nd convolution\n",
    "print(\"Shape of conv1 feature maps: \")\n",
    "\n",
    "# Apply 2nd convolutional layer, with striding of 2\n",
    "\n",
    "\n",
    "# Print output shape\n",
    "print(\"Shape of conv2 feature maps with stride of 2: \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clear previous ops from the graph\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Higher level APIs\n",
    "\n",
    "So far, we've primarily been building neural networks with the TensorFlow Core API, the lowest level of abstraction.\n",
    "This gives the user the finest amount of control over the TensorFlow graph, as well as providing the clearest picture of what's going on under the hood.\n",
    "This can be important as you're learning concepts and the various frameworks, and sometimes the flexibility is necessary if you're trying to build something outside of the standard models.\n",
    "\n",
    "However, most of the time, we do find ourselves repeating the same fairly standard lines of code, which can slow us down. \n",
    "Worse, it clutters up our code unnecessarily and introduces room for bugs and typos.\n",
    "And finally, as researchers or engineers, we would like to spend most of our time thinking on the highest levels of abstractions: I want to add a convolution layer here, then a fully connected there, etc.\n",
    "Having to code all the small details are distractions that can detract from our ability to translate ideas into code.\n",
    "\n",
    "For this reason, several people have built high level APIs on top of TensorFlow, with Keras, tf.layers, and TF-Slim* being notable examples.\n",
    "We'll briefly touch on TF-Slim and tf.layers here, but we encourage you to look into the options and decide which best suit your needs and preferences.\n",
    "\n",
    "**Note: TF-Slim is no longer officially supported by the TensorFlow team*\n",
    "\n",
    "#### VGG-16\n",
    "To illustrate what I mean about levels of abstraction, let's consider [VGG-16](https://arxiv.org/abs/1409.1556). This style of architecture came in 2nd during the 2014 ImageNet Large Scale Visual Recognition Challenge and is famous for its simplicity and depth. The model looks like this:\n",
    "\n",
    "<img src=\"Figures/vgg16.png\" alt=\"vgg16\" style=\"width: 600px;\"/>\n",
    "<center>Figure from [*Deep CNN and Weak Supervision Learning for visual recognition*](https://blog.heuritech.com/2016/02/29/a-brief-report-of-the-heuritech-deep-learning-meetup-5/)</center>\n",
    "\n",
    "The architecture is pretty straight-forward: simply stack multiple 3x3 convolutional filters one after another, interleave with 2x2 maxpools, double the number of convolutional filters after each maxpool, flatten, and finish with fully connected layers. A couple ideas behind this model:\n",
    "\n",
    "- Instead of using larger filters, VGG notes that the receptive field of two stacked layers of 3x3 filters is 5x5, and with 3 layers, 7x7. Using 3x3's allows VGG to insert additional non-linearities and requires fewer weight parameters to learn.\n",
    "\n",
    "- Doubling the width of the network every time the features are spatially downsampled (maxpooled) gives the model more representational capacity while achieving spatial compression.\n",
    "\n",
    "In code, setting up the computation graph for just feedforwad part of VGG-16 with the TensorFlow Core API is kind of a lot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\*\n",
    "\n",
    "\\*\n",
    "\n",
    "\\*\n",
    "\n",
    "\\*\n",
    "\n",
    "\\*\n",
    "\n",
    "Spoilers below\n",
    "\n",
    "\\*\n",
    "\n",
    "\\*\n",
    "\n",
    "\\*\n",
    "\n",
    "\\*\n",
    "\n",
    "\\*\n",
    "\n",
    "Don't peek yet\n",
    "\n",
    "\\*\n",
    "\n",
    "\\*\n",
    "\n",
    "\\*\n",
    "\n",
    "\\*\n",
    "\n",
    "\\*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the model\n",
    "with tf.name_scope('conv1_1') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 3, 64], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(images, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[64], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv1 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "with tf.name_scope('conv1_2') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 64, 64], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(conv1, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[64], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv1 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "pool1 = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', name='pool1')\n",
    "    \n",
    "with tf.name_scope('conv2_1') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 64, 128], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(pool1, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[128], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv2 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "with tf.name_scope('conv2_2') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 128, 128], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(conv2, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[128], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv2 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "pool2 = tf.nn.max_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', name='pool2')\n",
    "    \n",
    "with tf.name_scope('conv3_1') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 128, 256], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(pool2, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[256], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv3 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "with tf.name_scope('conv3_2') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 256, 256], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(conv3, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[256], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv3 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "with tf.name_scope('conv3_3') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 256, 256], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(conv3, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[256], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv3 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "pool3 = tf.nn.max_pool(conv3, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', name='pool3')\n",
    "    \n",
    "with tf.name_scope('conv4_1') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 256, 512], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(pool3, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[512], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv4 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "with tf.name_scope('conv4_2') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 512, 512], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(conv4, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[512], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv4 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "with tf.name_scope('conv4_3') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 512, 512], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(conv4, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[512], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv4 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "pool4 = tf.nn.max_pool(conv4, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', name='pool4')\n",
    "\n",
    "with tf.name_scope('conv5_1') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 512, 512], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(pool4, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[512], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv5 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "with tf.name_scope('conv5_2') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 512, 512], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(conv5, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[512], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv5 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "with tf.name_scope('conv5_3') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 512, 512], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(conv5, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[512], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv5 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "pool5 = tf.nn.max_pool(conv5, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', name='pool5')\n",
    "    \n",
    "with tf.name_scope('fc_6') as scope:\n",
    "    flat = tf.reshape(pool5, [-1, 7*7*512])\n",
    "    weights = tf.Variable(tf.truncated_normal([7*7*512, 4096], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    mat = tf.matmul(flat, weights)\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[4096], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(mat, biases)\n",
    "    fc6 = tf.nn.relu(bias, name=scope)\n",
    "    fc6_drop = tf.nn.dropout(fc6, keep_prob=0.5, name='dropout')\n",
    "\n",
    "with tf.name_scope('fc_7') as scope:\n",
    "    weights = tf.Variable(tf.truncated_normal([4096, 4096], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    mat = tf.matmul(fc6, weights)\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[4096], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(mat, biases)\n",
    "    fc7 = tf.nn.relu(bias, name=scope)\n",
    "    fc7_drop = tf.nn.dropout(fc7, keep_prob=0.5, name='dropout')\n",
    "    \n",
    "with tf.name_scope('fc_8') as scope:\n",
    "    weights = tf.Variable(tf.truncated_normal([4096, 1000], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    mat = tf.matmul(fc7, weights)\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[1000], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(mat, biases)\n",
    "\n",
    "predictions = bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TF-Slim\n",
    "Enter TF-Slim. The same VGG-16 model can be expressed as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "slim = tf.contrib.slim\n",
    "\n",
    "# Define the model:\n",
    "with slim.arg_scope([slim.conv2d, slim.fully_connected],\n",
    "                    activation_fn=tf.nn.relu,\n",
    "                    weights_initializer=tf.truncated_normal_initializer(0.0, 0.01),\n",
    "                    weights_regularizer=slim.l2_regularizer(0.0005)):\n",
    "    net = slim.repeat(images, 2, slim.conv2d, 64, [3, 3], scope='conv1')\n",
    "    net = slim.max_pool2d(net, [2, 2], scope='pool1')\n",
    "    net = slim.repeat(net, 2, slim.conv2d, 128, [3, 3], scope='conv2')\n",
    "    net = slim.max_pool2d(net, [2, 2], scope='pool2')\n",
    "    net = slim.repeat(net, 3, slim.conv2d, 256, [3, 3], scope='conv3')\n",
    "    net = slim.max_pool2d(net, [2, 2], scope='pool3')\n",
    "    net = slim.repeat(net, 3, slim.conv2d, 512, [3, 3], scope='conv4')\n",
    "    net = slim.max_pool2d(net, [2, 2], scope='pool4')\n",
    "    net = slim.repeat(net, 3, slim.conv2d, 512, [3, 3], scope='conv5')\n",
    "    net = slim.max_pool2d(net, [2, 2], scope='pool5')\n",
    "    net = slim.fully_connected(net, 4096, scope='fc6')\n",
    "    net = slim.dropout(net, 0.5, scope='dropout6')\n",
    "    net = slim.fully_connected(net, 4096, scope='fc7')\n",
    "    net = slim.dropout(net, 0.5, scope='dropout7')\n",
    "    net = slim.fully_connected(net, 1000, activation_fn=None, scope='fc8')\n",
    "\n",
    "predictions = net"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much cleaner. For the TF-Slim version, it's much more obvious what the network is doing, writing it is faster, and typos and bugs are much less likely.\n",
    "\n",
    "Things to notice:\n",
    "\n",
    "- Weight and bias variables for every layer are automatically generated and tracked. Also, the \"in_channel\" parameter for determining weight dimension is automatically inferred from the input. This allows you to focus on what layers you want to add to the model, without worrying as much about boilerplate code. \n",
    "\n",
    "- The repeat() function allows you to add the same layer multiple times. In terms of variable scoping, repeat() will add \"_#\" to the scope to distinguish the layers, so we'll still have layers of scope \"`conv1_1`, `conv1_2`, `conv2_1`, etc...\".\n",
    "\n",
    "- The non-linear activation function (here: ReLU) is wrapped directly into the layer. In more advanced architectures with batch normalization, that's included as well.\n",
    "\n",
    "- With slim.argscope(), we're able to specify defaults for common parameter arguments, such as the type of activation function or weights_initializer. Of course, these defaults can still be overridden in any individual layer, as demonstrated in the finally fully connected layer (fc8).\n",
    "\n",
    "If you're reusing one of the famous architectures (like VGG-16), TF-Slim already has them defined, so it becomes even easier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### tf.layers\n",
    "TF-Slim is built on top of tf.layers, and given that TF-Slim is considered deprecated while tf.layers is still supported, you may find it useful to at least be familiar with Layers.\n",
    "Layers tends to be pretty popular in the research community, as it provides a nice balance of abstraction and flexibility. \n",
    "Usage is pretty similar between Slim and Layers; in fact, many TF-Slim functions are just wrappers of tf.layers functions.\n",
    "Some caution is warranted, as some of the default arguments may be different; be sure to check the documentation!\n",
    "Let's take a look:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\kevin_000\\Anaconda3\\envs\\tensorflow\\lib\\site-packages\\h5py\\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Core:\n",
      "conv output shape: (?, 28, 28, 32)\n",
      "maxpool output shape: (?, 14, 14, 32)\n",
      "flatten output shape: (?, 6272)\n",
      "fully connected output shape: (?, 256)\n",
      "\n",
      "Layers:\n",
      "\n",
      "Slim:\n"
     ]
    }
   ],
   "source": [
    "# Clear default graph\n",
    "\n",
    "\n",
    "# Input\n",
    "\n",
    "\n",
    "\n",
    "# TF Core version\n",
    "print(\"Core:\")\n",
    "with tf.name_scope('conv1_core') as scope:\n",
    "    kernel = tf.Variable(tf.truncated_normal([3, 3, 1, 32], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    conv = tf.nn.conv2d(x, kernel, [1, 1, 1, 1], padding='SAME')\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[32], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(conv, biases)\n",
    "    conv1 = tf.nn.relu(bias, name=scope)\n",
    "\n",
    "pool1 = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME', name='pool1_core')\n",
    "flat = tf.reshape(pool1, [-1, 14*14*32])\n",
    "\n",
    "with tf.name_scope('fc_1') as scope:\n",
    "    weights = tf.Variable(tf.truncated_normal([14*14*32, 256], dtype=tf.float32, stddev=1e-1), name='weights')\n",
    "    mat = tf.matmul(flat, weights)\n",
    "    biases = tf.Variable(tf.constant(0.0, shape=[256], dtype=tf.float32), trainable=True, name='biases')\n",
    "    bias = tf.nn.bias_add(mat, biases)\n",
    "    fc1 = tf.nn.relu(bias, name=scope)\n",
    "    \n",
    "print(\"conv output shape: {0}\".format(conv1.shape))\n",
    "print(\"maxpool output shape: {0}\".format(pool1.shape))\n",
    "print(\"flatten output shape: {0}\".format(flat.shape))\n",
    "print(\"fully connected output shape: {0}\".format(fc1.shape))\n",
    "\n",
    "\n",
    "# Layers version\n",
    "print(\"\\nLayers:\")\n",
    "\n",
    "\n",
    "# print(\"conv output shape: {0}\".format(conv_layers.shape))\n",
    "# print(\"maxpool output shape: {0}\".format(mp_layers.shape))\n",
    "# print(\"flatten output shape: {0}\".format(flat_layers.shape))\n",
    "# print(\"fully connected output shape: {0}\".format(fc_layers.shape))\n",
    "\n",
    "\n",
    "# Slim version\n",
    "slim = tf.contrib.slim\n",
    "\n",
    "print(\"\\nSlim:\")\n",
    "\n",
    "\n",
    "# print(\"conv output shape: {0}\".format(conv_slim.shape))\n",
    "# print(\"maxpool output shape: {0}\".format(mp_slim.shape))\n",
    "# print(\"flatten output shape: {0}\".format(flat_slim.shape))\n",
    "# print(\"fully connected output shape: {0}\".format(fc_slim.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-trained models\n",
    "\n",
    "TF-Slim provides weights pre-trained on the ImageNet dataset available for [download](https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models). First a quick tutorial on saving and restoring models:\n",
    "\n",
    "#### Saving and restoring\n",
    "\n",
    "One of the nice features of modern machine learning frameworks is the ability to save model parameters in a clean way. While this may not have been a big deal for the MNIST logistic regression model because training only took a few seconds, it's easy to see why you wouldn't want to have to re-train a model from scratch every time you wanted to do inference or make a small change if training takes days or weeks.\n",
    "\n",
    "TensorFlow provides this functionality with its [Saver()](https://www.tensorflow.org/programmers_guide/variables#saving_and_restoring) class. While I just said that saving the weights for the MNIST logistic regression model isn't necessary because of how it is easy to train, let's do it anyway for illustrative purposes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tqdm import trange\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "# Import data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)\n",
    "\n",
    "# Create the model\n",
    "x = tf.placeholder(tf.float32, [None, 784], name='x')\n",
    "W = tf.Variable(tf.zeros([784, 10]), name='W')\n",
    "b = tf.Variable(tf.zeros([10]), name='b')\n",
    "y = tf.nn.bias_add(tf.matmul(x, W), b, name='y')\n",
    "\n",
    "# Define loss and optimizer\n",
    "y_ = tf.placeholder(tf.float32, [None, 10], name='y_')\n",
    "cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y))\n",
    "train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)\n",
    "\n",
    "# Variable Initializer\n",
    "init_op = tf.global_variables_initializer()\n",
    "\n",
    "# Create a Saver object for saving weights\n",
    "\n",
    "\n",
    "# Create a Session object, initialize all variables\n",
    "sess = tf.Session()\n",
    "sess.run(init_op)\n",
    "\n",
    "# Train\n",
    "for _ in trange(1000):\n",
    "    batch_xs, batch_ys = mnist.train.next_batch(100)\n",
    "    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})\n",
    "    \n",
    "# Save model\n",
    "\n",
    "print(\"Model saved in file: \" )\n",
    "    \n",
    "# Test trained model\n",
    "correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "print('Test accuracy: {0}'.format(sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels})))\n",
    "\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, the differences from what we worked with yesterday:\n",
    "\n",
    "- In lines 9-12, 15, there are now 'names' properties attached to certain ops and variables of the graph. There are many reasons to do this, but here, it will help us identify which variables are which when restoring. \n",
    "- In line 23, we create a Saver() object, and in line 35, we save the variables of the model to a checkpoint file. This will create a series of files containing our saved model.\n",
    "\n",
    "Otherwise, the code is more or less the same.\n",
    "\n",
    "To restore the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tqdm import trange\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "# Import data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)\n",
    "\n",
    "# Create a Session object, initialize all variables\n",
    "sess = tf.Session()\n",
    "\n",
    "# Restore weights\n",
    "\n",
    "print(\"Model restored.\")\n",
    "\n",
    "# Define graph\n",
    "\n",
    "    \n",
    "# Test trained model\n",
    "correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "print('Test accuracy: {0}'.format(sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels})))\n",
    "\n",
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importantly, notice that we didn't have to retrain the model. Instead, the graph and all variable values were loaded directly from our checkpoint files. In this example, this probably takes just as long, but for more complex models, the utility of saving/restoring is immense.\n",
    "\n",
    "### TF-Slim model zoo\n",
    "\n",
    "One of the biggest and most surprising unintended benefits of the ImageNet competition was deep networks' transfer learning properties: CNNs trained on ImageNet classification could be re-used as general purpose feature extractors for other tasks, such as object detection. Training on ImageNet is very intensive and expensive in both time and computation, and requires a good deal of set-up. As such, the availability of weights already pre-trained on ImageNet has significantly accelerated and democratized deep learning research.\n",
    "\n",
    "Pre-trained models of several famous architectures are listed in the TF Slim portion of the [TensorFlow repository](https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models). Also included are the papers that proposed them and their respective performances on ImageNet. Side note: remember though that accuracy is not the only consideration when picking a network; memory and speed are important to keep in mind as well.\n",
    "\n",
    "Each entry has a link that allows you to download the checkpoint file of the pre-trained network. Alternatively, you can download the weights as part of your program. A tutorial can be found [here](https://github.com/tensorflow/models/blob/master/research/slim/slim_walkthrough.ipynb), but the general idea:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import dataset_utils\n",
    "import tensorflow as tf\n",
    "\n",
    "url = \"http://download.tensorflow.org/models/vgg_16_2016_08_28.tar.gz\"\n",
    "checkpoints_dir = './checkpoints'\n",
    "\n",
    "if not tf.gfile.Exists(checkpoints_dir):\n",
    "    tf.gfile.MakeDirs(checkpoints_dir)\n",
    "\n",
    "dataset_utils.download_and_uncompress_tarball(url, checkpoints_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import tensorflow as tf\n",
    "from nets import vgg\n",
    "\n",
    "slim = tf.contrib.slim\n",
    "\n",
    "# Load images\n",
    "images = ...\n",
    "\n",
    "# Pre-process\n",
    "processed_images = ...\n",
    "\n",
    "# Create the model, use the default arg scope to configure the batch norm parameters.\n",
    "with slim.arg_scope(vgg.vgg_arg_scope()):\n",
    "    logits, _ = vgg.vgg_16(processed_images, num_classes=1000, is_training=False)\n",
    "    \n",
    "probabilities = tf.nn.softmax(logits)\n",
    "\n",
    "# Load checkpoint values\n",
    "init_fn = slim.assign_from_checkpoint_fn(\n",
    "    os.path.join(checkpoints_dir, 'vgg_16.ckpt'),\n",
    "    slim.get_model_variables('vgg_16'))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}