lecture_17_backpropagation_and_convolution.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Convolutional Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recall convolutional neural networks layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Dilated convolution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example of max-pooling\n",
    "<img src=\"images/ft/dilation_1.gif\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Weight shearing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "First way to make model invariant for different (accptable) aspects is weights sharing\n",
    "<br>\n",
    "Instead of using different weight for each neuron let's repeat them values time after time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/cnn/weight_sharing_1.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How can we achieve weights sharing?\n",
    "Ad restriction per layer to have copy of the weight or use other approach"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Convolutions on matrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What if want to have an input with depth for convolution:\n",
    "$$I \\in \\mathbb{R}^{h \\times w \\times c}$$\n",
    "<br>\n",
    "and the smaller weights matrix:\n",
    "  $$\\begin{align} K_i &= \\begin{pmatrix}\n",
    "           K_{11}, K_{12}, \\dots, K_{1k_2} \\\\\n",
    "           K_{21}, K_{22}, \\dots, K_{2k_2} \\\\\n",
    "           \\vdots \\\\\n",
    "           K_{k_11}, K_{k_12}, \\dots, K_{k_1k_2} \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We calculate convolution per slide with:\n",
    "$$\n",
    "\\begin{align}\n",
    "(I \\ast K)_{ij} &= \\sum_{m = 0}^{k_1 - 1} \\sum_{n = 0}^{k_2 - 1} \\sum_{c = 1}^{C} K_{m,n,c} \\cdot I_{i+m, j+n, c} + b \\tag {4}\n",
    "\\end{align}\n",
    "$$\n",
    "<br>\n",
    "Note that convolution does not have strides for channels and always produces one two dimensional matrix, flattens channels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recall deltas in backpropagation\n",
    "$$\n",
    "\\delta^l_j = \\sum_k \\frac{\\partial C}{\\partial z^{l+1}_k} \\frac{\\partial z^{l+1}_k}{\\partial z^l_j}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The weights sharing makes the backpropagation for convolution complicated. It's hard to remember and easy to get lost in deltas and gradients. So I found the excellent <a href=\"https://mc.ai/backpropagation-for-convolution-with-strides/\">blog</a> which makes it easy to understand. I'll follow the above blog."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Consider the input feature map as $\\mathbb{R}^{5 \\times 5} matrix$:\n",
    "$$\\begin{align} X &= \\begin{pmatrix}\n",
    "           x_{11}, x_{12}, x_{13}, x_{14}, x_{15} \\\\\n",
    "           x_{21}, x_{22}, x_{23}, x_{24}, x_{25} \\\\\n",
    "           x_{31}, x_{32}, x_{33}, x_{34}, x_{35} \\\\\n",
    "           x_{41}, x_{42}, x_{43}, x_{44}, x_{45} \\\\\n",
    "           x_{51}, x_{52}, x_{53}, x_{54}, x_{55} \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$\n",
    "<br>\n",
    "We assume that depth is $1$ for better understanding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Weights as $\\mathbb{R}^{3 \\times 3}$ matrix:\n",
    "$$\\begin{align} W &= \\begin{pmatrix}\n",
    "           w_{11}, w_{12}, w_{13} \\\\\n",
    "           w_{21}, w_{22}, w_{23} \\\\\n",
    "           w_{31}, w_{32}, w_{33} \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$\n",
    "<br>\n",
    "With stride $s = 2$ and zero padding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Then according the:\n",
    "$$\n",
    "H_o = \\frac{H - F_h + 2 P}{S_h} + 1\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "W_o = \\frac{W - F_w + 2 P}{S_w} + 1\n",
    "$$\n",
    "<br>\n",
    "formulas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Well have a:\n",
    "$$\n",
    "H_o = \\frac{5 - 3 + 2 \\cdot 0}{2} + 1 = 2 = W_o\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "And the output $\\mathbb{R}^{2 \\times 2} matrix$:\n",
    "$$\\begin{align} Y &= \\begin{pmatrix}\n",
    "           y_{11}, y_{12} \\\\\n",
    "           y_{21}, y_{22} \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now calculate forward propagation:\n",
    "\n",
    "$$\\begin{align} \\begin{pmatrix}\n",
    "           x_{11} \\cdot w_{11}, x_{12} \\cdot w_{12}, x_{13} \\cdot w_{13}, x_{14}, x_{15} \\\\\n",
    "           x_{21} \\cdot w_{21}, x_{22} \\cdot w_{22}, x_{23} \\cdot w_{23}, x_{24}, x_{25} \\\\\n",
    "           x_{31} \\cdot w_{31}, x_{32} \\cdot w_{32}, x_{33} \\cdot w_{33}, x_{34}, x_{35} \\\\\n",
    "           x_{41}, x_{42}, x_{43}, x_{44}, x_{45} \\\\\n",
    "           x_{51}, x_{52}, x_{53}, x_{54}, x_{55} \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The first convolution step will be:\n",
    "<table align=\"center\">\n",
    "    <tr>\n",
    "        <td>$x_{11} \\cdot w_{11}$</td>\n",
    "        <td>$x_{12} \\cdot w_{12}$</td>\n",
    "        <td>$x_{13} \\cdot w_{13}$</td>\n",
    "        <td>$x_{14}$</td>\n",
    "        <td>$x_{15}$</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>$x_{21} \\cdot w_{21}$</td>\n",
    "        <td>$x_{22} \\cdot w_{22}$</td>\n",
    "        <td>$x_{23} \\cdot w_{23}$</td>\n",
    "        <td>$x_{24}$</td>\n",
    "        <td>$x_{25}$</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>$x_{31} \\cdot w_{31}$</td>\n",
    "        <td>$x_{32} \\cdot w_{32}$</td>\n",
    "        <td>$x_{33} \\cdot w_{33}$</td>\n",
    "        <td>$x_{34}$</td>\n",
    "        <td>$x_{35}$</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>$x_{41}$</td>\n",
    "        <td>$x_{42}$</td>\n",
    "        <td>$x_{43}$</td>\n",
    "        <td>$x_{44}$</td>\n",
    "        <td>$x_{45}$</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>$x_{51}$</td>\n",
    "        <td>$x_{52}$</td>\n",
    "        <td>$x_{53}$</td>\n",
    "        <td>$x_{54}$</td>\n",
    "        <td>$x_{55}$</td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For the second step:\n",
    "<table align=\"center\">\n",
    "    <tr>\n",
    "        <td>$x_{11}$</td>\n",
    "        <td>$x_{12}$</td>\n",
    "        <td>$x_{13} \\cdot w_{11}$</td>\n",
    "        <td>$x_{14} \\cdot w_{12}$</td>\n",
    "        <td>$x_{15} \\cdot w_{13}$</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>$x_{21}$</td>\n",
    "        <td>$x_{22}$</td>\n",
    "        <td>$x_{23} \\cdot w_{21}$</td>\n",
    "        <td>$x_{24} \\cdot w_{22}$</td>\n",
    "        <td>$x_{25} \\cdot w_{23}$</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>$x_{31}$</td>\n",
    "        <td>$x_{32}$</td>\n",
    "        <td>$x_{33} \\cdot w_{31}$</td>\n",
    "        <td>$x_{34} \\cdot w_{32}$</td>\n",
    "        <td>$x_{35} \\cdot w_{33}$</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>$x_{41}$</td>\n",
    "        <td>$x_{42}$</td>\n",
    "        <td>$x_{43}$</td>\n",
    "        <td>$x_{44}$</td>\n",
    "        <td>$x_{45}$</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>$x_{51}$</td>\n",
    "        <td>$x_{52}$</td>\n",
    "        <td>$x_{53}$</td>\n",
    "        <td>$x_{54}$</td>\n",
    "        <td>$x_{55}$</td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the third step:\n",
    "<table align=\"center\">\n",
    "    <tr>\n",
    "        <td>$x_{11}$</td>\n",
    "        <td>$x_{12}$</td>\n",
    "        <td>$x_{13}$</td>\n",
    "        <td>$x_{14}$</td>\n",
    "        <td>$x_{15}$</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>$x_{21}$</td>\n",
    "        <td>$x_{22}$</td>\n",
    "        <td>$x_{23}$</td>\n",
    "        <td>$x_{24}$</td>\n",
    "        <td>$x_{25}$</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>$x_{31} \\cdot w_{11}$</td>\n",
    "        <td>$x_{32} \\cdot w_{12}$</td>\n",
    "        <td>$x_{33} \\cdot w_{13}$</td>\n",
    "        <td>$x_{34}$</td>\n",
    "        <td>$x_{35}$</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>$x_{41} \\cdot w_{21}$</td>\n",
    "        <td>$x_{42} \\cdot w_{22}$</td>\n",
    "        <td>$x_{43} \\cdot w_{23}$</td>\n",
    "        <td>$x_{44}$</td>\n",
    "        <td>$x_{45}$</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>$x_{51} \\cdot w_{31}$</td>\n",
    "        <td>$x_{52} \\cdot w_{32}$</td>\n",
    "        <td>$x_{53} \\cdot w_{33}$</td>\n",
    "        <td>$x_{54}$</td>\n",
    "        <td>$x_{55}$</td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "And for the fourth step:\n",
    "<table align=\"center\">\n",
    "    <tr>\n",
    "        <td>$x_{11}$</td>\n",
    "        <td>$x_{12}$</td>\n",
    "        <td>$x_{13}$</td>\n",
    "        <td>$x_{14}$</td>\n",
    "        <td>$x_{15}$</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>$x_{21}$</td>\n",
    "        <td>$x_{22}$</td>\n",
    "        <td>$x_{23}$</td>\n",
    "        <td>$x_{24}$</td>\n",
    "        <td>$x_{25}$</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>$x_{31}$</td>\n",
    "        <td>$x_{32}$</td>\n",
    "        <td>$x_{33} \\cdot w_{11}$</td>\n",
    "        <td>$x_{34} \\cdot w_{12}$</td>\n",
    "        <td>$x_{35} \\cdot w_{13}$</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>$x_{41}$</td>\n",
    "        <td>$x_{42}$</td>\n",
    "        <td>$x_{43} \\cdot w_{21}$</td>\n",
    "        <td>$x_{44} \\cdot w_{22}$</td>\n",
    "        <td>$x_{45} \\cdot w_{23}$</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>$x_{51}$</td>\n",
    "        <td>$x_{52}$</td>\n",
    "        <td>$x_{53} \\cdot w_{31}$</td>\n",
    "        <td>$x_{54} \\cdot w_{32}$</td>\n",
    "        <td>$x_{55} \\cdot w_{33}$</td>\n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We need to calculate step backward with respect to gradient of layer before activations:\n",
    "<br>\n",
    "From:\n",
    "$$\\begin{align} \\nabla Y &= \\begin{pmatrix}\n",
    "           \\frac{\\partial C}{ \\partial y_{11}}, \\frac{\\partial C}{\\partial y_{12}} \\\\\n",
    "           \\frac{\\partial C}{\\partial y_{21}}, \\frac{\\partial C}{\\partial y_{22}} \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We should be able to calculate:\n",
    "$$\\begin{align} \\nabla X &= \\begin{pmatrix}\n",
    "           \\frac{\\partial C}{ \\partial x_{11}}, \\frac{\\partial C}{ \\partial x_{12}}, \\frac{\\partial C}{ \\partial x_{13}}, \\frac{\\partial C}{ \\partial x_{14}}, \\frac{\\partial C}{ \\partial x_{15}} \\\\\n",
    "           \\frac{\\partial C}{ \\partial x_{21}}, \\frac{\\partial C}{ \\partial x_{22}}, \\frac{\\partial C}{ \\partial x_{23}}, \\frac{\\partial C}{ \\partial x_{24}}, \\frac{\\partial C}{ \\partial x_{25}} \\\\\n",
    "           \\frac{\\partial C}{ \\partial x_{31}}, \\frac{\\partial C}{ \\partial x_{32}}, \\frac{\\partial C}{ \\partial x_{33}}, \\frac{\\partial C}{ \\partial x_{34}}, \\frac{\\partial C}{ \\partial x_{35}} \\\\\n",
    "           \\frac{\\partial C}{ \\partial x_{41}}, \\frac{\\partial C}{ \\partial x_{42}}, \\frac{\\partial C}{ \\partial x_{43}}, \\frac{\\partial C}{ \\partial x_{44}}, \\frac{\\partial C}{ \\partial x_{45}} \\\\\n",
    "           \\frac{\\partial C}{ \\partial x_{51}}, \\frac{\\partial C}{ \\partial x_{52}}, \\frac{\\partial C}{ \\partial x_{53}}, \\frac{\\partial C}{ \\partial x_{54}}, \\frac{\\partial C}{ \\partial x_{55}} \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Each $x_{i,j}$ contributes in one or several results in $Y$ and according the chain rule:\n",
    "$$\n",
    "\\frac{\\partial C}{\\partial x_{m, n}} = \\sum_{i, j}\\frac{\\partial C}{\\partial y_{i, j}} \\cdot \\frac{\\partial y_{i, j}}{\\partial x_{m, n}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So we have:\n",
    "$$\n",
    "y_{11} = x_{11} \\cdot w_{11} + x_{12} \\cdot w_{12} + x_{13} \\cdot w_{13} + x_{21} \\cdot w_{21} + x_{22} \\cdot w_{22} + x_{23} \\cdot w_{23} + x_{31} \\cdot w_{31} + x_{32} \\cdot w_{32} + x_{33} \\cdot w_{33} \\\\\n",
    "y_{12} = x_{13} \\cdot w_{11} + x_{14} \\cdot w_{12} + x_{15} \\cdot w_{13} + x_{23} \\cdot w_{21} + x_{24} \\cdot w_{22} + x_{25} \\cdot w_{23} + x_{33} \\cdot w_{31} + x_{34} \\cdot w_{32} + x_{35} \\cdot w_{33} \\\\\n",
    "y_{21} = x_{31} \\cdot w_{11} + x_{32} \\cdot w_{12} + x_{33} \\cdot w_{13} + x_{41} \\cdot w_{21} + x_{42} \\cdot w_{22} + x_{43} \\cdot w_{23} + x_{51} \\cdot w_{31} + x_{52} \\cdot w_{32} + x_{53} \\cdot w_{33} \\\\\n",
    "y_{22} = x_{33} \\cdot w_{11} + x_{34} \\cdot w_{12} + x_{35} \\cdot w_{13} + x_{43} \\cdot w_{21} + x_{44} \\cdot w_{22} + x_{45} \\cdot w_{23} + x_{53} \\cdot w_{31} + x_{54} \\cdot w_{32} + x_{55} \\cdot w_{33}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's calculate the gradient for the first element $x_{11}$:\n",
    "<br>\n",
    "$$\n",
    "y_{11} = \\pmb{x_{11}} \\cdot w_{11} + x_{12} \\cdot w_{12} + x_{13} \\cdot w_{13} + x_{21} \\cdot w_{21} + x_{22} \\cdot w_{22} + x_{23} \\cdot w_{23} + x_{31} \\cdot w_{31} + x_{32} \\cdot w_{32} + x_{33} \\cdot w_{33} \\\\\n",
    "$$\n",
    "<br>\n",
    "Here $x_{11}$ only contributes in $y_{11}$ and therefore:\n",
    "$$\n",
    "\\frac{\\partial C}{\\partial x_{1, 1}} = \\frac{\\partial C}{\\partial y_{11}}\\frac {\\partial y_{11}} {\\partial {x_{11}}} = \\frac{\\partial C}{\\partial y_{11}} \\cdot w_{11}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now, let's consider $x_{12}$:\n",
    "<br>\n",
    "$$\n",
    "y_{11} = x_{11} \\cdot w_{11} + \\pmb{x_{12}} \\cdot w_{12} + x_{13} \\cdot w_{13} + x_{21} \\cdot w_{21} + x_{22} \\cdot w_{22} + x_{23} \\cdot w_{23} + x_{31} \\cdot w_{31} + x_{32} \\cdot w_{32} + x_{33} \\cdot w_{33} \\\\\n",
    "$$\n",
    "<br>\n",
    "Here $x_{12}$ also only contributes in $y_{11}$ and therefore:\n",
    "$$\n",
    "\\frac{\\partial C}{\\partial x_{12}} = \\frac{\\partial C}{\\partial y_{11}}\\frac {\\partial y_{11}} {\\partial {x_{12}}} = \\frac{\\partial C}{\\partial y_{12}} \\cdot w_{12}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The $x_{13}$ contributes in the $y_{11}$ and $y_{12}$:\n",
    "<br>\n",
    "$$\n",
    "y_{11} = x_{11} \\cdot w_{11} + x_{12} \\cdot w_{12} + \\pmb{x_{13}} \\cdot w_{13} + x_{21} \\cdot w_{21} + x_{22} \\cdot w_{22} + x_{23} \\cdot w_{23} + x_{31} \\cdot w_{31} + x_{32} \\cdot w_{32} + x_{33} \\cdot w_{33} \\\\\n",
    "y_{12} = \\pmb{x_{13}} \\cdot w_{11} + x_{14} \\cdot w_{12} + x_{15} \\cdot w_{13} + x_{23} \\cdot w_{21} + x_{24} \\cdot w_{22} + x_{25} \\cdot w_{23} + x_{33} \\cdot w_{31} + x_{34} \\cdot w_{32} + x_{35} \\cdot w_{33} \\\\\n",
    "$$\n",
    "<br>\n",
    "And thus:\n",
    "$$\n",
    "\\frac{\\partial C}{\\partial x_{13}} = \\frac{\\partial C}{\\partial y_{11}}\\frac {\\partial y_{11}}{\\partial x_{13}} + \\frac{\\partial C}{\\partial y_{12}} \\frac {\\partial y_{12}} {\\partial x_{13}} = \\frac{\\partial C}{\\partial y_{11}} \\cdot w_{12} +  \\frac{\\partial C}{\\partial y_{12}} \\cdot w_{11}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The input $x_{33}$ contributes in the all four outputs:\n",
    "$$\n",
    "y_{11} = x_{11} \\cdot w_{11} + x_{12} \\cdot w_{12} + x_{13} \\cdot w_{13} + x_{21} \\cdot w_{21} + x_{22} \\cdot w_{22} + x_{23} \\cdot w_{23} + x_{31} \\cdot w_{31} + x_{32} \\cdot w_{32} + \\pmb{x_{33}} \\cdot w_{33} \\\\\n",
    "y_{12} = x_{13} \\cdot w_{11} + x_{14} \\cdot w_{12} + x_{15} \\cdot w_{13} + x_{23} \\cdot w_{21} + x_{24} \\cdot w_{22} + x_{25} \\cdot w_{23} + \\pmb{x_{33}} \\cdot w_{31} + x_{34} \\cdot w_{32} + x_{35} \\cdot w_{33} \\\\\n",
    "y_{21} = x_{31} \\cdot w_{11} + x_{32} \\cdot w_{12} + \\pmb{x_{33}} \\cdot w_{13} + x_{41} \\cdot w_{21} + x_{42} \\cdot w_{22} + x_{43} \\cdot w_{23} + x_{51} \\cdot w_{31} + x_{52} \\cdot w_{32} + x_{53} \\cdot w_{33} \\\\\n",
    "y_{22} = \\pmb{x_{33}} \\cdot w_{11} + x_{34} \\cdot w_{12} + x_{35} \\cdot w_{13} + x_{43} \\cdot w_{21} + x_{44} \\cdot w_{22} + x_{45} \\cdot w_{23} + x_{53} \\cdot w_{31} + x_{54} \\cdot w_{32} + x_{55} \\cdot w_{33}\n",
    "$$\n",
    "<br>\n",
    "and therefore:\n",
    "$$\n",
    "\\frac{\\partial C}{\\partial x_{33}} = \\frac{\\partial C}{\\partial y_{11}}\\frac {\\partial y_{11}}{\\partial x_{33}} + \\frac{\\partial C}{\\partial y_{12}} \\frac {\\partial y_{12}} {\\partial x_{33}} + \\frac{\\partial C}{\\partial y_{21}}\\frac {\\partial y_{21}}{\\partial x_{33}} + \\frac{\\partial C}{\\partial y_{22}} \\frac {\\partial y_{22}} {\\partial x_{33}} = \\frac{\\partial C}{\\partial y_{11}} \\cdot w_{33} +  \\frac{\\partial C}{\\partial y_{12}} \\cdot w_{31} + \\frac{\\partial C}{\\partial y_{21}} \\cdot w_{13} +  \\frac{\\partial C}{\\partial y_{22}} \\cdot w_{11}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Finally we can run dilated convolution:\n",
    "<br>\n",
    "$$\\begin{align} \\nabla Y &= \\begin{pmatrix}\n",
    "           0, 0, 0, 0, 0, 0, 0 \\\\\n",
    "           0, 0, 0, 0, 0, 0, 0 \\\\\n",
    "           0, 0, \\frac{\\partial C}{ \\partial y_{11}}, 0, \\frac{\\partial C}{\\partial y_{12}}, 0, 0 \\\\\n",
    "           0, 0, \\frac{\\partial C}{\\partial y_{21}}, 0, \\frac{\\partial C}{\\partial y_{22}}, 0, 0 \\\\\n",
    "           0, 0, 0, 0, 0, 0, 0 \\\\\n",
    "           0, 0, 0, 0, 0, 0, 0 \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$\n",
    "<br>\n",
    "with the:\n",
    "$$\\begin{align} W &= \\begin{pmatrix}\n",
    "           w_{33}, w_{32}, w_{31} \\\\\n",
    "           w_{22}, w_{22}, w_{21} \\\\\n",
    "           w_{13}, w_{12}, w_{11} \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The above is equivalent to the:\n",
    "<br>\n",
    "$$\\begin{align} \\nabla Y &= \\begin{pmatrix}\n",
    "           0, 0, 0, 0, 0, 0, 0 \\\\\n",
    "           0, 0, 0, 0, 0, 0, 0 \\\\\n",
    "           0, 0, \\frac{\\partial C}{ \\partial y_{11}}, \\frac{\\partial C}{\\partial y_{12}}, 0, 0 \\\\\n",
    "           0, 0, \\frac{\\partial C}{\\partial y_{21}}, \\frac{\\partial C}{\\partial y_{22}}, 0, 0 \\\\\n",
    "           0, 0, 0, 0, 0, 0, 0 \\\\\n",
    "           0, 0, 0, 0, 0, 0, 0 \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$\n",
    "<br>\n",
    "with the:\n",
    "$$\\begin{align} W &= \\begin{pmatrix}\n",
    "           w_{33}, 0, w_{32}, 0,  w_{31} \\\\\n",
    "           w_{22}, 0, w_{22}, 0,  w_{21} \\\\\n",
    "           w_{13}, 0, w_{12}, 0, w_{11} \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So we have a dilated convolution with flipped kernel on $180$ degrees.\n",
    "<br>\n",
    "Padding the next layer gradient tensor with $(k1-1, k2-1)$ and dilate with $s - 1$ zeros, will backpropagate error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Task\n",
    "\n",
    "Do the same with channels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here is the example of convolution, where input has three channels:\n",
    "<img src=\"images/cnn/convolution_4.gif\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Hint\n",
    "Here picture is the similar:\n",
    "$$\n",
    "\\frac{\\partial C}{\\partial x_{33}} = \\sum_{d=1}^D (\\frac{\\partial C}{\\partial y_{11}}\\frac {\\partial y_{11}}{\\partial x_{33d}} + \\frac{\\partial C}{\\partial y_{12}} \\frac {\\partial y_{12}} {\\partial x_{33d}} + \\frac{\\partial C}{\\partial y_{21}}\\frac {\\partial y_{21}}{\\partial x_{33d}} + \\frac{\\partial C}{\\partial y_{22}} \\frac {\\partial y_{22}} {\\partial x_{33d}}) = \\sum_{d=1}^D (\\frac{\\partial C}{\\partial y_{11}} \\cdot w_{33d} +  \\frac{\\partial C}{\\partial y_{12}} \\cdot w_{31d} + \\frac{\\partial C}{\\partial y_{21}} \\cdot w_{13d} +  \\frac{\\partial C}{\\partial y_{22}} \\cdot w_{11d})\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If we change $x_{ij}$ with the $a^l_{ij}$ as activation for the previous layer and $y_{ij}$ with the $z^{l+1}_{ij}$ and recall that:\n",
    "<br>\n",
    "$$\n",
    "\\delta^l_j = \\frac{\\partial C}{\\partial z^l_j}\n",
    "$$\n",
    "Then we'll get the:\n",
    "$$\\begin{align} \\nabla Y &= \\begin{pmatrix}\n",
    "           \\delta^{l+1}_{11}, \\delta^{l+1}_{12} \\\\\n",
    "           \\delta^{l+1}_{21}, \\delta^{l+1}_{22} \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can conclude that we have a backpropagation with the $\\delta^{l+1}$ kernel flipping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Pooling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Pooling layer is not learnable, for instance after max-pooling only maximum values affect the error:\n",
    "<img src=\"images/cnn/pooling_1.png\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For max-pooling layer backpropagarion only considers maximum values per sliding window, only the maximum values have influence on the error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For average pooling we can propagate:\n",
    "$$\n",
    "\\frac{1}{K_1 \\times K_2}\n",
    "$$\n",
    "<br>\n",
    "error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Feture map visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Convolutional neural networks has a hierarchical structure. According to that fact, we can imagine, that learning happens hierarchically.\n",
    "- First layers detect near by edges\n",
    "- Middle layers more complex edges and color maps\n",
    "- Last layers detect object patterns\n",
    "- Then linear classifiers distinguish object according to the extracted features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's visualize weights of one for the first layers filters / kernels of different models:\n",
    "<img src=\"images/ft/weights_vis_1.png\" height=\"800\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If we visualize weights layer by layer:\n",
    "<img src=\"images/ft/weights_vis_2.png\" height=\"800\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If we visualize learned feature maps activations as images, we can observe that fact:\n",
    "<img src=\"images/ft/features_1.png\" height=\"800\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here we see different feature maps visualization:\n",
    "<img src=\"images/ft/features_2.jpg\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Different models extract features in a different hierarchy but pattern is preserved:\n",
    "<img src=\"images/ft/features_3.jpg\" height=\"200\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here we can observe features for different images:\n",
    "<img src=\"images/ft/features_4.jpg\" height=\"800\" width=\"800\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Feature extraction / embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's take one of the pre-trained (on ImageNet) models, VGG, Inception,  ResNet,  etc and remove all the last layers before convolutional layers:\n",
    "- For VGG16 remove last two fully connected layers\n",
    "- For Inception and ResNet remove all the layer after adaptive (global) average pooling\n",
    "<br>\n",
    "So our model generates vector from the image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torchvision.models import resnet50, resnet34, vgg16\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "net = vgg16(pretrained=True)\n",
    "net"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "?? net"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "model = nn.Sequential(*list(net.children())[:-1])\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "x = torch.randn(1, 3, 399, 399)\n",
    "with torch.no_grad():\n",
    "    y1 = net(x)\n",
    "    y2 = model(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "y1.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y2.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y2 = torch.flatten(y2, 1)\n",
    "y2.size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So we have $2048$ dimensional vectors, we can run our model on the our dataset of images and generate $2048$ dimensional vectors.\n",
    "$$\n",
    "f: \\mathbb{R}^{3 \\times H \\times W} \\mapsto \\mathbb{R}^d\n",
    "$$\n",
    "<br>\n",
    "Our model maps each $C \\times H \\times W$ (they might be different for adaptive average pooling) dimensional image to the fixed $d$ dimensional vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Vectors have \"distance\" property.\n",
    "<br>\n",
    "If we store this vectors and run K-nearest neighbor search we can observe that similarity search is working even if our dataset was not used during the training.\n",
    "<br>\n",
    "Note: Search results depend on model and domain of training set and dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Similarity search examples:\n",
    "<img src=\"images/ft/sim_1.png\" height=\"1000\" width=\"1000\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Dimensionality reduction and clustering:\n",
    "<img src=\"images/ft/sim_2.png\" height=\"1000\" width=\"1000\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Transfer-learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can see that first layers extract essential features which are pretty similar for all images. Second layers extract more complex features and last layers more domain-specific features\n",
    "<br>\n",
    "Can we use this information for different task. Would it be enough information, enough features if use it pre-trained model on the different dataset?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "With the following approach:\n",
    "- We extract features from the images with the pre-trained model\n",
    "- Train different model with this features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Turns out that this approach works and it's called transfer earning.\n",
    "For transfer learning we should consider the following:\n",
    "- Is the model is trained on the similar domain\n",
    "- Is the model trained on the enough data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The state-of-the art result achieved with model trained on ImageNet classification task\n",
    "- It has different and well-distributed images\n",
    "- More precise labeled\n",
    "<br>\n",
    "Or it has enough images to extract \"all-possible\" features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are several approaches:\n",
    "- Use extracted features and train different model\n",
    "- Freeze the weights and train only classifier\n",
    "- Fine-tune whole model with discriminative learning rates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "First approach needs pre-extraction of the feature vectors and training different model on them\n",
    "- Extract features\n",
    "- Train different classifier (SVM, RF, GB) on them"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For the second approach we put our layers on top the model and train it:\n",
    "- Put custom layers on model\n",
    "- Freeze feature extraction layers weights\n",
    "- Train custom layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "model_fn = nn.Sequential(*list(model.children()) + [nn.Linear(25088, 500), nn.Dropout(p=0.3), \n",
    "                                                    nn.Linear(500, 20)])\n",
    "model_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For third approach, we put our layers on top the model and train it with different learning rate:\n",
    "- Put custom layers on model\n",
    "- Train full model using larger learning rate for last layers, smaller maybe $ \\frac{1}{100}$ for the middle layers and $\\frac{1}{1000}$ for the first layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Pre-trained classifier also used for different tasks\n",
    "- Segmentation\n",
    "- Detection\n",
    "- Image search / metric learning\n",
    "- Auto-encoders\n",
    "- GAN\n",
    "- etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Transfer learning also works for other tasks, such as NLP models "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/cnn/questions.jpg\" height=\"600\" width=\"600\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Thank you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}