day2/BuildingPerceptronsForClassification.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2203a0d",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import make_blobs\n",
    "\n",
    "%matplotlib notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b52d12da",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Classification with a Perceptron\n",
    "\n",
    "##### Version 0.2\n",
    "\n",
    "***\n",
    "\n",
    "By AA Miller (Northwestern/CIERA)  \n",
    "11 September 2023"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42c0f473",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Perceptrons are a type of articifical neuron. We will construct a basic neuron in pure python today and use it to classify data in a \"simple\" two class problem. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b21f531",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Consider the following image - what do you see? \n",
    "\n",
    "<img style=\"display: block; margin-left: auto; margin-right: auto\" src=\"images/number8.png\" width=\"450\" align=\"middle\">\n",
    "<div align=\"right\"> <font size=\"-3\">(data credit: MNIST) </font></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3807e243",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Without hesitation, I am certain that you were able to identify the above image as the number 8."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c15481c9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Breakout Problem 1**\n",
    "\n",
    "Using everything we have learned this week about machine learning – devise how to use the Random Forest algorithm to build a binary classifier to separate the number 8 from other handwritten digits. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db489e7a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "*Take a few minutes to discuss with your partner*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a9f3b01",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are many possible approaches - \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f19b8410",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "  -  build an algorithm that identifies \"circles\"   \n",
    "  $~~~~$(number 3 does not have fully closed circles)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "421544e1",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "  -  build an algorithm that indentifies lines  \n",
    "  $~~~~$(number 4 is only straight lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7999e8f0",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "  -  examine only the bottom half of the image  \n",
    "  $~~~~$(tops of 8 and 9 are similar, but bottoms different)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27e362a4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "At this point we only have a few features, they are all very complex to derive, and it isn't at all clear that we would successfully separate 8 and 0."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ce8e59a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In sum, we are on the road towards success.\n",
    "\n",
    "And yet,  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e48ee95b",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "when you looked at the image of the 8, you immediately recognized the contents of the image at the start of this lecture."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2893f0d8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We each have a super computer in our heads. One trained over many, many generations of evolution to immediately recognize what is right in front of our eyes. Constructing \"rules\" (especially very *precise* rules) to teach a computer to make the same recognition is extremely challenging. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cad167fb",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For complex tasks, like computer vision/classifying handwritten digits, we need a *new* type of machine learning relative to what we discussed earlier this week."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ad83225",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "(We will need deep learning to accomplish this task. We will develop these ideas over the next three days.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1162d104",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To start that process, I would like to introduce the *perception* – an artificial neuron."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5625118b",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "I'm no biologist, here's my (over-simplified) model of a biological neuron: a neuron receives multiple inputs (from other neurons or other signals present within the body), then weighs the relative information before becoming \"activated\" and further sending signals, or remaining dormant. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08c4cea2",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "(It is often said in popular literature that deep learning is designed to work like the human brain. \n",
    "\n",
    "This is very inaccurate, instead it is far more appropriate to say that the principles behind deep learning are inspired by our biological understanding of how the brain works. There is a lot that we do not understand about the brain, and it is worthwhile to remember this distinction.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c332ad3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Problem 1) The Perceptron"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c495c1a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "A perceptron takes several binary inputs, $x_1, x_2, \\ldots, x_n$, and, ultimately,  produces a single binary output. \n",
    "\n",
    "Each input has a relevant weight, $w_1, w_2, \\ldots, w_n$, which is multiplied by its corresponding input, before taking the sum of all the weighted inputs and comparing that to some threshold. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d48ac60e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If the weighted sum is greater than the threshold, then the output $= 1$ and the perceptron is \"activated.\"\n",
    "\n",
    "Otherwise the output $= 0$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "189b6ec7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here is a graphical representation of the perceptron:\n",
    "\n",
    "<img style=\"display: block; margin-left: auto; margin-right: auto\" src=\"images/perceptron.png\" width=\"650\" align=\"middle\">\n",
    "<div align=\"right\"> <font size=\"-3\">(credit: https://towardsdatascience.com/the-perceptron-3af34c84838c) </font></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da5d0781",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here is a mathematical representation of the perceptron: \n",
    "\n",
    "\n",
    "\n",
    "$$\\mathrm{output} = \\left\\{ \\begin{array}{lcr}\n",
    "0 & \\mathrm{if} \\; \\sum_n x_n w_n & \\le \\mathrm{threshold} \\\\\n",
    "1 & \\mathrm{if} \\; \\sum_n x_n w_n & > \\mathrm{threshold}\n",
    "\\end{array}\\right.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9af0321",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "A single perceptron can be used to answer questions that rely on many varied inputs. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9e21b8c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Consider the question: \"Should I walk the dog?\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26d182fe",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The answer to that question depends on other questions: \n",
    "  -  Is it raining? \n",
    "  -  Is the dog asleep?\n",
    "  -  Has is been more than 6 hours since the dog last went outside?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ff4cf9c",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Ultimately we need a binary output (to walk the dog, or not walk the dog) based on several binary inputs. In other words, we can use a perceptron to determine whether to walk the dog. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5bf4ab0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 1a**\n",
    "\n",
    "Write a function `walk_dog` that acts as a perceptron to decide whether or not you should walk the dog. \n",
    "\n",
    "The function should take three arugments, a tuple with the binary answer to your three questions, an optional tuple with the relative weights for each input (default = `(-2, -1, 5)`), and an optional threshold (default = `2.5`). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f232df7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def walk_dog( # complete\n",
    "    '''Perceptron to calculate whether we should walk the dog\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    questions : array-like, size = 3\n",
    "    weights : array-lik, optional (default = np.array([-2, -1, 5]))\n",
    "    threshold : float, optional (default = 2.5)\n",
    "        decision threshold for whether to walk the dog or not\n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    walk : bool\n",
    "        Boolean variable to indicate whether to walk the dog\n",
    "    '''\n",
    "    \n",
    "    # complete\n",
    "    # complete\n",
    "    # complete\n",
    "    # complete\n",
    "    # complete"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "520ea966",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 1b**\n",
    "\n",
    "Use the newly created perceptron to determine whether to walk the dog if: \n",
    "  1. it is raining outside\n",
    "  2. the dog is asleep\n",
    "  3. the dog last went out 2.5 hours ago\n",
    "\n",
    "*Hint* - recall that the inputs should be binary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2f54268",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "walk_dog(# complete"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38f4ef58",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 1c**\n",
    "\n",
    "After sleeping for four hours, it is still raining, but the dog wakes up (i.e., it has been 6.5 hr since the dog last went out). Should you walk the dog?\n",
    "\n",
    "*Hint* - recall that the inputs should be binary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b08c88d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "walk_dog(# complete"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8e1c2f3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Problem 2) Generic Perceptron"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81d10bf6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can simplify our representation of the perceptron by replacing the sum with a dot product, $w_n \\cdot x_n = \\sum_n w_n x_n$, and we can move the threshold to the other side of the inequality, which we will now refer to as the bias $b$. \n",
    "\n",
    "$$\\mathrm{output} = \\left\\{ \\begin{array}{lcr}\n",
    "0 & \\mathrm{if} \\; w_n \\cdot x_n + b & \\le 0 \\\\\n",
    "1 & \\mathrm{if} \\; w_n \\cdot x_n + b & > 0\n",
    "\\end{array}\\right.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b3ad13f",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "(in this notation, the bias can be thought of how easy it is for the neuron to be activated)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5defe893",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now, build a generic perceptron that can take any collection of input signals and weights, as well as a bias, to determine the binary output from the artificial neuron."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02326f48",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "(this will prove useful for more than just walking the dog)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3afca76e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 2a**\n",
    "\n",
    "Write a generic function `perceptron` that takes as input arrays called `signals` and `weights` as well as a float called `bias`. The function should return a boolean indicating whether or not the perceptron is \"activated\". "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6af71465",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def perceptron(# complete\n",
    "    '''Generic perceptron function\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    signals : array-like\n",
    "        the input signals for the perceptron\n",
    "        \n",
    "    weights : array-like\n",
    "        the weight applied to each input\n",
    "        \n",
    "    bias : float\n",
    "        the value required for activation\n",
    "        \n",
    "    Returns\n",
    "    -------\n",
    "    activated : bool\n",
    "        whether or not the perceptron is activated\n",
    "    '''\n",
    "    \n",
    "    # complete\n",
    "    # complete\n",
    "    # complete\n",
    "    # complete"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f98b383",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 2b** \n",
    "\n",
    "Is the perceptron activated if the signal = [2.3, 5.3, 1.2, 3.4], the weights = [-3, 2, 0.5, -1], and no bias (i.e., bias = 0)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01a84485",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "perceptron(# complete"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0eaed647",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 2c** \n",
    "\n",
    "What if the signal and weights do not change but the bias = -2? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25d2fe35",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "perceptron(# complete"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "752234ce",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Problem 3) Perceptrons for classification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92abc052",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Perceptrons can be used for binary classification problems!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec79f03b",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "(perhaps this is no surprise given that this session is about machine learning...)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2db543b4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To demonstrate how this works we will generate some synthetic two-dimensional data, but the principle can easily be scaled to an arbitrarily large number of dimensions. \n",
    "\n",
    "We use `scikit-learn` to simulate two classes in a 2d plane using the [`make_blobs()`](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_blobs.html) function. We will only include 30 samples so the data are easy to visualize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1bff508",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "X, y = make_blobs(n_samples=30, centers=2, n_features=2,\n",
    "                  center_box = (0,4), random_state=1885)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7724dc34",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "activated = y == 1\n",
    "ax.plot(X[activated,0], X[activated,1], 'o', \n",
    "        ms = 9, mew=2, mfc='None')\n",
    "ax.plot(X[~activated,0], X[~activated,1], '+', \n",
    "        ms=15, mew=2)\n",
    "ax.set_xlabel('X1', fontsize=15)\n",
    "ax.set_ylabel('X2', fontsize=15)\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2b610a9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How can we use a perceptron to classify this data? \n",
    "\n",
    "In this case we have two inputs, and thus two weights, plus the bias to determine whether the perceptron is activated (class = 1, the open circles in the previous plot).\n",
    "\n",
    "We also have 30 observations that we can use to train the algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdd2c68e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For a perceptron, training means updating the weights to better reflect the training data. \n",
    "\n",
    "Here's the pseudo-code: \n",
    "  1. Apply the perceptron to one of the data points\n",
    "  2. Adjust the weights and the bias if the perceptron makes an incorrect classification\n",
    "  3. Repeat this procedure over all N datapoints\n",
    "  4. Repeat this procedure for M iterations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb883ae9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How do we adjust the weights? For every sample we evaluate the model error (is the classification correct or not). We then adjust the weight to reduce the error for the following prediction. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b08e6ebe",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For a perceptron, these updates can be calculated simply as: \n",
    "\n",
    "$$w_\\mathrm{updated} = w_\\mathrm{current} + \\eta\\,\\, (y_\\mathrm{true} - y_\\mathrm{pred})\\,\\, x,$$\n",
    "\n",
    "$$b_\\mathrm{updated} = b_\\mathrm{current} + \\eta\\,\\, (y_\\mathrm{true} - y_\\mathrm{pred})$$\n",
    "\n",
    "where $w_\\mathrm{updated}$ is the new value for the weight, $w_\\mathrm{current}$ is the current value for the weight, $b_\\mathrm{updated}$ is the new value for the bias, $b_\\mathrm{current}$ is the current value for the bias, $\\eta$ is the called the *learning rate*, $x$ is the value of the input signal, and $(y_\\mathrm{true} - y_\\mathrm{pred})$ captures whether or not the classification was correct."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62f1484d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The learning rate is a small number that adjusts the weight in the direction of being more accurate. It is selected by the user, though familiar tricks like cross validation can be used to identify an optimal size. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9070049",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To train the perceptron, we need to decide the total number of iterations $M$ to pass through the training data. These iterations are called *epochs*. Within each epoch, we update the weights and bias to improve our predictions on the generated data. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d890f6c",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Updating the bias is similar to updating the weights, but we exclude the value of the input $x$ as this does not affect the bias."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b54dddb",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 3a**\n",
    "\n",
    "Write a function `train_perceptron` that accepts as input `X`, `y`, `weights`, `bias`, `epochs`, and `learning rate`. The function should train the perceptron for $M$ epochs. During each epoch, the weights and bias should be updated using the equation given above while looping over every source in the training set. \n",
    "\n",
    "*Hint* – it is useful to track the number of misclassifications that occur during each epoch. \n",
    "\n",
    "*Hint 2* – for this problem we only care about training, but if you eventually wanted to classify data with the perceptron then you would need to extract the weights and bias from the function, or, even better, write the perceptron as a class object that be trained and also classify (similar to scikit-learn). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "798326f7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def train_perceptron():\n",
    "    '''Train a perceptron to classify binary labels via numerical features\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    X : array-like\n",
    "        Feature array for the data, in the style of scikit-learn\n",
    "    y : array-like, type = bool\n",
    "        Label array for the data\n",
    "    weights : array-like\n",
    "        Weights for the input signals to the perceptron\n",
    "    bias : array-like\n",
    "        Bias value for the perceptron\n",
    "    epochs : int\n",
    "        Number of instances for training the perceptron\n",
    "    learning_rate : float\n",
    "        Relative step size for tuning the weights and bias\n",
    "    '''\n",
    "    # complete\n",
    "    # complete\n",
    "    # complete\n",
    "    # complete\n",
    "    # complete\n",
    "    # complete\n",
    "    # complete\n",
    "    # complete\n",
    "    # complete\n",
    "    # complete"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88e3ba8a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 3b**\n",
    "\n",
    "Train the perceptron. Use weights of [.1, 1], a bias of 0, train for 20 epochs, with a learning rate $\\eta = 0.005$. \n",
    "\n",
    "*Note* – as we will see below the perceptron is highly sensitive to the initial guess for weights and biases.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d8ebbe0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "train_perceptron( # complete"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe60c0ee",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We see that the accuracy slowly improves over the course of the 20 epochs. \n",
    "\n",
    "In other words, the machine... IT IS LEARNING."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2264fef8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Problem 3c**\n",
    "\n",
    "Adjust the weights, or bias, or number of epochs, or learning rate, or all of them, to see how the changes affect the output of the perceptron. \n",
    "\n",
    "What do you notice as these changes are made?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d2da17f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "train_perceptron( # complete"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6dc895e1",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "*write answer here*\n",
    "\n",
    "It is possible to build a perceptron that gets worse at classification. This can happen if the learning rate is extremely large, essentially flipping on or off the activation for every source during each epoch. \n",
    "\n",
    "It is also possible to build a perceptron that effectively never learns anything if the initial weights are extremely far from the optimal solution. The weights effectively define a line (or hyperplane in more than two dimensions) to separate the classes. If the line is very distant from the data itself, then the weights cannot be easily updated to improve the classification. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "856ddcca",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You have now built a perceptron. You have also seen it's limitations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d1ca1c3",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The strength of machine learning solutions lies in their ability to identify and capture non-linear structure within data. \n",
    "\n",
    "But the perceptron is almost too non-linear. For inputs that are close to the activation boundary, very small changes in the weights can rapidly lead to an extreme difference in outcome. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9beb1bda",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "It would be better if our adjustments led to more gradual changes, so that, if possible, the quality of the model improved during each epoch of the classifier. This can be achieved with a different model for our artificial neuron."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7252b747",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Consider, for instance, a neuron that is activated via the sigmoid function:\n",
    "\n",
    "$$\\sigma(z) \\equiv \\frac{1}{1 + e^{-z}}$$\n",
    "\n",
    "where $z$ is the previously defined activation for a neuron: $w \\cdot x - b$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12cd9a7e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can visually show the difference between the perceptron and sigmoid neurons.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b08006a5",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "ax. plot([-10, 0, 0, 10], [0,0,1,1], lw=3, label='Perceptron')\n",
    "xgrid = np.linspace(-10, 10, 1000)\n",
    "ax.plot(xgrid, 1/(1 + np.exp(-xgrid)), lw=2, label='Sigmoid')\n",
    "ax.set_xlim(-7.5, 7.5)\n",
    "ax.legend()\n",
    "ax.set_xlabel('input*weights - bias', fontsize=14)\n",
    "ax.set_ylabel('output', fontsize=14)\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2764f3c9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The above plot shows that the sigmoid function is a \"smooth\" version of the perceptron. We can exploit this smoothness to achieve better learning outcomes. Small changes in the weights and biases will produce small changes in the sigmoid function, enabling gradual improvement, which cannot be said for the perceptron near the region of activation.  \n",
    "\n",
    "\n",
    "This will prove important in the next lecture."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff188a96",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Challenge Problem\n",
    "\n",
    "Train a perceptron to classify the number 8 in the handwritten digits data set. \n",
    "\n",
    "*Hint* - you will need to convert the 2d data into a vector format."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16b0601e",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Appendix\n",
    "\n",
    "Functions to load plots shown in the notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78d03ef9",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.datasets import fetch_openml\n",
    "\n",
    "train_samples = 5000\n",
    "\n",
    "# Load data from https://www.openml.org/d/554\n",
    "X, y = fetch_openml('mnist_784', version=1, return_X_y=True, as_frame=False)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(4,4))\n",
    "\n",
    "ax.imshow(X[41].reshape(28,28)[3:-3,3:-3], \n",
    "          cmap='binary')\n",
    "ax.set_xticks([])\n",
    "ax.set_yticks([])\n",
    "fig.tight_layout()\n",
    "fig.savefig('./images/number8.png')"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "livereveal": {
   "height": 768,
   "scroll": true,
   "start_slideshow_at": "selected",
   "theme": "solarized",
   "width": 1024
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}