Exercise-03.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST handwritten digits classification with nearest neighbors \n",
    "\n",
    "In this notebook, we'll use [nearest-neighbor classifiers](http://scikit-learn.org/stable/modules/neighbors.html#nearest-neighbors-classification) to classify MNIST digits using scikit-learn (version 0.20 or later required).\n",
    "\n",
    "First, the needed imports. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "from pml_utils import get_mnist, show_failures\n",
    "\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import neighbors, __version__\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix, classification_report\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set()\n",
    "\n",
    "from packaging.version import Version\n",
    "assert(Version(__version__) >= Version(\"0.20\")), \"Version >= 0.20 of sklearn is required.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we load the MNIST data. First time we need to download the data, which can take a while."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train, X_test, y_test = get_mnist('MNIST')\n",
    "\n",
    "print('MNIST data loaded: train:',len(X_train),'test:',len(X_test))\n",
    "print('X_train:', X_train.shape)\n",
    "print('y_train:', y_train.shape)\n",
    "print('X_test', X_test.shape)\n",
    "print('y_test', y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training data (`X_train`) is a matrix of size (60000, 784), i.e. it consists of 60000 digits expressed as 784 sized vectors (28x28 images flattened to 1D). `y_train` is a 60000-dimensional vector containing the correct classes (\"0\", \"1\", ..., \"9\") for each training digit.\n",
    "\n",
    "Let's take a closer look. Here are the first 10 training digits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "pltsize=1\n",
    "plt.figure(figsize=(10*pltsize, pltsize))\n",
    "\n",
    "for i in range(10):\n",
    "    plt.subplot(1,10,i+1)\n",
    "    plt.axis('off')\n",
    "    plt.imshow(X_train[i,:].reshape(28, 28), cmap=\"gray\")\n",
    "    plt.title('Class: '+y_train[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1-NN classifier\n",
    "\n",
    "### Initialization\n",
    "\n",
    "Let's create first a 1-NN classifier.  Note that with nearest-neighbor classifiers there is no internal (parameterized) model and therefore no learning required.  Instead, calling the `fit()` function simply stores the samples of the training data in a suitable data structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "n_neighbors = 1\n",
    "clf_nn = neighbors.KNeighborsClassifier(n_neighbors)\n",
    "clf_nn.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference\n",
    "\n",
    "And try to classify some test samples with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "pred_nn = clf_nn.predict(X_test[:200,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We observe that the classifier is rather slow, and classifying the whole test set would take quite some time. What is the reason for this?\n",
    "\n",
    "The accuracy of the classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "print('Predicted', len(pred_nn), 'digits with accuracy:',\n",
    "      accuracy_score(y_test[:len(pred_nn)], pred_nn))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Faster 1-NN classifier\n",
    "\n",
    "### Initialization\n",
    "\n",
    "One way to make our 1-NN classifier faster is to use less training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "n_neighbors = 1\n",
    "n_data = 1024\n",
    "clf_nn_fast = neighbors.KNeighborsClassifier(n_neighbors)\n",
    "clf_nn_fast.fit(X_train[:n_data,:], y_train[:n_data])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inference\n",
    "\n",
    "Now we can use the classifier created with reduced data to classify our whole test set in a reasonable amount of time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "pred_nn_fast = clf_nn_fast.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The classification accuracy is however now not as good:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "print('Predicted', len(pred_nn_fast), 'digits with accuracy:',\n",
    "      accuracy_score(y_test, pred_nn_fast))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Confusion matrix\n",
    "\n",
    "We can compute the confusion matrix to see which digits get mixed the most:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "labels=[str(i) for i in range(10)]\n",
    "print('Confusion matrix (rows: true classes; columns: predicted classes):'); print()\n",
    "cm=confusion_matrix(y_test, pred_nn_fast, labels=labels)\n",
    "print(cm); print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotted as an image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.matshow(cm, cmap=plt.cm.gray)\n",
    "plt.xticks(range(10))\n",
    "plt.yticks(range(10))\n",
    "plt.grid(None)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy, precision and recall\n",
    "\n",
    "Classification accuracy for each class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "for i,j in enumerate(cm.diagonal()/cm.sum(axis=1)): print(\"%d: %.4f\" % (i,j))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Precision and recall for each class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "print(classification_report(y_test, pred_nn_fast, labels=labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Failure analysis\n",
    "\n",
    "We can also inspect the results in more detail. Let's use the `show_failures()` helper function (defined in `pml_utils.py`) to show the wrongly classified test digits.\n",
    "\n",
    "The helper function is defined as:\n",
    "\n",
    "```\n",
    "show_failures(predictions, y_test, X_test, trueclass=None, predictedclass=None, maxtoshow=10)\n",
    "```\n",
    "\n",
    "where:\n",
    "- `predictions` is a vector with the predicted classes for each test set image\n",
    "- `y_test` the _correct_ classes for the test set images\n",
    "- `X_test` the test set images\n",
    "- `trueclass` can be set to show only images for a given correct (true) class\n",
    "- `predictedclass` can be set to show only images which were predicted as a given class\n",
    "- `maxtoshow` specifies how many items to show\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "show_failures(pred_nn_fast, y_test, X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `show_failures()` to inspect failures in more detail. For example:\n",
    "\n",
    "* show failures in which the true class was \"5\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "show_failures(pred_nn_fast, y_test, X_test, trueclass='5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* show failures in which the prediction was \"0\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "show_failures(pred_nn_fast, y_test, X_test, predictedclass='0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* show failures in which the true class was \"0\" and the prediction was \"2\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "show_failures(pred_nn_fast, y_test, X_test, trueclass='0', predictedclass='2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can observe that the classifier makes rather \"easy\" mistakes, and there might thus be room for improvement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model tuning\n",
    "\n",
    "Try to improve the accuracy of the nearest-neighbor classifier while preserving a reasonable runtime to classify the whole test set. Things to try include using more than one neighbor (with or without weights) or increasing the amount of training data.  See the documentation for [KNeighborsClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html#sklearn-neighbors-kneighborsclassifier).\n",
    "\n",
    "See also http://scikit-learn.org/stable/modules/neighbors.html#nearest-neighbors-classification for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}