3. Facial Keypoint Detection, Complete Pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Face and Facial Keypoint detection\n",
    "\n",
    "After you've trained a neural network to detect facial keypoints, you can then apply this network to *any* image that includes faces. The neural network expects a Tensor of a certain size as input and, so, to detect any face, you'll first have to do some pre-processing.\n",
    "\n",
    "1. Detect all the faces in an image using a face detector (we'll be using a Haar Cascade detector in this notebook).\n",
    "2. Pre-process those face images so that they are grayscale, and transformed to a Tensor of the input size that your net expects. This step will be similar to the `data_transform` you created and applied in Notebook 2, whose job was tp rescale, normalize, and turn any image into a Tensor to be accepted as input to your CNN.\n",
    "3. Use your trained model to detect facial keypoints on the image.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next python cell we load in required libraries for this section of the project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Select an image \n",
    "\n",
    "Select an image to perform facial keypoint detection on; you can select any image of faces in the `images/` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import cv2\n",
    "# load in color image for face detection\n",
    "image = cv2.imread('images/obamas.jpg')\n",
    "\n",
    "# switch red and blue color channels \n",
    "# --> by default OpenCV assumes BLUE comes first, not RED as in many images\n",
    "image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n",
    "\n",
    "# plot the image\n",
    "fig = plt.figure(figsize=(9,9))\n",
    "plt.imshow(image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Detect all faces in an image\n",
    "\n",
    "Next, you'll use one of OpenCV's pre-trained Haar Cascade classifiers, all of which can be found in the `detector_architectures/` directory, to find any faces in your selected image.\n",
    "\n",
    "In the code below, we loop over each face in the original image and draw a red square on each face (in a copy of the original image, so as not to modify the original). You can even [add eye detections](https://docs.opencv.org/3.4.1/d7/d8b/tutorial_py_face_detection.html) as an *optional* exercise in using Haar detectors.\n",
    "\n",
    "An example of face detection on a variety of images is shown below.\n",
    "\n",
    "<img src='images/haar_cascade_ex.png' width=80% height=80%/>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# load in a haar cascade classifier for detecting frontal faces\n",
    "face_cascade = cv2.CascadeClassifier('detector_architectures/haarcascade_frontalface_default.xml')\n",
    "\n",
    "# run the detector\n",
    "# the output here is an array of detections; the corners of each detection box\n",
    "# if necessary, modify these parameters until you successfully identify every face in a given image\n",
    "faces = face_cascade.detectMultiScale(image, 1.2, 2)\n",
    "\n",
    "# make a copy of the original image to plot detections on\n",
    "image_with_detections = image.copy()\n",
    "\n",
    "# loop over the detected faces, mark the image where each face is found\n",
    "for (x,y,w,h) in faces:\n",
    "    # draw a rectangle around each detected face\n",
    "    # you may also need to change the width of the rectangle drawn depending on image resolution\n",
    "    cv2.rectangle(image_with_detections,(x,y),(x+w,y+h),(255,0,0),3) \n",
    "\n",
    "fig = plt.figure(figsize=(9,9))\n",
    "\n",
    "plt.imshow(image_with_detections)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading in a trained model\n",
    "\n",
    "Once you have an image to work with (and, again, you can select any image of faces in the `images/` directory), the next step is to pre-process that image and feed it into your CNN facial keypoint detector.\n",
    "\n",
    "First, load your best model by its filename."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from models import Net\n",
    "\n",
    "net = Net()\n",
    "\n",
    "## TODO: load the best saved model parameters (by your path name)\n",
    "## You'll need to un-comment the line below and add the correct name for *your* saved model\n",
    "# net.load_state_dict(torch.load('saved_models/keypoints_model_1.pt'))\n",
    "\n",
    "## print out your net and prepare it for testing (uncomment the line below)\n",
    "# net.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Keypoint detection\n",
    "\n",
    "Now, we'll loop over each detected face in an image (again!) only this time, you'll transform those faces in Tensors that your CNN can accept as input images.\n",
    "\n",
    "### TODO: Transform each detected face into an input Tensor\n",
    "\n",
    "You'll need to perform the following steps for each detected face:\n",
    "1. Convert the face from RGB to grayscale\n",
    "2. Normalize the grayscale image so that its color range falls in [0,1] instead of [0,255]\n",
    "3. Rescale the detected face to be the expected square size for your CNN (224x224, suggested)\n",
    "4. Reshape the numpy image into a torch image.\n",
    "\n",
    "You may find it useful to consult to transformation code in `data_load.py` to help you perform these processing steps.\n",
    "\n",
    "\n",
    "### TODO: Detect and display the predicted keypoints\n",
    "\n",
    "After each face has been appropriately converted into an input Tensor for your network to see as input, you'll wrap that Tensor in a Variable() and can apply your `net` to each face. The ouput should be the predicted the facial keypoints. These keypoints will need to be \"un-normalized\" for display, and you may find it helpful to write a helper function like `show_keypoints`. You should end up with an image like the following with facial keypoints that closely match the facial features on each individual face:\n",
    "\n",
    "<img src='images/michelle_detected.png' width=30% height=30%/>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "image_copy = np.copy(image)\n",
    "\n",
    "# loop over the detected faces from your haar cascade\n",
    "for (x,y,w,h) in faces:\n",
    "    \n",
    "    # Select the region of interest that is the face in the image \n",
    "    roi = image_copy[y:y+h, x:x+w]\n",
    "    \n",
    "    ## TODO: Convert the face region from RGB to grayscale\n",
    "\n",
    "    ## TODO: Normalize the grayscale image so that its color range falls in [0,1] instead of [0,255]\n",
    "    \n",
    "    ## TODO: Rescale the detected face to be the expected square size for your CNN (224x224, suggested)\n",
    "    \n",
    "    ## TODO: Reshape the numpy image shape (H x W x C) into a torch image shape (C x H x W)\n",
    "    \n",
    "    ## TODO: Make facial keypoint predictions using your loaded, trained network \n",
    "    ## perform a forward pass to get the predicted facial keypoints\n",
    "\n",
    "    ## TODO: Display each detected face and the corresponding keypoints        \n",
    "        \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:ai]",
   "language": "python",
   "name": "conda-env-ai-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}