t81_558_class_13_05_edge.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 13: Advanced/Other Topics**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 13 Video Material\n",
    "\n",
    "* Part 13.1: Flask and Deep Learning Web Services [[Video]](https://www.youtube.com/watch?v=H73m9XvKHug&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_13_01_flask.ipynb)\n",
    "* Part 13.2: Deploying a Model to AWS  [[Video]](https://www.youtube.com/watch?v=8ygCyvRZ074&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_13_02_cloud.ipynb)\n",
    "* Part 13.3: Using a Keras Deep Neural Network with a Web Application  [[Video]](https://www.youtube.com/watch?v=OBbw0e-UroI&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_13_03_web.ipynb)\n",
    "* Part 13.4: When to Retrain Your Neural Network [[Video]](https://www.youtube.com/watch?v=K2Tjdx_1v9g&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_13_04_retrain.ipynb)\n",
    "* **Part 13.5: AI at the Edge: Using Keras on a Mobile Device**  [[Video]]() [[Notebook]](t81_558_class_13_05_edge.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 13.5: Using a Keras Deep Neural Network with a Web Application\n",
    "\n",
    "In this section, we will see how to deploy a neural network to an iOS mobile device.  Android is also another option that I plan to support at some point.  However, for now, I am focusing on iOS.  Apple added its [CoreML](https://developer.apple.com/documentation/coreml) library that makes it considerably easier to deploy a deep neural network than it used to be. The example in this part will focus on creating a simple computer vision mobile application for image recognition.  All computation will occur on the actual device.  Compared with \"cloud compute,\" which occurs in the cloud, edge computing occurs on physical devices the user has physical access to.\n",
    "\n",
    "Apple makes [several pre-trained](https://developer.apple.com/machine-learning/models/) neural networks available for CoreML.  It is also possible to convert Keras models into the format needed by CoreML.  For this example, we will convert a Keras pre-trained model to CoreML.  This technique gives a good demonstration of this conversion that can use for other Keras models that you've created. \n",
    "\n",
    "Please note the following two requirements set forth by Apple.\n",
    "\n",
    "* You will need a Mac running [XCode]() to create an iOS application.\n",
    "* You must have a free Apple Developer account to deploy your app to your iOS device.  [Sign up here](https://developer.apple.com/).\n",
    "* To add your application to the [Apple App Store](https://www.apple.com/ios/app-store/) and deploy to other people's hardware, you mist [enroll](https://developer.apple.com/support/compare-memberships/) in the 100 USD/year developer program.\n",
    "\n",
    "### Converting Keras to CoreML\n",
    "\n",
    "The following code exports the MobileNet network to an H5 file. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "conda create -y --name coreml python=3.6\n",
    "source activate coreml\n",
    "conda install -y jupyter\n",
    "conda install -y scipy\n",
    "pip install --exists-action i --upgrade sklearn\n",
    "pip install --exists-action i --upgrade pandas\n",
    "pip install --exists-action i --upgrade pandas-datareader\n",
    "pip install --exists-action i --upgrade matplotlib\n",
    "pip install --exists-action i --upgrade pillow\n",
    "pip install --exists-action i --upgrade tqdm\n",
    "pip install --exists-action i --upgrade requests\n",
    "pip install --exists-action i --upgrade h5py\n",
    "pip install --exists-action i --upgrade pyyaml\n",
    "pip install --exists-action i --upgrade tensorflow==1.14\n",
    "pip install --exists-action i --upgrade keras==2.2.4\n",
    "pip install --exists-action i --upgrade coremltools\n",
    "conda update -y --all\n",
    "python -m ipykernel install --user --name coreml \n",
    "        --display-name \"Python 3.6 (coreml)\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W0729 09:30:33.498896 140735591678848 deprecation_wrapper.py:119] From /Users/jheaton/miniconda3/envs/coreml/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.\n",
      "\n",
      "W0729 09:30:35.553636 140735591678848 deprecation.py:506] From /Users/jheaton/miniconda3/envs/coreml/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n"
     ]
    }
   ],
   "source": [
    "# Export MobileNet to an H5 file\n",
    "import os\n",
    "from keras.applications import MobileNet\n",
    "\n",
    "save_path = \"./dnn/\"\n",
    "model = MobileNet(weights='imagenet',include_top=True)\n",
    "model.save(os.path.join(save_path,\"mobilenet.h5\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, as of August 2019, CoreML does not support TensorFlow 2.0. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow version: 1.14.0\n",
      "Keras version: 2.2.4\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import keras\n",
    "import coremltools\n",
    "\n",
    "print(f\"TensorFlow version: {tf.__version__}\")\n",
    "print(f\"Keras version: {keras.__version__}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "r = requests.get('https://data.heatonresearch.com/data'+\\\n",
    "                 '/t81-558/imagenet_class_index.json')\n",
    "\n",
    "js = r.json()\n",
    "\n",
    "lookup = ['' for x in range(1000)]\n",
    "for idx in js:\n",
    "    lookup[int(idx)] = js[idx][1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 : input_1, <keras.engine.input_layer.InputLayer object at 0xa2ef30b70>\n",
      "1 : conv1_pad, <keras.layers.convolutional.ZeroPadding2D object at 0xa2ef30fd0>\n",
      "2 : conv1, <keras.layers.convolutional.Conv2D object at 0xa2ef30d68>\n",
      "3 : conv1_bn, <keras.layers.normalization.BatchNormalization object at 0xa2ed79240>\n",
      "4 : conv1_relu, <keras.layers.advanced_activations.ReLU object at 0xa2ed796d8>\n",
      "5 : conv_dw_1, <keras.layers.convolutional.DepthwiseConv2D object at 0xa2ed79898>\n",
      "6 : conv_dw_1_bn, <keras.layers.normalization.BatchNormalization object at 0xa2ed79550>\n",
      "7 : conv_dw_1_relu, <keras.layers.advanced_activations.ReLU object at 0xa2eff9e80>\n",
      "8 : conv_pw_1, <keras.layers.convolutional.Conv2D object at 0xa2f0c35f8>\n",
      "9 : conv_pw_1_bn, <keras.layers.normalization.BatchNormalization object at 0xa2f083860>\n",
      "10 : conv_pw_1_relu, <keras.layers.advanced_activations.ReLU object at 0xa2f0e1588>\n",
      "11 : conv_pad_2, <keras.layers.convolutional.ZeroPadding2D object at 0xa2f1c0588>\n",
      "12 : conv_dw_2, <keras.layers.convolutional.DepthwiseConv2D object at 0xa2f183da0>\n",
      "13 : conv_dw_2_bn, <keras.layers.normalization.BatchNormalization object at 0xa2f236780>\n",
      "14 : conv_dw_2_relu, <keras.layers.advanced_activations.ReLU object at 0xa2f236908>\n",
      "15 : conv_pw_2, <keras.layers.convolutional.Conv2D object at 0xa2f27fe80>\n",
      "16 : conv_pw_2_bn, <keras.layers.normalization.BatchNormalization object at 0xa2f297828>\n",
      "17 : conv_pw_2_relu, <keras.layers.advanced_activations.ReLU object at 0xa2f332eb8>\n",
      "18 : conv_dw_3, <keras.layers.convolutional.DepthwiseConv2D object at 0xa2f3b0da0>\n",
      "19 : conv_dw_3_bn, <keras.layers.normalization.BatchNormalization object at 0xa2f391860>\n",
      "20 : conv_dw_3_relu, <keras.layers.advanced_activations.ReLU object at 0xa2f42ce48>\n",
      "21 : conv_pw_3, <keras.layers.convolutional.Conv2D object at 0xa2f45cf28>\n",
      "22 : conv_pw_3_bn, <keras.layers.normalization.BatchNormalization object at 0xa2f505e10>\n",
      "23 : conv_pw_3_relu, <keras.layers.advanced_activations.ReLU object at 0xa2f4d1ac8>\n",
      "24 : conv_pad_4, <keras.layers.convolutional.ZeroPadding2D object at 0xa2f2f92e8>\n",
      "25 : conv_dw_4, <keras.layers.convolutional.DepthwiseConv2D object at 0xa2f58c940>\n",
      "26 : conv_dw_4_bn, <keras.layers.normalization.BatchNormalization object at 0xa2f64e400>\n",
      "27 : conv_dw_4_relu, <keras.layers.advanced_activations.ReLU object at 0xa2f64eac8>\n",
      "28 : conv_pw_4, <keras.layers.convolutional.Conv2D object at 0xa2f689b70>\n",
      "29 : conv_pw_4_bn, <keras.layers.normalization.BatchNormalization object at 0xa2f6a2588>\n",
      "30 : conv_pw_4_relu, <keras.layers.advanced_activations.ReLU object at 0xa2f735b00>\n",
      "31 : conv_dw_5, <keras.layers.convolutional.DepthwiseConv2D object at 0xa2f782be0>\n",
      "32 : conv_dw_5_bn, <keras.layers.normalization.BatchNormalization object at 0xa2f79c550>\n",
      "33 : conv_dw_5_relu, <keras.layers.advanced_activations.ReLU object at 0xa2f834d68>\n",
      "34 : conv_pw_5, <keras.layers.convolutional.Conv2D object at 0xa2f8f5400>\n",
      "35 : conv_pw_5_bn, <keras.layers.normalization.BatchNormalization object at 0xa2f897940>\n",
      "36 : conv_pw_5_relu, <keras.layers.advanced_activations.ReLU object at 0xa2f8d92b0>\n",
      "37 : conv_pad_6, <keras.layers.convolutional.ZeroPadding2D object at 0xa2f9a16a0>\n",
      "38 : conv_dw_6, <keras.layers.convolutional.DepthwiseConv2D object at 0xa2f969e80>\n",
      "39 : conv_dw_6_bn, <keras.layers.normalization.BatchNormalization object at 0xa2fa09a58>\n",
      "40 : conv_dw_6_relu, <keras.layers.advanced_activations.ReLU object at 0xa2fa09f98>\n",
      "41 : conv_pw_6, <keras.layers.convolutional.Conv2D object at 0xa2fa93dd8>\n",
      "42 : conv_pw_6_bn, <keras.layers.normalization.BatchNormalization object at 0xa2f6e8f60>\n",
      "43 : conv_pw_6_relu, <keras.layers.advanced_activations.ReLU object at 0xa2fab9d68>\n",
      "44 : conv_dw_7, <keras.layers.convolutional.DepthwiseConv2D object at 0xa2fb90f28>\n",
      "45 : conv_dw_7_bn, <keras.layers.normalization.BatchNormalization object at 0xa2fb729e8>\n",
      "46 : conv_dw_7_relu, <keras.layers.advanced_activations.ReLU object at 0xa2fbb6358>\n",
      "47 : conv_pw_7, <keras.layers.convolutional.Conv2D object at 0xa2fcad2e8>\n",
      "48 : conv_pw_7_bn, <keras.layers.normalization.BatchNormalization object at 0xa2fc6df28>\n",
      "49 : conv_pw_7_relu, <keras.layers.advanced_activations.ReLU object at 0xa2fccf198>\n",
      "50 : conv_dw_8, <keras.layers.convolutional.DepthwiseConv2D object at 0xa2fdaa358>\n",
      "51 : conv_dw_8_bn, <keras.layers.normalization.BatchNormalization object at 0xa2fd69e80>\n",
      "52 : conv_dw_8_relu, <keras.layers.advanced_activations.ReLU object at 0xa2fe05320>\n",
      "53 : conv_pw_8, <keras.layers.convolutional.Conv2D object at 0xa2fea3780>\n",
      "54 : conv_pw_8_bn, <keras.layers.normalization.BatchNormalization object at 0xa2fe7fe80>\n",
      "55 : conv_pw_8_relu, <keras.layers.advanced_activations.ReLU object at 0xa2fec6828>\n",
      "56 : conv_dw_9, <keras.layers.convolutional.DepthwiseConv2D object at 0xa2ff19f60>\n",
      "57 : conv_dw_9_bn, <keras.layers.normalization.BatchNormalization object at 0xa2ff45160>\n",
      "58 : conv_dw_9_relu, <keras.layers.advanced_activations.ReLU object at 0xa30015940>\n",
      "59 : conv_pw_9, <keras.layers.convolutional.Conv2D object at 0xa3009af28>\n",
      "60 : conv_pw_9_bn, <keras.layers.normalization.BatchNormalization object at 0xa30077710>\n",
      "61 : conv_pw_9_relu, <keras.layers.advanced_activations.ReLU object at 0xa3010ddd8>\n",
      "62 : conv_dw_10, <keras.layers.convolutional.DepthwiseConv2D object at 0xa30192f98>\n",
      "63 : conv_dw_10_bn, <keras.layers.normalization.BatchNormalization object at 0xa301747f0>\n",
      "64 : conv_dw_10_relu, <keras.layers.advanced_activations.ReLU object at 0xa3020ddd8>\n",
      "65 : conv_pw_10, <keras.layers.convolutional.Conv2D object at 0xa3023aeb8>\n",
      "66 : conv_pw_10_bn, <keras.layers.normalization.BatchNormalization object at 0xa30271860>\n",
      "67 : conv_pw_10_relu, <keras.layers.advanced_activations.ReLU object at 0xa302acf28>\n",
      "68 : conv_dw_11, <keras.layers.convolutional.DepthwiseConv2D object at 0xa3032fa58>\n",
      "69 : conv_dw_11_bn, <keras.layers.normalization.BatchNormalization object at 0xa3032f470>\n",
      "70 : conv_dw_11_relu, <keras.layers.advanced_activations.ReLU object at 0xa303a8160>\n",
      "71 : conv_pw_11, <keras.layers.convolutional.Conv2D object at 0xa304c5588>\n",
      "72 : conv_pw_11_bn, <keras.layers.normalization.BatchNormalization object at 0xa304647b8>\n",
      "73 : conv_pw_11_relu, <keras.layers.advanced_activations.ReLU object at 0xa304e9550>\n",
      "74 : conv_pad_12, <keras.layers.convolutional.ZeroPadding2D object at 0xa30569a90>\n",
      "75 : conv_dw_12, <keras.layers.convolutional.DepthwiseConv2D object at 0xa305c45f8>\n",
      "76 : conv_dw_12_bn, <keras.layers.normalization.BatchNormalization object at 0xa3063b9b0>\n",
      "77 : conv_dw_12_relu, <keras.layers.advanced_activations.ReLU object at 0xa3063bef0>\n",
      "78 : conv_pw_12, <keras.layers.convolutional.Conv2D object at 0xa306bad30>\n",
      "79 : conv_pw_12_bn, <keras.layers.normalization.BatchNormalization object at 0xa3069a860>\n",
      "80 : conv_pw_12_relu, <keras.layers.advanced_activations.ReLU object at 0xa3077ef98>\n",
      "81 : conv_dw_13, <keras.layers.convolutional.DepthwiseConv2D object at 0xa30802da0>\n",
      "82 : conv_dw_13_bn, <keras.layers.normalization.BatchNormalization object at 0xa307e3940>\n",
      "83 : conv_dw_13_relu, <keras.layers.advanced_activations.ReLU object at 0xa30858dd8>\n",
      "84 : conv_pw_13, <keras.layers.convolutional.Conv2D object at 0xa3091c860>\n",
      "85 : conv_pw_13_bn, <keras.layers.normalization.BatchNormalization object at 0xa308fbf60>\n",
      "86 : conv_pw_13_relu, <keras.layers.advanced_activations.ReLU object at 0xa3093f828>\n",
      "87 : global_average_pooling2d_1, <keras.layers.pooling.GlobalAveragePooling2D object at 0xa309c3eb8>\n",
      "88 : reshape_1, <keras.layers.core.Reshape object at 0xa309d9a90>\n",
      "89 : conv_preds, <keras.layers.convolutional.Conv2D object at 0xa309c3278>\n",
      "90 : reshape_2, <keras.layers.core.Reshape object at 0xa30ad2978>\n",
      "91 : act_softmax, <keras.layers.core.Activation object at 0xa30ad2f60>\n"
     ]
    }
   ],
   "source": [
    "coreml_model = coremltools.converters.keras.convert(model,\n",
    "    input_names=\"image\",\n",
    "    image_input_names=\"image\",\n",
    "    image_scale=1/255.0,\n",
    "    class_labels=lookup,\n",
    "    is_bgr=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "coreml_model.save(os.path.join(save_path,\"mobilenet.mlmodel\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating an IOS CoreML Application\n",
    "\n",
    "We will now use the neural network created in the last section to create an IOS application that will classify what its camera sees.  This application will be a single image classification, not the multi-image classification that we saw with YOLO.  Figure 13.IOS shows this application running on my iPhone here:\n",
    "\n",
    "**Figure 13.IOS: IOS Image Classify**\n",
    "![IOS Image Classify](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/ios-1.png \"IOS Image Classify\")\n",
    "\n",
    "You can find the complete source code (in XCode) for this application  at the following URL:\n",
    "\n",
    "* [GitHub: IOS Classify](https://github.com/jeffheaton/ios_video_classify)\n",
    "\n",
    "To create this application from scratch (in XCode), follow these steps:\n",
    "\n",
    "1. Install XCode\n",
    "2. Register for Apple Developer account (if you wish to deploy to iOS device)\n",
    "3. Create a new XCode Project\n",
    "4. Delete storyboard\n",
    "5. Remove project references to storyboard\n",
    "6. Add camera prompt to security settings\n",
    "7. Replace the contents of the view controller with the included\n",
    "8. Test on IOS device\n",
    "\n",
    "The YouTube video for this module goes through the above process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More Reading\n",
    "\n",
    "There are several excellent tutorials on IOS and CoreML development.  The following articles were beneficial in the creation of this material.\n",
    "\n",
    "* [Running Keras models on iOS with CoreML](https://www.pyimagesearch.com/2018/04/23/running-keras-models-on-ios-with-coreml/)\n",
    "* [How to build an image recognition iOS app with Apple’s CoreML and Vision APIs](https://www.freecodecamp.org/news/ios-coreml-vision-image-recognition-3619cf319d0b/)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.7 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}