people_deployment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Smart Queue Monitoring System\n",
    "\n",
    "In this project you will build a people counter app to reduce congestion in queuing systems by guiding people to the least congested queue. You will have to use Intel's OpenVINO API and the person detection model from their open model zoo to build this project. It demonstrates how to create a smart video IoT solution using Intel® hardware and software tools. This solution detects people in a designated area, providing the number of people in the frame.\n",
    "\n",
    "## Overview of how it works\n",
    "Your code should read the equivalent of command line arguments and loads a network and image from the video input to the Inference Engine (IE) plugin. A job is submitted to an edge compute node with a hardware accelerator such as Intel® HD Graphics GPU, Intel® Movidius™ Neural Compute Stick 2 and Intel® Arria® 10 FPGA.\n",
    "After the inference is completed, the output videos are appropriately stored in the /results/[device] directory, which can then be viewed within the Jupyter Notebook instance.\n",
    "\n",
    "## Demonstration objectives\n",
    "* Video as input is supported using **OpenCV**\n",
    "* Inference performed on edge hardware (rather than on the development node hosting this Jupyter notebook)\n",
    "* **OpenCV** provides the bounding boxes, labels and other information\n",
    "* Visualization of the resulting bounding boxes\n",
    "\n",
    "\n",
    "## Step 0: Set Up\n",
    "\n",
    "### 0.1: Import dependencies\n",
    "\n",
    "Run the below cell to import Python dependencies needed for displaying the results in this notebook\n",
    "(tip: select the cell and use **Ctrl+enter** to run the cell)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'demoTools'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-26-d6b5d7af76c0>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mIPython\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdisplay\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mHTML\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mdemoTools\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdemoutils\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'demoTools'"
     ]
    }
   ],
   "source": [
    "#Import your dependencies here\n",
    "\n",
    "#Some dependencies have been imported for you\n",
    "#You can find demoTools in `~/Reference-samples/iot-devcloud`\n",
    "\n",
    "from IPython.display import HTML\n",
    "from demoTools.demoutils import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 0.2  (Optional-step): Original video without inference\n",
    "\n",
    "If you are curious to see the input video, run the following cell to view the original video stream used for inference and people counter.\n",
    "\n",
    "*Tip*: Before doing this make sure to upload your video to the ./resources folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ln -sf bin/video_files/Scenario\\ 1\\ -\\ Manufacturing.mp4\n",
    "!ln -sf bbin/video_files/Scenario\\ 3\\ -\\ Transportation.mp4\n",
    "!ln -sf bin/video_files/Scenario\\ 2\\ -\\ Retail.mp4\n",
    "videoHTML('People Counter Video', ['manufacturing.mov'])\n",
    "videoHTML('People Counter Video', ['transportation.mov'])\n",
    "videoHTML('People Counter Video', ['retail.mov'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Using Intel® Distribution of OpenVINO™ toolkit\n",
    "\n",
    "We will be using Intel® Distribution of OpenVINO™ toolkit Inference Engine (IE) to locate people in frame.\n",
    "There are five steps involved in this task:\n",
    "\n",
    "1. Download the model using the open_model_zoo\n",
    "2. Choose a device and create IEPlugin for the device\n",
    "3. Read the Model using IENetwork\n",
    "4. Load the IENetwork into the Plugin\n",
    "5. Run inference.\n",
    "\n",
    "### 1.1 Downloading Model\n",
    "\n",
    "Write a command to download the  **person-detection-retail-0013** model in an IR format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Write your command here \n",
    "\n",
    "!wget /opt/intel/openvino/deployment_tools/tools/model_downloader/downloader.py person-detection-retail-0013 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 : Inference on a video\n",
    "\n",
    "By now you should have already completed the inference code in <a href=\"person_detect.py\">person_detect.py</a>. If you haven't done so already, then you should do it now.\n",
    "\n",
    "The Python code should take in command line arguments for video, model etc.\n",
    "\n",
    "While the type of command line options is up to you, the command below is an example \n",
    "\n",
    "```\n",
    "python3 main.py -m ${MODELPATH} \\\n",
    "                -i ${INPUT_FILE} \\\n",
    "                -o ${OUTPUT_FILE} \\\n",
    "                -d ${DEVICE} \\\n",
    "                -pt ${THRESHOLD}\\\n",
    "                -l /opt/intel/openvino/deployment_tools/inference_engine/lib/intel64/libcpu_extension_sse4.so\n",
    "\n",
    "```\n",
    "\n",
    "##### The description of the arguments used in the argument parser is the command line executable equivalent.\n",
    "* -m location of the pre-trained IR model which has been pre-processed using the model optimizer. There is automated support built in this argument to support both FP32 and FP16 models targeting different hardware\n",
    "* -i  location of the input video stream\n",
    "* -o location where the output file with inference needs to be stored (results/[device])\n",
    "* -d type of Hardware Acceleration (CPU, GPU, MYRIAD, HDDL or HETERO:FPGA,CPU)\n",
    "* -pt probability threshold value for the person detection\n",
    "* -l absolute path to the shared library (or cpu extensions) and is currently optimized for core/xeon (/opt/intel/openvino/deployment_tools/inference_engine/lib/intel64/libcpu_extension_sse4.so)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Creating job file\n",
    "\n",
    "To run inference on the video, we need more compute power.\n",
    "We will run the workload on several edge compute nodes present in the IoT DevCloud. We will send work to the edge compute nodes by submitting the corresponding non-interactive jobs into a queue. For each job, we will specify the type of the edge compute server that must be allocated for the job.\n",
    "\n",
    "The job file is written in Bash, and will be executed directly on the edge compute node.\n",
    "You will have to create the job file.\n",
    "\n",
    "You can either create one file for all the jobs that you want to run or create multiple files for each of the jobs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting person_detect_job.sh\n"
     ]
    }
   ],
   "source": [
    "%%writefile person_detect_job.sh\n",
    "\n",
    "\n",
    "model=$1\n",
    "device=$2 \n",
    "input_file=$3 \n",
    "output_file=$4\n",
    "extensions=$5\n",
    "# threshold=$6 Default 0.5\n",
    "\n",
    "\n",
    "python3 person_detect.py --model ${model} --extensions ${extensions} --visualise --video ${input_file} --"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Understand how jobs are submitted into the queue\n",
    "\n",
    "Now that we have the job script, we can submit the jobs to edge compute nodes. In the IoT DevCloud, you can do this using the `qsub` command.\n",
    "We can submit people_counter to several different types of edge compute nodes simultaneously or just one node at a time.\n",
    "\n",
    "There are three options of `qsub` command that we use for this:\n",
    "- `-l` : this option let us select the number and the type of nodes using `nodes={node_count}:{property}`. \n",
    "- `-F` : this option let us send arguments to the bash script. \n",
    "- `-N` : this option let us name the job so that it is easier to distinguish between them.\n",
    "\n",
    "If you are curious to see the available types of nodes on the IoT DevCloud, run the following optional cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pbsnodes | grep compnode | awk '{print $3}' | sort | uniq -c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, the properties describe the node, and number on the left is the number of available nodes of that architecture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Job queue submission\n",
    "\n",
    "Each of the cells below should submit a job to different edge compute nodes.\n",
    "The output of the cell is the `JobID` of your job, which you can use to track progress of a job.\n",
    "\n",
    "**Note** You can submit all jobs at once or one at a time. \n",
    "\n",
    "After submission, they will go into a queue and run as soon as the requested compute resources become available. \n",
    "(tip: **shift+enter** will run the cell and automatically move you to the next cell. So you can hit **shift+enter** multiple times to quickly run multiple cells)\n",
    "\n",
    "#### Submitting to an edge compute node with an Intel® CPU\n",
    "In the cell below, write a script to submit a job to an <a \n",
    "    href=\"https://software.intel.com/en-us/iot/hardware/iei-tank-dev-kit-core\">IEI \n",
    "    Tank* 870-Q170</a> edge node with an <a \n",
    "    href=\"https://ark.intel.com/products/88186/Intel-Core-i5-6500TE-Processor-6M-Cache-up-to-3-30-GHz-\">Intel® Core™ i5-6500TE processor</a>. The inference workload will run on the CPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16785.v-qsvr-1.devcloud-edge\r\n"
     ]
    }
   ],
   "source": [
    "#Submit job to the queue\n",
    "!qsub person_detect_job.sh -d . -l nodes=1:tank-870:i5-6500te -F \"intel/person-detection-retail-0013/FP16/person-detection-retail-0013 CPU bin/video_files/manufacturing.mp4 bin/libcpu_extension_sse4.so\"\n",
    "#Progress indicators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Job ID                    Name             User            Time Use S Queue\r\n",
      "------------------------- ---------------- --------------- -------- - -----\r\n",
      "16774.v-qsvr-1             ...ub-singleuser u39058          04:42:23 R jupyterhub     \r\n",
      "16805.v-qsvr-1             ...detect_job.sh u39058          00:00:35 R batch          \r\n"
     ]
    }
   ],
   "source": [
    "!qstat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Submitting to an edge compute node with Intel® Core CPU and using the onboard Intel® GPU\n",
    "In the cell below, write a script to submit a job to an <a \n",
    "    href=\"https://software.intel.com/en-us/iot/hardware/iei-tank-dev-kit-core\">IEI \n",
    "    Tank* 870-Q170</a> edge node with an <a href=\"https://ark.intel.com/products/88186/Intel-Core-i5-6500TE-Processor-6M-Cache-up-to-3-30-GHz-\">Intel® Core i5-6500TE</a>. The inference workload will run on the Intel® HD Graphics 530 card integrated with the CPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16805.v-qsvr-1.devcloud-edge\r\n"
     ]
    }
   ],
   "source": [
    "#Submit job to the queue\n",
    "!qsub person_detect_job.sh -d . -l nodes=1:tank-870:i5-6500te:intel-hd-530 -F \"intel/person-detection-retail-0013/FP16/person-detection-retail-0013 MULTI:GPU,CPU bin/video_files/manufacturing.mp4 bin/libcpu_extension_sse4.so\"\n",
    "#Progress indicators\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Submitting to an edge compute node with Intel® NCS 2 (Neural Compute Stick 2)\n",
    "In the cell below, write a script to submit a job to an <a \n",
    "    href=\"https://software.intel.com/en-us/iot/hardware/iei-tank-dev-kit-core\">IEI \n",
    "    Tank 870-Q170</a> edge node with an <a href=\"https://ark.intel.com/products/88186/Intel-Core-i5-6500TE-Processor-6M-Cache-up-to-3-30-GHz-\">Intel Core i5-6500te CPU</a>. The inference workload will run on an <a \n",
    "    href=\"https://software.intel.com/en-us/neural-compute-stick\">Intel Neural Compute Stick 2</a> installed in this  node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Submit job to the queue\n",
    "!qsub person_detect_job.sh -d . -l nodes=1:tank-870:i5-6500te: -F \"intel/person-detection-retail-0013/FP16/person-detection-retail-0013 MULTI:GPU,CPU bin/video_files/manufacturing.mp4 bin/libcpu_extension_sse4.so\"\n",
    "#Progress indicators\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Submitting to an edge compute node with IEI Mustang-F100-A10 (Intel® Arria® 10 FPGA)\n",
    "In the cell below, write a script to submit a job to an <a \n",
    "    href=\"https://software.intel.com/en-us/iot/hardware/iei-tank-dev-kit-core\">IEI \n",
    "    Tank 870-Q170</a> edge node with an <a href=\"https://ark.intel.com/products/88186/Intel-Core-i5-6500TE-Processor-6M-Cache-up-to-3-30-GHz-\">Intel Core™ i5-6500te CPU</a> . The inference workload will run on the <a href=\"https://www.ieiworld.com/mustang-f100/en/\"> IEI Mustang-F100-A10 </a> card installed in this node."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Submit job to the queue\n",
    " \n",
    "#Progress indicators\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Check if the jobs are done\n",
    "\n",
    "To check on the jobs that were submitted, use a command to check the status of the job.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Enter your command here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Briefly indicate what each of the current status column shows \n",
    "\n",
    "For example,\n",
    "- `JOB ID`is in Q state, it is in the queue waiting for available resources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Wait!***\n",
    "\n",
    "Please wait for the inference jobs and video rendering to complete before proceeding to the next step.\n",
    "\n",
    "## Step 3: View Results\n",
    "\n",
    "Write a short utility script that will display these videos within the notebook.\n",
    "\n",
    "*Tip*: See `demoutils.py` if you are interested in understanding further on how the results are displayed in notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Write your script for Intel Core CPU video results here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Write your script for Intel Core CPU +GPU video results here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Write your script for Intel CPU + Intel NCS2 video results here "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Write your script for Intel® Arria® 10 FPGA video results here "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Assess Performance\n",
    "\n",
    "This is where you need to write code to asses how well your model is performing.\n",
    "You need to compare the following timings for all the models:\n",
    "\n",
    "- Model loading time\n",
    "- Average Inference Time\n",
    "\n",
    "Preferable, show your results in the form of a bar chart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO Write your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}