Course_PerceptionROS_Project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Project: Have your own simulated Aibo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Project Intro",
    "image": true,
    "name": "perception_project_intro1",
    "width": "15cm"
   },
   "source": [
    "<img src=\"img/perception_project_intro1.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, this project is meant for practising all that you have learned in this course with a robot that was made 20 years ago, which even now, has top notch perception. You are going to replicate that.<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Your Own Simplified Aibo ERS7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Aibo Robot",
    "image": true,
    "name": "perception_proyect_aibopresentation",
    "width": "12cm"
   },
   "source": [
    "<img src=\"img/perception_proyect_aibopresentation.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is Aibo. Created by the Digital Creatures Lab of SONY and directed by <a href=\"https://en.wikipedia.org/wiki/Toshitada_Doi\">Toshitada Doi</a>, this robo-dog was created in 1999. But even by current robotics standards, this robot was amazing. It had object, blob, face, and voice recognition. It also auto-charged and walked by itself. It even had self-character evolution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will not use the whole version, just for simplification reasons. You will use an adapted simulated version.<br>\n",
    "These are its main features:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* One of its main differences is that it **won't move through quadruped locomotion**. The robot is fixed through a base and it will be moved in the exact same way as you have moved objects and people throughout this course. The reason is simple: this is not a course about locomotion, and this way, you won't have the problems associated with natural locomotion and perception, such as having the robot falling and losing track of everything.<br>\n",
    "You will be able to move all the joints freely, just the feet won't be touching the ground.<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* You will have to publish a Twist command into the topic **/aiboERS7/cmd_vel**. If you want to move it around manually to do some tests, you can use the launch file to move it with the keyboard:<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "roslaunch aibo_description aibo_move_with_keyboard.launch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* To move the robot joints of Aibo, you do it the same as you would with any other robot with joints. Just publish an angle value into the topic and the joint will move to that position. You have all these joints available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "/aibo_tc/L_ear_joint_position_controller/command                                           \n",
    "/aibo_tc/L_ear_tilt_position_controller/command                                          \n",
    "/aibo_tc/R_ear_joint_position_controller/command                                           \n",
    "/aibo_tc/R_ear_tilt_position_controller/command                                          \n",
    "/aibo_tc/headPan_position_controller/command                                           \n",
    "/aibo_tc/headTilt_position_controller/command                                          \n",
    "/aibo_tc/legLB1_position_controller/command                                          \n",
    "/aibo_tc/legLB2_position_controller/command                                          \n",
    "/aibo_tc/legLB3_position_controller/command                                          \n",
    "/aibo_tc/legLF1_position_controller/command                                          \n",
    "/aibo_tc/legLF2_position_controller/command                                          \n",
    "/aibo_tc/legLF3_position_controller/command                                          \n",
    "/aibo_tc/legRB1_position_controller/command                                          \n",
    "/aibo_tc/legRB2_position_controller/command                                          \n",
    "/aibo_tc/legRB3_position_controller/command                                          \n",
    "/aibo_tc/legRF1_position_controller/command                                          \n",
    "/aibo_tc/legRF2_position_controller/command                                          \n",
    "/aibo_tc/legRF3_position_controller/command                                          \n",
    "/aibo_tc/mouth_joint_position_controller/command                                           \n",
    "/aibo_tc/neck_joint_position_controller/command                                          \n",
    "/aibo_tc/tailPan_position_controller/command                                           \n",
    "/aibo_tc/tailTilt_position_controller/command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try, for example, to **wiggle the tail**. To do that, you will have to publish **/aibo_tc/tailPan_position_controller/command** into the topic.<br>\n",
    "The most important joint topics for perception are, of course, the ones related to the head position:<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "/aibo_tc/headPan_position_controller/command                                 \n",
    "/aibo_tc/headTilt_position_controller/command\n",
    "/aibo_tc/neck_joint_position_controller/command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Through these topics, you will be able to position the head camera where you want."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Head tilt:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "HeadTilt Aibo",
    "image": true,
    "name": "perception_project_headtilt",
    "width": "10cm"
   },
   "source": [
    "<table style=\"width:100%\">\n",
    "  <tr>\n",
    "    <th>\n",
    "    <figure>\n",
    "      <img id=\"fig-C.1\" src=\"img/perception_project_headtilt.gif\" width=\"300\"/>\n",
    "       <center> <figcaption><h2>HeadTilt</h2></figcaption></center>\n",
    "    </figure>\n",
    "\n",
    "    </th>\n",
    "  </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Head pan:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "HeadPan Aibo",
    "image": true,
    "name": "perception_project_headpan",
    "width": "10cm"
   },
   "source": [
    "<table style=\"width:100%\">\n",
    "  <tr>\n",
    "    \n",
    "    <th>\n",
    "        <figure>\n",
    "      <img id=\"fig-C.2\" src=\"img/perception_project_headpan.gif\" width=\"300\"/>\n",
    "       <center> <figcaption><h2>HeadPan</h2></figcaption></center>\n",
    "    </figure>\n",
    "    </th>\n",
    "    \n",
    "    \n",
    "  </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neck:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Neck movement Aibo",
    "image": true,
    "name": "perception_project_neck",
    "width": "10cm"
   },
   "source": [
    "<table style=\"width:100%\">\n",
    "  <tr>\n",
    "    \n",
    "    <th>\n",
    "        <figure>\n",
    "      <img id=\"fig-C.3\" src=\"img/perception_project_neck.gif\" width=\"300\"/>\n",
    "       <center> <figcaption><h2>Neck</h2></figcaption></center>\n",
    "    </figure>\n",
    "    </th>\n",
    "    \n",
    "    \n",
    "\n",
    "  </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tail:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Tail movement Aibo",
    "image": true,
    "name": "perception_project_tail",
    "width": "10cm"
   },
   "source": [
    "<table style=\"width:100%\">\n",
    "  <tr>\n",
    "    \n",
    "    \n",
    "    <th>\n",
    "        <figure>\n",
    "      <img id=\"fig-C.4\" src=\"img/perception_project_tail.gif\" width=\"300\"/>\n",
    "       <center> <figcaption><h2>Tail</h2></figcaption></center>\n",
    "    </figure>\n",
    "    </th>\n",
    "\n",
    "  </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* This simulated Aibo has, instead of its normal RGB CDD camera 362 × 492, a small Xtion PointCloud sensor. This will allow you to have **RGB images, depthImages, and PointCloud data**. This is vital for object detection and people tracking. Think of it as a Remastered version of AiboERS7. The sensor will be in the same spot where the original simple RGB camera was.<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Aibo Sketch",
    "image": true,
    "name": "perception_project_sketch2",
    "width": "15cm"
   },
   "source": [
    "<img src=\"img/perception_project_sketch2.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Manipulation Course Image",
    "image": true,
    "name": "perception_project_sensors1",
    "width": "13cm"
   },
   "source": [
    "* You will face some real problems with these kinds of sensors. For example, the PointCloud's sensors have a maximum range, but also a minimum range. This means that when objects get too far or too near, you won't get any readings. This means that you will have these kind of situations:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Aibo in RVIZ with DepthSensor and RGB camera",
    "image": true,
    "name": "perception_project_sensors1",
    "width": "15cm"
   },
   "source": [
    "<img src=\"img/perception_project_sensors1.png\" /><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you can see that the RGB camera is getting the ball. But, the PointCloud isn't. This is because of the minimum range of the PointCloud Sensor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* You will have the following main image data topics, plus their different tranformations:<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "/head_camera/rgb/image_raw: Typical RGB image.\n",
    "/head_camera/rgb/camera_info: Information about the RGB camera.\n",
    "\n",
    "/head_camera/depth_registered/image_raw: This is Depth-2D-Image made through distance readings of the PointCloud.\n",
    "/head_camera/depth_registered/camera_info: Information about the DepthImage.\n",
    "\n",
    "/head_camera/depth_registered/points : This is the PointCloud sensor data.\n",
    "Type:\n",
    "sensor_msgs/PointCloud2\n",
    "...\n",
    "uint8[] data: Main data container where all the point cloud info is.\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you have a brief description of what each one gives as output:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RGB, Depth and PointCloud"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### camera_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Camera info topics give information about all the parameters of the real or Simulated camera:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table style=\"float:left;background: #407EAF\">\n",
    "<tr>\n",
    "<th>\n",
    "<p class=\"transparent\">Execute in WebShell #1</p>\n",
    "</th>\n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rostopic info /head_camera/rgb/camera_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will give you the message type **sensor_msgs/CameraInfo**. Although you could use the command **rosmsg show** to learn the variables inside the topic, you will get a better explanation of all the variables if you do the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table style=\"float:left;background: #407EAF\">\n",
    "<tr>\n",
    "<th>\n",
    "<p class=\"transparent\">Execute in WebShell #1</p>\n",
    "</th>\n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "roscd sensor_msgs/msg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vim CameraInfo.msg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doing this will get you a detailed explanation of what the eahc variable is.<br>\n",
    "This topic can be used in RVIZ to draw the optical parameters of where the camera is pointing to.<br>\n",
    "To do so, just add the Element **CameraInfo** into RVIZ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Camera Info Element:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Camera Info Element",
    "image": true,
    "name": "perception_project_camerainforvizelem",
    "width": "7cm"
   },
   "source": [
    "<img src=\"img/perception_project_camerainforvizelem.png\"/><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Camera Info:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Camera Info",
    "image": true,
    "name": "perception_project_camerainfo",
    "width": "7cm"
   },
   "source": [
    "<img src=\"img/perception_project_camerainfo.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is very useful in perception to know if the robot should be seeing the object or not. You can use any camera info topics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "/head_camera/rgb/camera_info\n",
    "/head_camera/depth_registered/camera_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Image and Pointcloud"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you can see a comparison of how the RGB, DepthImage, and PointCloud see the girl, Olive:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RGB:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Girl named Olive in RGB Camera",
    "image": true,
    "name": "perception_project_cam1",
    "width": "10cm"
   },
   "source": [
    "<img src=\"img/perception_project_cam1.png\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deapth2DCamera:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Girl named Olive in Deapth2DCamera",
    "image": true,
    "name": "perception_project_cam2",
    "width": "10cm"
   },
   "source": [
    "<img src=\"img/perception_project_cam2.png\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PointCloudCamera:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Girl named Olive in PointCloudCamera",
    "image": true,
    "name": "perception_project_cam3",
    "width": "10cm"
   },
   "source": [
    "<img src=\"img/perception_project_cam3.png\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can visualise the PointClouds in many ways. These are two examples, one with the PointCloud2 RVIZ element and the other with the DepthCloud element.<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PointCloud:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Ball seen in PointCloud element",
    "image": true,
    "name": "perception_project_pcl1",
    "width": "10cm"
   },
   "source": [
    "<img src=\"img/perception_project_pcl1.png\" width=\"600\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DepthCloud:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Ball seen in DepthCloud element",
    "image": true,
    "name": "perception_project_pcl2",
    "width": "10cm"
   },
   "source": [
    "<img src=\"img/perception_project_pcl2.png\" width=\"600\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Camera-Optic frame problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, finally, a topic that is a very common problem in perception is the reference frames of the sensors. It's really common that the frame of the place where the sensor is mounted is inverted from the sensor frame. This is because the sensor readings might be inversed, and in order to visualise data in a coherent way, the sensor frames are reversed. This could also be the case if the processing algorithm, for some reason, outputs data inversely. For all this, you have to be careful to see how the reference frames are mounted and data is visualised in RVIZ.<br>\n",
    "Here you have how the camrea_fram (the physical frame where the camera is mounted) and the optical_frame (the frame used by the sensor) are oriented. See that they are inverted. This is because the sensor, in this case, works that way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Camera Frame:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Camera Frame",
    "image": true,
    "name": "perception_project_cameraframe",
    "width": "10cm"
   },
   "source": [
    "<img src=\"img/perception_project_cameraframe.png\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optical Frame:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Optical Frame",
    "image": true,
    "name": "perception_project_opticalframe",
    "width": "10cm"
   },
   "source": [
    "<img src=\"img/perception_project_opticalframe.png\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Elements of the Simulated World"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the world in which Aibo is spawned, you will find the following elements that will be used for the various tasks to be performed in this project:<br>\n",
    "\n",
    "* Line following Map: The floor has patterns painted on it.<br>\n",
    "They will be used for the line following tasks and also to navigate around.<br>\n",
    "For example, to find the AiboBone toy, you will have to follow the white lines and the coloured stars to be able to find it with perception. You will start on the black star and work your way from there."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Line Map",
    "image": true,
    "name": "perception_project_map",
    "width": "10cm"
   },
   "source": [
    "<img id=\"fig-E.1\" src=\"img/perception_project_map.png\" width=\"300\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Timmy: The boy that is moving around. He will be used for the people tracking exercise. He moves in a circle indefinitely."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Boy named Timmy",
    "image": true,
    "name": "perception_projec_timmy",
    "width": "10cm"
   },
   "source": [
    "<img id=\"fig-E.2\" src=\"img/perception_projec_timmy.png\" width=\"300\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Olive: It's the girl sitting near the green star. She stays static. She will be used for the face perception exercises."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Girl named Olive",
    "image": true,
    "name": "perception_project_olive",
    "width": "10cm"
   },
   "source": [
    "<img id=\"fig-E.3\" src=\"img/perception_project_olive.png\" width=\"300\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Aibo Ball: This is the classic ball used by the original Aibo. It will be used mainly for the blob tracking exercise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Aibo ball",
    "image": true,
    "name": "perception_projec_ball",
    "width": "10cm"
   },
   "source": [
    "<img id=\"fig-E.4\" src=\"img/perception_projec_ball.png\" width=\"300\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Aibo Bone: Also an element shipped in from the original Aibo. It will be used for object recognition. This model was extracted from the <a href=\"https://www.thingiverse.com/thing:2141985\">model,</a> generated by <a href=\"https://www.thingiverse.com/dani_b/about\">Dany_b</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Aibo bone",
    "image": true,
    "name": "perception_projec_bone",
    "width": "10cm"
   },
   "source": [
    "<img id=\"fig-E.5\" src=\"img/perception_projec_bone.png\" width=\"300\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Project exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After this description, it's time to work.<br>\n",
    "Here you have the list of exercises proposed to practice what you have learned about Basic perception with ROS.<br>\n",
    "You have one for each unit, plus some extras:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unit 1: Vision basics exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You have to make the Aibo Robot look for its pink ball and get to its location.**<br>\n",
    "When you are within less than 0.2 meters, you have to make Aibo go into a pointing pose, or any other pose that indicates where the ball is to anyone that sees it.<br>\n",
    "It could be pointing the head to the ball, or a paw, whatever you want. Use only the RGB image for this exercise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unit 2: Follow a line exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You have to make the Aibo Robot follow a white line on the floor until it reaches a green star, which represents the wireless charging zone**.<br>\n",
    "Then, flex four limbs, which would make the real Aibo lower its body to touch the ground.<br>\n",
    "For this exercise, you will have to use the OpenCV that you learned in the line following unit. You will have to track:\n",
    "\n",
    "* The white path.\n",
    "* The different colour stars to know if you are on track, or recover when lost.\n",
    "* The green star to know where to stop and lower your body."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unit 3: Flat surface and object recognition exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You have to make the Aibo Robot recognise its nice bone and go to it.**<br>\n",
    "Once you are there, you will have to make the gesture of the Aibo peeing on it, as the original one did with the bone because it was used as a toy fire hydrant. See this <a href=\"https://www.youtube.com/watch?v=CEU0R67tUiQ\">video</a>.<br>\n",
    "For this, you will have to do **3d object recognition.**<br>\n",
    "But, as you know, the 3D object recognition doesn't work unless you are really close to the object. So, you will need to use the line following to navigate to the white star, where the Aibo Bone is, first.<br>\n",
    "You might also need to do pink blob tracking to get closer to the Aibo bone.<br>\n",
    "And it will be then that you can start the object recognition and localise the object.<br>\n",
    "Finally, you will have to execute a sequence of movements mimicking the video, while maintaining the relative position and orientation based on the latest object tracking.<br>\n",
    "As for the surface detection, you will only have to detect the ground and be able to indicate the distance to the ground. It has to be **published in a topic** that gives that information. This would be very useful for the real robot picking up objects from the ground."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unit 4: People perception exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You have to make the Aibo Robot search for human faces, get to their position, and when Aibo is close enough (around 1 meter), wag its tail.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It has to also lock its view so that Aibo is always looking at the person detected when it's near, closer than 1.0 meter.<br>\n",
    "You will probably have to use some kind of TF tranformations to tranform the face location detection data to the camera frame and move the head accordingly.<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You make it as complicated as you want. You can just move the headpan and tilt of the Aibo head, or you can even use the TF structure to make the calculation. Then, go to the next human and repeat the process.<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, you have two people :Olive and Timmy. Olive is static, so to search for her, you will just have to follow lines until you find the green star and then look for human faces. In Timmy's case, you will have to do the same, but will have to bear in mind that he's moving, so it will be more challenging."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unit 5: Face recognition exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You have to look for your owner, Olive.**<br>\n",
    "In this scene, there will be two humans. You will have to find Olive and give her paw. If it's Timmy, you will have to continue until you find Olive. Here you have the photos for the face recogniser:<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Olive:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Girl named Olive face closeup",
    "image": true,
    "name": "perception_project_olivezoom",
    "width": "10cm"
   },
   "source": [
    "<img src=\"img/perception_project_olivezoom.png\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Timmy:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "caption": "Boy named Timmy face closeup",
    "image": true,
    "name": "perception_project_timmyzoom",
    "width": "10cm"
   },
   "source": [
    "<img src=\"img/perception_project_timmyzoom.png\" width=\"400\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unit 6: People tracking exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You have to make the Aibo Robot follow the standing person, get within 3 meters and wigle the tail.**<br>\n",
    "For this, keep in mind that you won't have laser readings. But, there is a way of using the PointCloud data as laser readings. You will have to figure out how if you want to use the leg detector.<br>\n",
    "Although, you don't actually need the leg detector; with the other detectors, you can track people.<br>\n",
    "It's up to you to decide which approach you want to try."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also the **standing person will be moved by you** with the **keyboard**. Once you have all the simulation loaded, just launch in a webshell the following launch:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "roslaunch person_sim move_person_standing.launch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should get somthing similar to this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/aibo_peopletracking2.gif\" width=\"600\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extra exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Create an action server that, when called, you can ask him to do any of the tasks developed earlier. This would be the action server used by speech recognition to perform tasks through voice. In this case, it will have to be an action message, like so:<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#goal, a string that states which action to perform like: look_for_humans, look_for_olive, follow_timmy, search_ball, search_bone and go_to_charge.\n",
    "std_msgs/String action_to_perform\n",
    "---\n",
    "#result, it should describe the result\n",
    "std_msgs/String action_result\n",
    "---\n",
    "#feedback, should describe what is happening in that moment in the action that is being performed\n",
    "std_msgs/String action_feedback"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Create a launch file that launches RVIZ with your custom configuration to see everything you need"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Congratulations! You finished the Perception Project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would be delighted to know what different strategies you came up with. If you are interested in posting it, tweet it to us at <a href=\"\">@_TheConstruct_</a>. We're excited to see what you did!"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  },
  "latex_metadata": {
   "chapter": "7 - Project",
   "chapter_title": "Unit 7. Project",
   "course_title": "ROS PERCEPTION IN 5 DAYS"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}