4-object-detection-segmentation/4-intro-segmentation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "intro-segmentation.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "<a name='0'></a>\n",
        "\n",
        "# Introduction to Pixel-Level Recognition Tasks"
      ],
      "metadata": {
        "id": "4L4NTKueAQPp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Pixel-level recognition or dense prediction tasks are probably most popular computer vision tasks after image classification and object detection. In this notebook, we will learn about pixel level-recognition tasks which are semantic segmentation, instance segmentation, panoptic segmentation and many more other tasks."
      ],
      "metadata": {
        "id": "2GLiRaAl2TyJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Outline***:\n",
        "\n",
        "* [1. Pixel-level Recognition: Overview](#1)\n",
        "- [2. Semantic Segmentation](#2)\n",
        "  - [2.1 Semantic Segmentation Approaches](#2-1)\n",
        "      - [2.1.1 Fully Convolutional Networks](#2-1-1)\n",
        "      - [2.1.2 Deconvolution Networks for Semantic Segmentation](#2-1-2)\n",
        "      - [2.1.3 DeepLab Network for Semantic Segmentation](#2-1-3)\n",
        "- [3. Instance Segmentation](#3)\n",
        "  - [3.1 Instance Segmentation Approaches](#3-1)\n",
        "      - [3.1.1 Mask R-CNN for Instance Segmentation](#3-1-1)\n",
        "- [4. Panoptic Segmentation](#4)\n",
        "- [5. Pose Estimation](#5)\n",
        "- [6. Modern Applications of Segmentation Tasks](#6)\n",
        "- [7. Final Notes](#7)\n",
        "- [8. References and Further Learning](#8)"
      ],
      "metadata": {
        "id": "k9B0Bizd0DJe"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='1'></a>\n",
        "## 1. Pixel-level Recognition: Overview"
      ],
      "metadata": {
        "id": "lwk0TGVJ2QgJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the previous notebook, we learned about object detection, a task of recognizing the objects in the image and drawing the bounding boxes around them. Image segmentation tasks or pixel-level recognition tasks operate on pixel level.\n",
        "\n",
        "Semantic segmentation is one of the popular image segmentation tasks and hence most people use image segmentation to refer to semantic segmentation. Semantic segmentation is the task of assigning every pixel to a semantic label. It's also called per-pixel classification, or as [Jitendra Malik](https://en.wikipedia.org/wiki/Jitendra_Malik) said, it's just `entitification`. Other types of segmentation are instance segmentation that segment each separate object instance with a mask(or segment object instances) and panoptic segmentation that combines both instance segmentation and semantic segmentation. We will discuss these tasks in details in the next sections.\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1b51Jrjo6n7Syq1BrRKg6bbJgkiZ15Yqi)"
      ],
      "metadata": {
        "id": "GoT5BiAqHlhp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='2'></a>\n",
        "\n",
        "## 2. Semantic Segmentation"
      ],
      "metadata": {
        "id": "FerESS-9cwkw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Semantic segmentation is the first popular segmentation tasks that is used to label(semantically) each pixel in the image. In essence, semantic segmentation is pixel classification.\n",
        "\n",
        "In image classification, we primarily deal with recognizing image category. In object detection, the deals are object class labels and their spatial extents(bounding boxes). In semantic segmentation, none of those. We are merely interested in classifying pixels that form a particular semantic object in the image.\n",
        "\n",
        "Quoting Justin Johnson, semantic segmentation does not distinguish the instances of the categories. It simply labels all the pixels.\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1OS7TIp1DbyKBTq8Zgwigj4lhwXX0CZdu)\n",
        "\n",
        "Semantic segmentation adresses **stuffs** or uncountable/unstructured parts of the image such as sky, road, river, etc...\n",
        "\n",
        "The evaluation metric of semantic segmentation is [mean Intersection-Over-Union(mIoU)](https://keras.io/api/metrics/segmentation_metrics/)."
      ],
      "metadata": {
        "id": "h2QVo9iaedCe"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='2-1'></a>\n",
        "\n",
        "### 2.1 Semantic Segmentation Approaches"
      ],
      "metadata": {
        "id": "I-lWW7on6b8O"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "One of the earliest approaches of semantic segmentation based on deep learning methods was sliding window approach. The idea of sliding window was to use ConvNets to extract features from image patches/windows and predict the class label(pixel-wise) of each patch or window. However, sliding window approach is not efficient because you have to run ConvNet for each image patch or window and these ConvNets operating on image patches didn't share weights.\n",
        "\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1lRbaz_RizMp4CmgABdy1KneEbVPVqsMK)"
      ],
      "metadata": {
        "id": "Jdjrm3XEvdzR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Sliding window segmentation approach is no longer used. Let's review some popular semantic segmentation architectures starting with Fully Convolutional Neural Networks."
      ],
      "metadata": {
        "id": "B2qrOF8lgeke"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='2-1-1'></a>\n",
        "\n",
        "#### 2.1.1 Fully Convolutional Networks "
      ],
      "metadata": {
        "id": "b6q54OgKg_Ak"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Another old semantic segmentation approach is [Fully Convolutional Networks by Jonathan et al.](https://arxiv.org/abs/1411.4038) or FCN in short.\n",
        "\n",
        "The idea of Fully Convolutional Neural Networks is to stack convolutional layers after another with some pooling layers in between and without any fully connected layers. FCN is trained end to end as a classification network operating on input image pixels with per-pixel cross entropy as a loss function.\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1u2m7H_JfiBVfQslhP313txyiP2hZJFb0)\n",
        "\n",
        "The downside however is, it can be expensive to train fully convolutional neural networks as most real-world images have high resolution. Also, as the same spatial size(weight & height of image) must be mantained throughout the whole network, we may use many convolutional layers to get better performance. FCN can use VGG, GoogLeNet and ResNet networks but it's rarely used to day.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "jvydRcd6_epz"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='2-1-2'></a>\n",
        "\n",
        "#### 2.1.2 Deconvolution Networks for Semantic Segmentation"
      ],
      "metadata": {
        "id": "5ylHu1L8tUkQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "A better semantic segmentation network based on FCN was presented by Noh et al. in [Learning Deconvolution Network for Semantic Segmentation, DeconvNet](https://arxiv.org/abs/1505.04366). DeconvNet is made of two main parts that are downsampling part and upsampling part. The downsampling part is made of convolutional and pooling layers and upsampling part is the opposite. It is made of deconvolution(or transposed convolution) and unpooling layers. Below image illustrates DeconvNet.\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1kyyjMMe5PgmFBHauQ4f_0-cyUUM3qmAz)\n",
        "\n",
        "Deconvolution and unpooling are the exact opposite of convolution and pooling and they are used for object reconstruction purpose. Below image illustrates unpooling and deconvolution but for more details behind them, you can read the [paper](https://arxiv.org/abs/1505.04366).\n",
        "\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1eopycyFmvh60M4WrkINyK3PSwL7Yiuab)\n",
        "\n",
        "Most semantic segmentation networks that are used to day typically use some forms of deconvolution. One of the most popular semantic segmentation networks that is also made of downsampling and upsampling parts is [U-Net](https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/) which is mostly used in medical images analysis(I also learned that U-Net was used in high resolution [building detection](https://arxiv.org/abs/2107.12283) and just recently, Google AI scientists also used a modified U-Net for [photo-realistic image generation](https://imagen.research.google)).\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1-jcgMTzrpGOKwqO-FzfulD5x1WKRZilq)\n",
        "\n",
        "Another semantic segmentation architecture that is not so popular but has the same form of previous architectures is [SegNet](https://arxiv.org/abs/1511.00561) which claims to be efficient in terms of memory and time. Its encoder(downsampling part) is VGG with no fully connected layers.\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=17Fak-cQcHMcbG2ll1mkNzvxeGmoGpgzL)\n",
        "\n",
        "All the architectures we discussed here are typically referred to as encoder-decoder networks. Encoder downsamples the input image, decoder upsamples the encoder output. Before we wrap up semantic segmentation approaches, let's see one of the most popular and modern semantic segmentation networks: [DeepLab](https://arxiv.org/pdf/1412.7062.pdf), first introduced in 2015 but undergone many improvements year after year."
      ],
      "metadata": {
        "id": "fFQZnuHaoawr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='2-1-3'></a>\n",
        "\n",
        "#### 2.1.3 DeepLab Network for Semantic Segmentation"
      ],
      "metadata": {
        "id": "igyc8Tbq4y4k"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "DeepLab is one of the modern semantic segmentation networks that has undergone many improvements since it's [initial releases](https://arxiv.org/abs/1606.00915)!\n",
        "\n",
        "DeepLab introduced the altrous convolution or dilated convolution and fully connected CRFs(Conditional Random Fields) for semantic segmentation. Atrous convolution is a special type of convolution that has upsampled filters. Atrous convolution improves the resolution of feature maps. As DeepLab [paper]((https://arxiv.org/abs/1606.00915) noted, \"*Compared to regular convolution with larger filters, atrous convolution allows us to effectively enlarge the field of view of filters without increasing the number of parameters or the amount of computation.*\"\n",
        "\n",
        "CRFs(Conditional Random Fields) was used to improve the ability of the model to capture fine details and to improve the pixel-level classification as whole. Below image illustrate the DeepLab segmentation system. The next image shows the difference between normal convolution and atrous convolution.\n",
        "\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1SRyddajed5AVfD92k9RkWGa82sqjQo1F)\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=13qZZIrHnxSfewsGqgkXOQIgw50dF8QnH)\n",
        "\n",
        "For more about [DeepLab segmentation systems](http://liangchiehchen.com/projects/DeepLab.html#attention%20model), check all its [official GitHub repository](https://github.com/google-research/deeplab2) and the following subsequent papers:\n",
        "\n",
        "- [Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs](https://arxiv.org/abs/1412.7062)\n",
        "* [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://arxiv.org/pdf/1606.00915.pdf)\n",
        "\n",
        "- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://arxiv.org/abs/1802.02611)\n",
        "- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://arxiv.org/abs/1706.05587)\n",
        "- [DeepLab2: A TensorFlow Library for Deep Labeling](https://arxiv.org/abs/2106.09748)\n",
        "\n",
        "You can also watch this fantastic YouTube [video](https://www.youtube.com/watch?v=HTgvG57JFYw) about DeepLab by MLT Artificial Intelligence. It will help you to understand pretty much what you need to know about DeepLab systems.\n",
        "**********************\n",
        "\n",
        "To summarize semantic segmentation approaches, most segmentation models falls into the schemes that are illustrated well below.\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1GsaHsDnJem7LQiBhR3Rn2lYpAkmkRToc)"
      ],
      "metadata": {
        "id": "vbdIecdxJSQZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='3'></a>\n",
        "\n",
        "## 3. Instance Segmentation"
      ],
      "metadata": {
        "id": "NlAmOSyRE1T0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Instance segmentation is also a type of dense prediction tasks(or pixel-level recognition). Instance segmentation deals with recognizing all objects in an image and identifying pixels that belongs to those objects. Instance segmentation can also be seen as ***delineating each object with a bounding box(CC:Alexander Kirillov).***\n",
        "\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1nTITrpqU0kLLZdgoyUWC0DwPKmh67BSB)"
      ],
      "metadata": {
        "id": "tWIq3ElvoVZo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "Instance segmentation adresses **things** or countable objects that are in image such as people, animals, etc...If you look at the above image on instance segmentation, you can see that the things are segmented while stuffs(semantic segmentation) are not.\n",
        "\n",
        "The modern instance segmentation algorithms are entirely based on deep learning. In the next section, we will review some instance segmentation approaches."
      ],
      "metadata": {
        "id": "sAw2zq903PWW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='3-1'></a>\n",
        "\n",
        "### 3.1 Instance Segmentation Approaches"
      ],
      "metadata": {
        "id": "szn4ZDX74cIZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Most instance segmentation algorithms fall into two approaches: top-down and bottom-up approaches.\n",
        "\n",
        "Top-down approaches involve object detection and segmentation. They first detect the object bounding boxes, and afterwards perform binary segmentation for each bounding to remain with delineated object instances. One of the most popular instance segmentation approaches of this category is [Mask R-CNN](https://arxiv.org/abs/1703.06870)(more on this later). Other examples include [PANet - Path Aggregation Network for Instance Segmentation](https://arxiv.org/abs/1803.01534)\n",
        "\n",
        "Bottom-up approaches start with pixels classification, and and then group similar pixels into object instances. Example of algorithms that follows such approach are [InstanceCut](https://arxiv.org/abs/1611.08272) and [DWT(Deep Watershed Transform for Instance Segmentation)](https://arxiv.org/abs/1611.08303).\n",
        "\n",
        "There are also other good performing instance segmentation approaches that don't follow any of those approaches. Examples include [INSTA-YOLO: Real-Time Instance Segmentation](https://arxiv.org/abs/2102.06777), [TensorMask: A Foundation for Dense Object Segmentation](https://arxiv.org/abs/1903.12174), [YOLACT: Real-time Instance Segmentation](https://arxiv.org/abs/1904.02689v2), [SOLO: Segmenting Objects by Locations](https://arxiv.org/abs/1912.04488v3), etc....Architectures such as INSTA-YOLO, YOLOACT, [RetinaMask](https://arxiv.org/pdf/1901.03353.pdf), and SOLO are single-stage instance segmentation architectures(inspired by single-stage detectors).\n",
        "\n",
        "The above categories were discussed in details in this fantastic CVPR 2020 [talk](https://www.youtube.com/watch?v=QCtHGT68RIU).\n",
        "\n",
        "The evaluation metric of instance segmentation is Average Precision(AP).\n",
        "\n",
        "Mask R-CNN is one of the instance segmentation architectures that really stands out among all other architectures. So, let's review it in details."
      ],
      "metadata": {
        "id": "dzCZ9sX49Cdr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "<a name='3-1-1'></a>\n",
        "\n",
        "#### 3.1.1 Mask R-CNN for Instance Segmentation"
      ],
      "metadata": {
        "id": "9Ui9S4QcQUlD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Mask R-CNN is one of the earliest and powerful instance segmentation architectures that is conceptually` simple, flexible and general`.\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=14SPTwI4tS7OFa-INsXlLd_FXniLDFWFV)\n",
        "\n",
        "From the architecture design standpoint, Mask R-CNN adds an extra branch(or head) for predicting the object masks in parallel to [Faster R-CNN](https://arxiv.org/abs/1506.01497v3) bounding box detection branch. Being a general instance segmentation architecture, it can also be extended to other tasks such as human pose estimation.\n",
        "\n",
        "Mask R-CNN is really an intuitive architecture. Below are some of its results. More details on the architecture can be found in the [paper](https://arxiv.org/abs/1703.06870) that is written well.\n",
        "\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1DCsBmT1DDmQqQJ3QRpgPeL205Aee4fjX)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "haDcwIrI9WqK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='4'></a>\n",
        "\n",
        "## 4. Panoptic Segmentation\n"
      ],
      "metadata": {
        "id": "KzGT2pHGCVuf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "So far, we have seen two most popular dense prediction tasks which are semantic segmentation and instance segmentation. But if we try to think about those two tasks and how they contribute to scene understanding, we actually see they are loosely connected and they don't help us understand the complete visual scene.\n",
        "\n",
        "Semantic segmentation classifies every pixel with a semantic label(or according to Richard Szeliski, semantic segmentation answers the question *what stuff does each pixel corresponds to*). Because the prediction is per-pixel, we can understand the scene layout, but we don't have a crue of individual objects that are in an image. On the otherhand, instance segmentation segments each object instance that is in an image but it also doesn't say much about the layout of the visual scene.\n",
        "\n",
        "Combining both semantic and instance segmentation helps us understand the scene better, and that's in fact a panoptic segmentation.\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1euvCFGQoWHpMf9pj8rdrKs1W6QRnIUVL)"
      ],
      "metadata": {
        "id": "GxLslxlcHkqN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Panoptic segmentation is a combination of semantic segmentation and instance segmentation. It helps us understand the **things**(countable objects that are in image such as people and animals) and **stuffs**(what remains after removing objects in image or unstructured parts of image such as sky, road).\n",
        "\n",
        "Panoptic segmentation refers to assigning a semantic label and object instance label to every pixel in the image. Quoting [Panoptic Segmentation](https://arxiv.org/pdf/1801.00868.pdf) paper, *the definition of `panoptic` is \"including everything visible in one view\", in our context panoptic refers to a unified, global view of segmentation.*\n",
        "\n",
        "Panoptic segmentation task was introduced in 2019. More about the task, its evaluation metric(Panoptic Quality, PQ) and other related thing can be learned through its [paper](https://arxiv.org/abs/1801.00868) that is well written.\n",
        "\n",
        "Some of the modern panoptic segmentation approaches are [Panoptic-DeepLab](https://arxiv.org/abs/1911.10194v3), [DeeperLab](https://arxiv.org/abs/1902.05093), [Panoptic FPN](https://arxiv.org/abs/1901.02446v2), [DETR](https://arxiv.org/abs/2005.12872v3) and [many more](https://paperswithcode.com/task/panoptic-segmentation)."
      ],
      "metadata": {
        "id": "ns2EkRuqC_LF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='5'></a>\n",
        "\n",
        "## 5. Pose Estimation"
      ],
      "metadata": {
        "id": "RfbfZkOmoPAD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Pose estimation is also another computer vision task that relies on segmentation methods.\n",
        "\n",
        "Pose estimation involves detecting the objects and localizing their keypoints or points of interests. Human pose estimation is a popular pose estimation or keypoint detection task that deals with detecting and localizing the keypoints of human such as head, shoulder, hands, legs, etc...\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=15alwCgO0ww5QjB3y7RUkKk19qrtvINKG)\n",
        "\n",
        "Some of the modern pose estimation approaches are [Mask R-CNN](https://arxiv.org/pdf/1703.06870.pdf)(yeah, Mask R-CNN is general architecture, you just add a keypoint head and it can detects keypoints), [OpenPose](https://arxiv.org/abs/1812.08008v2), [DensePose](https://arxiv.org/pdf/1802.00434v1.pdf), and [many more others](https://paperswithcode.com/task/keypoint-detection). Below image shows Mask R-CNN on human pose estimation.\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1fZIcOWeBKZ539xxSTMng7elHwHS6OEfj)\n",
        "\n",
        "\n",
        "[DensePose](http://densepose.org) that we noted above is also one of the popular pose estimation networks. Below image shows its results.\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1FodoiWkykvHVcHUOdIcg8EQdK6tV6aSS)\n"
      ],
      "metadata": {
        "id": "d5hg8BvoqB5I"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='6'></a>\n",
        "\n",
        "## 6. Modern Applications of Segmentation Tasks"
      ],
      "metadata": {
        "id": "kTKqF8JT2XG9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Image segmentation tasks have numerous real-world applications. In this section, we will glimpse at some few areas that make use of image segmentation methods.\n",
        "\n",
        "Medical image analysis is one of the most popular areas that use image segmentation networks for analyzing medical images such as [X-ray](https://en.wikipedia.org/wiki/Radiography), [CT(computed tomography) scans](https://en.wikipedia.org/wiki/CT_scan), [PET scans(A positron emission tomography)](https://en.wikipedia.org/wiki/Positron_emission_tomography), [MRI](https://en.wikipedia.org/wiki/Magnetic_resonance_imaging)(magnetic resonance imaging). The most popular applications of segmentation in medicine are lesion segmentation, [brain tumor segmentation](https://arxiv.org/pdf/1505.03540v3.pdf), liver segmentation, cell segmentation, etc... More about medical image segmentation can be learned [here](https://paperswithcode.com/task/medical-image-segmentation). This paper provides a great overview of deep learning in [medical image analysis](https://arxiv.org/pdf/1702.05747.pdf).\n",
        "\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1uspssf1Lw9Phec4txGxTXrpRXZzcecWg)\n",
        "\n",
        "Image segmentation is also used in self-driving cars or autonomous vehicles in general.\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1lZM87xkemDeLtHMgPUwDQAIdc9k77bZb)\n",
        "\n",
        "Lastly, modern phones camera use segmentation to enhance image quality. This is most notable in portraits where the background(stuff) is blurred and the focus is given to the person facing the camera.\n",
        "\n",
        "![image](https://drive.google.com/uc?export=view&id=1vyUDZtHQJikqLkhJEwQPeIKbBTOpDXcH)\n"
      ],
      "metadata": {
        "id": "bmBRvay0Be0J"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='7'></a>\n",
        "\n",
        "## 7. Final Notes"
      ],
      "metadata": {
        "id": "-Geli1qHeq0N"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this notebook, we learned pixel-level recognition, dense prediction or image segmentation tasks. Those tasks operate on pixe-level. \n",
        "\n",
        "The three most important segmentation tasks are semantic segmentation(classifies every pixel with a semantic label), instance segmentation(segments every object instance), and panoptic segmentation which combines both semantic and instance segmentation.\n",
        "\n",
        "About tools, most modern object detection tools also provides segmentation models. Example of tools: [Detectron2](https://github.com/facebookresearch/detectron2), [OpenMMLab(mmsegmentation\n",
        ")](https://github.com/open-mmlab/mmsegmentation), [Deeplab2](https://github.com/google-research/deeplab2), [KerasCV(under-build)](https://github.com/keras-team/keras-cv). For datasets, Paper With Code has an updated list of segmentation datasets([semantic](https://paperswithcode.com/datasets?task=semantic-segmentation&page=1), [instance](https://paperswithcode.com/datasets?task=instance-segmentation), [panoptic](https://paperswithcode.com/datasets?task=panoptic-segmentation&page=1))."
      ],
      "metadata": {
        "id": "l7NEMZONwF0i"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "<a name='8'></a>\n",
        "\n",
        "## 8. References and Further Learning"
      ],
      "metadata": {
        "id": "zgd44O_nxYik"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below are the main resources that served as a reference for this notebook. You can use them to learn more. In the next notebooks, we will practice image segmentation.\n",
        "\n",
        "* [Detection and Segmentation Lecture by Justin Johnson](https://www.youtube.com/watch?v=9AyMR4IhSWQ&list=PL5-TkQAfAZFbzxjBHtzdVCWE0Zbhomg7r&index=17)\n",
        "\n",
        "* [Pixel-Level Recognition CVPR 2020 talk by Alexander Kirillov](https://www.youtube.com/watch?v=QCtHGT68RIU)\n",
        "\n",
        "* [Dense Prediction Lecture Slides](https://courses.cs.washington.edu/courses/csep576/20sp/lectures/9_segmentation.pdf)\n"
      ],
      "metadata": {
        "id": "RILxowC8xdaK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## [BACK TO TOP](#0)"
      ],
      "metadata": {
        "id": "Q_oxLN0TBLgK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "1rtk7zPiBP0r"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}