OD_tutorial.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "OD_tutorial.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Li8F4XxaFF2u",
        "colab_type": "text"
      },
      "source": [
        "# Object Detection on CRAL\n",
        "\n",
        "\n",
        "Tutorial on how to use CRAL library for object detection using RetinaNet and YOLOv3."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yOH6qQpbMeSs",
        "colab_type": "text"
      },
      "source": [
        "Backbones supported for RetinaNet: `ResNet50` `ResNet101` `ResNet152`\n",
        "\n",
        "Backbones supported for YOLOv3: `DarkNet53`\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cuATNqVdHG91",
        "colab_type": "text"
      },
      "source": [
        "### 0. **Getting started** \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dTuMdzhSHyDa",
        "colab_type": "text"
      },
      "source": [
        "0.1 Install CRAL"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nmnXugucHcf8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install cral"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XreK2CgSH41q",
        "colab_type": "text"
      },
      "source": [
        "0.2 Create an account on [track.segmind.com](https://track.segmind.com/) to track runs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z7SPYhczIAjw",
        "colab_type": "text"
      },
      "source": [
        "0.3 Configure CRAL\n",
        "\n",
        "\n",
        "CRAL needs to be configured with `cral config` after installing the library. After running this, you will be asked for the email and password.\n",
        "Only after this will the tracking of the runs be possible."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dyqnvf3IHdR9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!cral config"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZYr8FMbytNcp",
        "colab_type": "text"
      },
      "source": [
        "0.4 Setting up dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PYfKsg45vLbb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Downloading and setting up data\n",
        "\n",
        "from zipfile import ZipFile\n",
        "import tempfile, os\n",
        "import tensorflow as tf\n",
        "\n",
        "zip_url = 'https://segmind-data.s3.ap-south-1.amazonaws.com/edge/data/aerial-vehicles-dataset.zip'\n",
        "path_to_zip_file = tf.keras.utils.get_file(\n",
        "    'aerial-vehicles-dataset.zip',\n",
        "    zip_url,\n",
        "    cache_dir= tempfile.gettempdir(), \n",
        "    cache_subdir= '',\n",
        "    extract=False)\n",
        "\n",
        "directory_to_extract_to = os.path.join(tempfile.gettempdir(),'aerial-vehicles-dataset')\n",
        "with ZipFile(path_to_zip_file, 'r') as zip_ref:\n",
        "    zip_ref.extractall(directory_to_extract_to)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Shl8RJpLGpon",
        "colab_type": "text"
      },
      "source": [
        "### 1. **Experiment tracking** \n",
        "\n",
        "\n",
        "This is a mandatory step to be run before training. It provides you statistics on training data and logs all the metrics, losses etc. in an online dashboard making it easier to evaluate the model and compare between different runs, which can help you set the right hyperparameters. \n",
        "\n",
        "Generate experiment ID on [track.segmind.com](https://track.segmind.com/) to track your run and refer [here](https://cral.segmind.com/en/api/tracking) for instructions to log custom metadata and artifacts."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8i9SFfdjE684",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from cral.tracking import set_experiment\n",
        "\n",
        "set_experiment('your-experiment-ID')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "XnF18hduLx3g"
      },
      "source": [
        "###2. **Initiate object detection pipeline**\n",
        "\n",
        "\n",
        "This initiates the classification pipeline. It also creates project related files that are needed later during training and inference."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P19ZmFbTLxeE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from cral.pipeline import ObjectDetectionPipe\n",
        "\n",
        "pipe= ObjectDetectionPipe()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1srDknJJK0L8",
        "colab_type": "text"
      },
      "source": [
        "###3. **Add data**\n",
        "\n",
        "\n",
        "This parses the dataset once to generate metadata and versions the data. \n",
        "\n",
        "You can use any the supported formats: `coco`, `pascal_voc` or `yolo` according to your dataset.\n",
        "\n",
        "If you do not have a seperate validation set, you can divide the training dataset into training and validation using *split* argument. Otherwise, provide the path to the validation images and annotations."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xi9UYmfZMxUn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "pipe.add_data(\n",
        "    train_images_dir= os.path.join(directory_to_extract_to, 'images'),\n",
        "    train_anno_dir= os.path.join(directory_to_extract_to, 'annotations','pascalvoc_xml'),\n",
        "    annotation_format= 'pascal_voc',\n",
        "    split= 0.2)                                           "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1GMJcv8VQuoH"
      },
      "source": [
        "###4. **Lock data**\n",
        "\n",
        "This parses the data and makes Tf-records. Tf-records are part of the [tf.data](https://www.tensorflow.org/guide/data) API which makes the data ingestion into the model faster and hence reduces the training time.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "yJpkaSnaQuoU",
        "colab": {}
      },
      "source": [
        "pipe.lock_data()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xpOkAb8hPjl1",
        "colab_type": "text"
      },
      "source": [
        "###5. **Set algorithm**\n",
        "\n",
        "You can instantiate different object detection models here.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mhvC7IcJNyWe",
        "colab_type": "text"
      },
      "source": [
        "###5.1. For RetinaNet\n",
        "\n",
        "\n",
        "You can use any of the supported backbones for RetinaNet. Do not forget to change **resnet50** to the backbone of your choice. \n",
        "\n",
        "\n",
        "You can find the list of supported backbones and accepted arguments [here](https://cral.segmind.com/api/models/ObjectDetection#retinanet). Also, take a look at how the hyperparamets work and how you can change them [here](https://blog.segmind.com/an-introduction-to-retinanet-and-how-we-make-it-easier-to-use/)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SATJJQImQVrO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from cral.models.object_detection import RetinanetConfig\n",
        "\n",
        "retinanet_config= RetinanetConfig()\n",
        "\n",
        "pipe.set_algo(\n",
        "    feature_extractor= 'resnet50',  \n",
        "    config= retinanet_config)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OIOTm-OZOYnh",
        "colab_type": "text"
      },
      "source": [
        "###5.2. For YOLOv3\n",
        "\n",
        "\n",
        "Currently CRAL supports DarkNet53 as a backbone for YOLOv3.\n",
        "\n",
        "\n",
        "You can find the list of accepted arguments for configuring your YOLOv3 model [here](https://cral.segmind.com/api/models/ObjectDetection#yolov3)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iBeVL9XIO2vU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from cral.models.object_detection import YoloV3Config\n",
        "\n",
        "yolo_config= YoloV3Config()\n",
        "\n",
        "pipe.set_algo(\n",
        "    feature_extractor= \"darknet-53\",\n",
        "    config= yolo_config)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oJOHEYjPaHGy"
      },
      "source": [
        "###6. **Train**\n",
        "\n",
        "Now that you have set the model, you can start training.\n",
        "Using train is very similar to the *fit* function in tensorflow.keras and it accepts many of the same arguments like epochs, batch_size etc. All your loss and metrics will be streamed from here onto [track.segmind.com](http://track.segmind.com/) automatically where you can see and evaluate your model easily."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "BksXpmWQaHG3",
        "colab": {}
      },
      "source": [
        "pipe.train(\n",
        "    num_epochs= 10,\n",
        "    batch_size= 4,\n",
        "    snapshot_prefix= 'test',\n",
        "    snapshot_path= tempfile.gettempdir(),\n",
        "    snapshot_every_n= 2)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ltp4Fq5fkZcG"
      },
      "source": [
        "###7. **Prediction**\n",
        "\n",
        "After training you can use the model to get predictions on images that maybe or may not be a part of your initial dataset.\n",
        "\n",
        "Calling `prediction_model` will give you a function that can be used to get predictions on images."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "7JMdQYmbkq8I",
        "colab": {}
      },
      "source": [
        "checkpoint_file= '/tmp/test_final'\n",
        "prediction_func= pipe.prediction_model(checkpoint_file= checkpoint_file)\n",
        "\n",
        "img_path= os.path.join(tempfile.gettempdir(), 'aerial-vehicles-dataset', 'images', 'DJI-00760-00022.jpg')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sZjq3VfWkq7-"
      },
      "source": [
        "###7.1. For RetinaNet"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "RXMsVSP9kZcT",
        "colab": {}
      },
      "source": [
        "from cral.models.object_detection import annotate_image\n",
        "from PIL import Image\n",
        "import numpy as np\n",
        "\n",
        "# make prediction\n",
        "bboxes, scores, labels= prediction_func(img_path)\n",
        "\n",
        "# draw predictions on image and save\n",
        "image_array= np.array(Image.open(img_path))\n",
        "drawn_image= annotate_image(image_array, bboxes, scores, labels, threshold=0.5)\n",
        "drawn_image.save('/tmp/image_retinanet.jpeg')\n",
        "\n",
        "# display result\n",
        "from IPython.display import Image, display\n",
        "display(Image('/tmp/image_retinanet.jpeg'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "OkRYUQqOQhWX"
      },
      "source": [
        "###7.2. For YOLOv3"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ZXzHCxTtQhWd",
        "colab": {}
      },
      "source": [
        "from cral.models.object_detection.yolov3_2.utils import annotate_image\n",
        "\n",
        "# make prediction\n",
        "orignal_image, predictions= prediction_func(image_path= img_path)\n",
        "\n",
        "# draw predictions on image, save and display it\n",
        "annotate_image(orignal_image, predictions, save_path= '/tmp/image_yolo.jpeg')"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}