notebook_examples/genome_visualization-oracle_open_data.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "@notebook{genome_visualization-oracle_open_data.ipynb,\n",
    "    title: Visual Genome Repository,\n",
    "    summary: Load visual data, define regions, and visualize objects using metadata to connect structured images to language.,\n",
    "    developed_on: generalml_p38_cpu_v1,\n",
    "    keywords: object annotation, genome visualization, oracle open data\n",
    "    license: Universal Permissive License v 1.0 (https://oss.oracle.com/licenses/upl/)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upgrade Oracle ADS to pick up latest features and maintain compatibility with Oracle Cloud Infrastructure.\n",
    "\n",
    "!pip install -U oracle-ads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oracle Data Science service sample notebook.\n",
    "\n",
    "Copyright (c) 2022 Oracle, Inc. All rights reserved. Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl.\n",
    "\n",
    "***\n",
    "\n",
    "# <font color=red>Visual Genome Repository</font>\n",
    "<p style=\"margin-left:10%; margin-right:10%;\">by <font color=\"teal\">Oracle for Research</font></p>\n",
    "\n",
    "---\n",
    "\n",
    "## Overview:\n",
    "\n",
    "Visual Genome is a dataset, a knowledge base, and an ongoing effort to connect structured image concepts to language which provides a multi-layered understanding of pictures. This dataset contains 15.13 GB of images. Specifically, it has 108,077 images along with annotation of objects, attributes, and relationships within each image.\n",
    "\n",
    "This notebook demonstrates how to download images and objects from Oracle Cloud Infrastructure (OCI) Object Storage, build dataframe from the JSON metadata files, how access the image data, define a region, and finally visualize regions along with descriptions on a chosen image.\n",
    "\n",
    "Developed on [General Machine Learning](https://docs.oracle.com/iaas/data-science/using/conda-gml-fam.htm) for CPU on Python 3.8 (version 1.0)\n",
    "\n",
    "---\n",
    "\n",
    "## Contents:\n",
    "\n",
    "- <a href=\"#intro\">Introduction</a>\n",
    "  - <a href='#data'>Dataset</a>\n",
    "  - <a href=\"#open_data\">What is Oracle Open Data?</a>\n",
    "- <a href='#df'>Building Dataframe for Visual Genome Metadata</a>\n",
    "   - <a href='#build'>Downloading Metadata and Building a Dataframe</a>\n",
    "- <a href='#visualize'>Visualizing Images and Regions</a>\n",
    "   - <a href='#image'>Getting Image Data</a>\n",
    "   - <a href='#region'>Region Data</a>\n",
    "   - <a href='#vze'>Visualize Regions</a>\n",
    "- <a href='#ref'>References</a>\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "Datasets are provided as a convenience. Datasets are considered third-party content and are not considered materials under your agreement with Oracle.\n",
    "    \n",
    "You can access the `Visual Genome` dataset license [here](https://creativecommons.org/licenses/by/4.0/).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import requests\n",
    "\n",
    "from io import BytesIO\n",
    "from matplotlib.patches import Rectangle\n",
    "from PIL import Image as PIL_Image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"intro\"></a>\n",
    "# Introduction\n",
    "\n",
    "<a id='data'></a>\n",
    "## Dataset\n",
    "\n",
    "**Data**: The Visual Genome is a large formalized knowledge representation for visual understanding. It has a complete set of descriptions and question answers that ground visual concepts to language. Each image, in the dataset, has an average of 35 objects, 26 attributes, and 21 pairwise relationships between objects. The objects, attributes, relationships, and noun phrases are canonicalized in regional descriptions and question-answer pairs to [WordNet](https://wordnet.princeton.edu/) synsets. Together, these annotations represent a large and dense dataset of image descriptions, objects, attributes, relationships, and question-answer pairs. The paper [A Hierarchical Approach for Generating Descriptive Image Paragraphs](https://arxiv.org/pdf/1611.06607v1.pdf) provides details on how the data was generated.\n",
    "\n",
    "**Directory Structure**: It consists of two top level directories, `VG_100K`, and `VG_100K_2`. These directories contain all of the jpg images. The JSON and text metadata files are available in the top level directory.\n",
    "\n",
    "**Template**: `https://objectstorage.us-ashburn-1.oraclecloud.com/n/idcxvbiyd8fn/b/visual_genome/o/<filename>`. For example, \n",
    "* Image: https://objectstorage.us-ashburn-1.oraclecloud.com/n/idcxvbiyd8fn/b/visual_genome/o/VG_100K/1592363.jpg\n",
    "* JSON: https://objectstorage.us-ashburn-1.oraclecloud.com/n/idcxvbiyd8fn/b/visual_genome/o/qa_objects.json.zip\n",
    "\n",
    "**Data Availability**: All data is available from the [Visual Genome repository](https://opendata.oraclecloud.com/ords/r/opendata/opendata/details?data_set_id=1), which is part of [Oracle Open Data](https://opendata.oraclecloud.com/ords/r/opendata/opendata/home).\n",
    "\n",
    "<a id=\"open_data\"></a>\n",
    "## What is Oracle Open Data?\n",
    "\n",
    "Oracle Open Data is a free service that curates information - spatial images, protein sequences, and annotated text files from the world's leading scientific databases. The repository connects researchers, developers, students, and educators with petabytes of open data from trusted resources. Use Oracle Open Data to view important metadata and sample code for each data set, which simplifies technical complexities and makes it easy for researchers to use.\n",
    "\n",
    "<a id='df'></a>\n",
    "# Building Dataframe for Visual Genome Metadata\n",
    "\n",
    "<a id='build'></a>\n",
    "## Downloading Metadata and Building a Dataframe\n",
    "\n",
    "OCI Object Storage enables customers to securely store any type of data in its native format. With built-in redundancy, OCI Object Storage is ideal for building modern applications that require scale and flexibility, as it can be used to consolidate multiple data sources for analytics, backup, or archive purposes.\n",
    "\n",
    "The Visual Genome metadata for images and objects are stored in OCI Object Storage in the JSON format. Pandas can read JSON and convert it to a tabular format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img_url = \"https://objectstorage.us-ashburn-1.oraclecloud.com/n/idcxvbiyd8fn/b/visual_genome/o/image_data_v1.json.zip\"\n",
    "print(\"Downloading image metadata...\")\n",
    "image_df = pd.read_json(img_url, compression=\"infer\")\n",
    "print(\"Download complete\")\n",
    "\n",
    "object_url = \"https://objectstorage.us-ashburn-1.oraclecloud.com/n/idcxvbiyd8fn/b/visual_genome/o/objects_v1_2.json.zip\"\n",
    "print(\"Downloading object metadata...\")\n",
    "object_df = pd.read_json(object_url, compression=\"infer\")\n",
    "print(\"Download complete\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For ease of use, you can merge these two data frames into one and visualize it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "full_df = image_df.merge(object_df, left_on=\"id\", right_on=\"image_id\")\n",
    "full_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='visualize'></a>\n",
    "# Visualizing Images and Regions\n",
    "\n",
    "<a id='image'></a>\n",
    "## Getting Image Data\n",
    "\n",
    "The following cell filters the `full_df` dataframe for `image_id = 107995`. It extracts the URL of the image and information about objects in it. This object information includes the size of the bounding box, its location, the label for the object and in ID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_id = 107995\n",
    "image_url, objects = full_df[full_df[\"id\"] == image_id][[\"oci_url\", \"objects\"]].values[\n",
    "    0\n",
    "]\n",
    "print(\"The url of the image is {}\".format(image_url))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can display the image using the `PIL_Image` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if file is on disk, use an HTTP handler and BytesIO to read file\n",
    "if image_url.startswith(\"http://\") or image_url.startswith(\"https://\"):\n",
    "    response = requests.get(image_url)\n",
    "    image = PIL_Image.open(BytesIO(response.content))\n",
    "else:\n",
    "    image = PIL_Image.open(image_url)\n",
    "image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='region'></a>\n",
    "## Region Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The object metadata contains the names of the objects, an object ID and information about the bounding box. In particular, `x` and `y` parameters give the pixel location of the **top left** corner of the region. The size of the region is determined by the width, `w`, and height, `h`, parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "objects[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell defines a `Region` class. It is used to manage information about an region."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Region:\n",
    "    def __init__(self, obj):\n",
    "        self.x = obj[\"x\"]\n",
    "        self.y = obj[\"y\"]\n",
    "        self.width = obj[\"w\"]\n",
    "        self.height = obj[\"h\"]\n",
    "        self.names = obj[\"names\"]\n",
    "        self.object_id = obj[\"object_id\"]\n",
    "        self.synsets = obj[\"synsets\"]\n",
    "        self.phrase = \"N/A\"\n",
    "        if len(obj[\"names\"]) > 0:\n",
    "            self.phrase = obj[\"names\"][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='vze'></a>\n",
    "## Visualize Regions\n",
    "\n",
    "You can visualize regions by overlaying them over on the image. The next cell defines a UDF `visualize_regions` function. This function accepts two parameters, `img` and `objects`. The `img` parameter is a `PIL.JpegImagePlugin.JpegImageFile` object of the background image. The `objects` parameter is a Python list that contains information the metadata of the regions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def visualize_regions(img, objects):\n",
    "    plt.imshow(img)\n",
    "    ax = plt.gca()  # get current axes\n",
    "    for obj in objects:\n",
    "        region = Region(obj)  # construct each region object\n",
    "        ax.add_patch(\n",
    "            Rectangle(\n",
    "                (region.x, region.y),  #\n",
    "                region.width,\n",
    "                region.height,\n",
    "                fill=False,\n",
    "                edgecolor=\"red\",\n",
    "                linewidth=3,\n",
    "            )\n",
    "        )\n",
    "        ax.text(\n",
    "            region.x,\n",
    "            region.y,\n",
    "            region.phrase,\n",
    "            style=\"italic\",\n",
    "            bbox={\"facecolor\": \"white\", \"alpha\": 0.7, \"pad\": 10},\n",
    "        )\n",
    "    fig = plt.gcf()  # get current figure\n",
    "    fig.set_size_inches(18, 10)  # resize the image with width 18 and height 10\n",
    "    plt.tick_params(labelbottom=\"off\", labelleft=\"off\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since there are many regions, only first 8 are displayed here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visualize_regions(image, objects[:8])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='ref'></a>\n",
    "# References\n",
    "\n",
    "- [ADS Library Documentation](https://accelerated-data-science.readthedocs.io/en/latest/index.html)\n",
    "- [Data Science YouTube Videos](https://www.youtube.com/playlist?list=PLKCk3OyNwIzv6CWMhvqSB_8MLJIZdO80L)\n",
    "- [OCI Data Science Documentation](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/data-science.htm)\n",
    "- [Oracle Data & AI Blog](https://blogs.oracle.com/datascience/)\n",
    "- [Understanding Conda Environments](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/use-notebook-sessions.htm#conda_understand_environments)\n",
    "- [Use Resource Manager to Configure Your Tenancy for Data Science](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/orm-configure-tenancy.htm)\n",
    "- [Visual Genome at Oralce Open Data](https://opendata.oraclecloud.com/ords/r/opendata/opendata/details?data_set_id=1&clear=RP,13&session=516912761537082)\n",
    "- [Visual Genome Tutorial](http://visualgenome.org/api/v0/api_beginners_tutorial.html)\n",
    "- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](http://visualgenome.org/static/paper/Visual_Genome.pdf)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.15"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}