HW2-PrepareData.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Venice Boat Classification - Prepare Dataset\n",
    "\n",
    "The data is split into two folders, named \"sc5\" for training, and \"sc5-2013-Mar-Apr-Test-20130412\" for testing. However, they do not have the same structure. Both folders are contained in the 'Dataset' folder\n",
    "* The training set handles separation of categories by folders, each one containing the corresponding images. \n",
    "* The testing set has a single folder with all the files merged, and a ground truth text file that relates the name of the file with its corresponding category."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prepare testing set\n",
    "In order to work with the same structure, the testing set will be organized in a new folder called 'sc5-test' in the same way as the training set, separated by categories.\n",
    "Nevertheless, some of the name of the categories between both sets are not the same. Namely, \n",
    "\n",
    "<table style=\"text-align:center\">\n",
    "    <thead>\n",
    "        <tr><th colspan=\"2\" style=\"text-align:center\">Category names</th></tr>\n",
    "        <tr><th style=\"text-align:center\">Train</th><th style=\"text-align:center\">Test</th></tr>\n",
    "    </thead>\n",
    "    <tbody>\n",
    "        <tr><td>Water</td><td>Snapshot Acqua</td></tr>\n",
    "        <tr><td>lanciafino10mbianca</td><td>Lancia: fino 10 m Bianca</td></tr>\n",
    "        <tr><td>Vaporetto ACTV</td><td>VaporettoACTV</td></tr>\n",
    "    </tbody>\n",
    "</table>\n",
    "\n",
    "Accordingly, the following code also standardizes the category folder names. The ground truth file, named 'sc5-2013-GroundTruth.txt', is taken out from the original folder and put into the 'Dataset' folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os \n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import shutil\n",
    "\n",
    "#Define paths\n",
    "base_path = 'Dataset'\n",
    "test_path = base_path + '/' + 'sc5-2013-Mar-Apr-Test-20130412'\n",
    "train_path = base_path + '/' + 'sc5'\n",
    "test_path_ordered = base_path + '/' + 'sc5-test'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total labels: 24\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['alilaguna',\n",
       " 'ambulanza',\n",
       " 'barchino',\n",
       " 'cacciapesca',\n",
       " 'caorlina',\n",
       " 'gondola',\n",
       " 'lanciafino10m',\n",
       " 'lanciafino10mbianca',\n",
       " 'lanciafino10mmarrone',\n",
       " 'lanciamaggioredi10mbianca',\n",
       " 'lanciamaggioredi10mmarrone',\n",
       " 'motobarca',\n",
       " 'motopontonerettangolare',\n",
       " 'motoscafoactv',\n",
       " 'mototopo',\n",
       " 'patanella',\n",
       " 'polizia',\n",
       " 'raccoltarifiuti',\n",
       " 'sandoloaremi',\n",
       " 'sanpierota',\n",
       " 'topa',\n",
       " 'vaporettoactv',\n",
       " 'vigilidelfuoco',\n",
       " 'water']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Create new folder for testing set\n",
    "if not os.path.exists(test_path_ordered):\n",
    "    os.mkdir(test_path_ordered)\n",
    "\n",
    "#Get list of labels and put them in lowercase\n",
    "labels_train = os.listdir(train_path)\n",
    "labels_train = list(map(lambda x: x.lower(), labels_train))\n",
    "print(\"Total labels: \" + str(len(labels_train)))\n",
    "labels_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['20130412_043104_54559.jpg;Snapshot Acqua',\n",
       " '20130412_043117_54573.jpg;Motobarca',\n",
       " '20130412_043148_54819.jpg;Vaporetto ACTV',\n",
       " '20130412_043218_54895.jpg;Mototopo',\n",
       " '20130412_043335_55056.jpg;Mototopo']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read the ground truth file\n",
    "f = open(os.path.join(base_path, 'sc5-2013-GroundTruth.txt'),'r')\n",
    "rows = f.read().split('\\n')\n",
    "rows[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Preprocessing begins...\n",
      "[INFO] 100 of 1970 elements processed.\n",
      "[INFO] 200 of 1970 elements processed.\n",
      "[INFO] 300 of 1970 elements processed.\n",
      "[INFO] 400 of 1970 elements processed.\n",
      "[INFO] 500 of 1970 elements processed.\n",
      "[INFO] 600 of 1970 elements processed.\n",
      "[INFO] 700 of 1970 elements processed.\n",
      "[INFO] 800 of 1970 elements processed.\n",
      "[INFO] 900 of 1970 elements processed.\n",
      "[INFO] 1000 of 1970 elements processed.\n",
      "[INFO] 1100 of 1970 elements processed.\n",
      "[INFO] 1200 of 1970 elements processed.\n",
      "[INFO] 1300 of 1970 elements processed.\n",
      "[INFO] 1400 of 1970 elements processed.\n",
      "[INFO] 1500 of 1970 elements processed.\n",
      "[INFO] 1600 of 1970 elements processed.\n",
      "[INFO] 1700 of 1970 elements processed.\n",
      "[INFO] 1800 of 1970 elements processed.\n",
      "[INFO] 1900 of 1970 elements processed.\n",
      "[INFO] Test folder preprocessing finished.\n"
     ]
    }
   ],
   "source": [
    "#Loop over images in the original folder, rename them and save them in new folder, in its corresponding category.\n",
    "i = 0\n",
    "num_rows = len(rows)\n",
    "print(\"[INFO] Preprocessing begins...\" )\n",
    "for row in rows:\n",
    "    if row == \"\": continue\n",
    "    [file, label] = row.split(';')\n",
    "    \n",
    "    #Removes spaces, colons and word 'snapshot'. Replaces word 'acqua' by 'water'\n",
    "    label = label.lower().replace(' ', '').replace(':','').replace('snapshot','') \n",
    "    if label == 'acqua':\n",
    "        label = 'water'\n",
    "    folder = os.path.join(test_path_ordered,label)\n",
    "    if not os.path.exists(folder):\n",
    "        os.mkdir(folder)\n",
    "    shutil.copyfile(os.path.join(test_path, file), os.path.join(folder,file))    \n",
    "    i += 1\n",
    "    if i%100 == 0:\n",
    "        print(\"[INFO] \" + str(i) + \" of \" + str(num_rows) + \" elements processed.\" )\n",
    "print(\"[INFO] Test folder preprocessing finished.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The folder names in the training folder are converted to lowercase too, to match the names in the testing folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "for folder in os.listdir(train_path):\n",
    "    os.rename(train_path +'/'+ folder,train_path  +'/'+ folder.lower())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optionally, the original test folder can be erased since no longer needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "shutil.rmtree(test_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Folder PATH listing for volume Datos\n",
      "Volume serial number is 4A52-21EF\n",
      "D:.\n",
      "+---.ipynb_checkpoints\n",
      "+---.vs\n",
      "|   \\---HW2\n",
      "|       \\---v15\n",
      "+---Dataset\n",
      "|   +---.ipynb_checkpoints\n",
      "|   +---dstest\n",
      "|   |   +---alilaguna\n",
      "|   |   +---ambulanza\n",
      "|   |   +---barchino\n",
      "|   |   +---gondola\n",
      "|   |   +---lanciafino10m\n",
      "|   |   +---lanciafino10mbianca\n",
      "|   |   +---lanciafino10mmarrone\n",
      "|   |   +---lanciamaggioredi10mbianca\n",
      "|   |   +---motobarca\n",
      "|   |   +---motopontonerettangolare\n",
      "|   |   +---motoscafoactv\n",
      "|   |   +---mototopo\n",
      "|   |   +---patanella\n",
      "|   |   +---polizia\n",
      "|   |   +---raccoltarifiuti\n",
      "|   |   +---sandoloaremi\n",
      "|   |   +---topa\n",
      "|   |   \\---vaporettoactv\n",
      "|   +---dstrain\n",
      "|   |   +---alilaguna\n",
      "|   |   +---ambulanza\n",
      "|   |   +---barchino\n",
      "|   |   +---gondola\n",
      "|   |   +---lanciafino10m\n",
      "|   |   +---lanciafino10mbianca\n",
      "|   |   +---lanciafino10mmarrone\n",
      "|   |   +---lanciamaggioredi10mbianca\n",
      "|   |   +---motobarca\n",
      "|   |   +---motopontonerettangolare\n",
      "|   |   +---motoscafoactv\n",
      "|   |   +---mototopo\n",
      "|   |   +---patanella\n",
      "|   |   +---polizia\n",
      "|   |   +---raccoltarifiuti\n",
      "|   |   +---sandoloaremi\n",
      "|   |   +---topa\n",
      "|   |   \\---vaporettoactv\n",
      "|   +---sc5\n",
      "|   |   +---alilaguna\n",
      "|   |   +---ambulanza\n",
      "|   |   +---barchino\n",
      "|   |   +---cacciapesca\n",
      "|   |   +---caorlina\n",
      "|   |   +---gondola\n",
      "|   |   +---lanciafino10m\n",
      "|   |   +---lanciafino10mbianca\n",
      "|   |   +---lanciafino10mmarrone\n",
      "|   |   +---lanciamaggioredi10mbianca\n",
      "|   |   +---lanciamaggioredi10mmarrone\n",
      "|   |   +---motobarca\n",
      "|   |   +---motopontonerettangolare\n",
      "|   |   +---motoscafoactv\n",
      "|   |   +---mototopo\n",
      "|   |   +---patanella\n",
      "|   |   +---polizia\n",
      "|   |   +---raccoltarifiuti\n",
      "|   |   +---sandoloaremi\n",
      "|   |   +---sanpierota\n",
      "|   |   +---topa\n",
      "|   |   +---vaporettoactv\n",
      "|   |   +---vigilidelfuoco\n",
      "|   |   \\---water\n",
      "|   \\---sc5-test\n",
      "|       +---alilaguna\n",
      "|       +---ambulanza\n",
      "|       +---barcamultipla\n",
      "|       +---barcaparziale\n",
      "|       +---barchino\n",
      "|       +---gondola\n",
      "|       +---lanciafino10m\n",
      "|       +---lanciafino10mbianca\n",
      "|       +---lanciafino10mmarrone\n",
      "|       +---lanciamaggioredi10mbianca\n",
      "|       +---motobarca\n",
      "|       +---motopontonerettangolare\n",
      "|       +---motoscafoactv\n",
      "|       +---mototopo\n",
      "|       +---mototopocorto\n",
      "|       +---patanella\n",
      "|       +---polizia\n",
      "|       +---raccoltarifiuti\n",
      "|       +---sandoloaremi\n",
      "|       +---topa\n",
      "|       +---vaporettoactv\n",
      "|       \\---water\n",
      "+---Inform\n",
      "+---Models\n",
      "+---Resources\n",
      "\\---__pycache__\n"
     ]
    }
   ],
   "source": [
    "#Final folder structure\n",
    "!tree /a"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}