data_preprocess.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preprocess \n",
    "\n",
    "## 1. Data of Alignment Stage\n",
    "\n",
    "Download the 558K subset of the [LAION-CC-SBU dataset with BLIP captions](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain) through below commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! mkdir -p data/LLaVA-Pretrain\n",
    "! wget -P data/LLaVA-Pretrain https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain/resolve/main/blip_laion_cc_sbu_558k.json\n",
    "! wget -P data/LLaVA-Pretrain https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain/resolve/main/images.zip\n",
    "! unzip data/LLaVA-Pretrain/images.zip -d data/LLaVA-Pretrain\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 2. Data of Extension and Fine-tuning Stage\n",
    "\n",
    "Please download the annotation of the final mixture [LLaVA](https://github.com/haotian-liu/LLaVA) instruction tuning data [llava_v1_5_mix665k.json](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/blob/main/llava_v1_5_mix665k.json)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! mkdir -p data/LLaVA-Instruct-150K\n",
    "! wget -P data/LLaVA-Instruct-150K https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/resolve/main/llava_v1_5_mix665k.json\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Download the images from constituting datasets:\n",
    "\n",
    "- COCO: [train2017](http://images.cocodataset.org/zips/train2017.zip)\n",
    "- GQA: [images](https://downloads.cs.stanford.edu/nlp/data/gqa/images.zip)\n",
    "- OCR-VQA: [download script](https://drive.google.com/drive/folders/1_GYPY5UkUy7HIcR0zq3ZCFgeZN7BAfm_?usp=sharing), **we save all files as `.jpg`**\n",
    "- TextVQA: [train_val_images](https://dl.fbaipublicfiles.com/textvqa/images/train_val_images.zip)\n",
    "- VisualGenome: [part1](https://cs.stanford.edu/people/rak248/VG_100K_2/images.zip), [part2](https://cs.stanford.edu/people/rak248/VG_100K_2/images2.zip)\n",
    "\n",
    "After downloading all of them, organize the data as follows in ./data/LLaVA-Instruct-150K,\n",
    "\n",
    "```\n",
    "├── coco\n",
    "│   └── train2017\n",
    "├── gqa\n",
    "│   └── images\n",
    "├── ocr_vqa\n",
    "│   └── images\n",
    "├── textvqa\n",
    "│   └── train_images\n",
    "└── vg\n",
    "    ├── VG_100K\n",
    "    └── VG_100K_2\n",
    "```\n",
    "\n",
    "## 3. Format Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import random\n",
    "import os \n",
    "from tqdm import tqdm\n",
    "\n",
    "mpath = \"./data/LLaVA-Pretrain\"\n",
    "fname = \"blip_laion_cc_sbu_558k.json\"\n",
    "\n",
    "with open(os.path.join(mpath, fname), \"r\") as f:\n",
    "    data = json.loads(f.read())\n",
    "\n",
    "print(len(data))\n",
    "print(data[-10000])\n",
    "\n",
    "new_data = []\n",
    "for i in tqdm(data):\n",
    "    image = f\"<image>{mpath}/{i['image']}</image>\"\n",
    "    user = i['conversations'][0]['value'].replace('<image>', image)\n",
    "    assistant = i['conversations'][1]['value']\n",
    "    info = {\n",
    "        \"conversations\": [\n",
    "            {\n",
    "                \"from\": \"user\",\n",
    "                \"value\": user\n",
    "            },\n",
    "            {\n",
    "                \"from\": \"assistant\",\n",
    "                \"value\": assistant\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "    new_data.append(info)\n",
    "\n",
    "length = len(new_data)\n",
    "print(length)\n",
    "\n",
    "with open(\"data/558k_pretrain.json\", 'w') as f:\n",
    "    f.write(json.dumps(new_data, ensure_ascii=False))\n",
    "\n",
    "print(new_data[random.randint(0, length)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import random\n",
    "import os\n",
    "from tqdm import tqdm\n",
    "\n",
    "mpath = \"./data/LLaVA-Instruct-150K\"\n",
    "fname = \"llava_v1_5_mix665k.json\"\n",
    "with open(os.path.join(mpath, fname), \"r\") as f:\n",
    "    data = json.loads(f.read())\n",
    "\n",
    "print(len(data))\n",
    "print(data[-10000])\n",
    "\n",
    "new_data = []\n",
    "for i in tqdm(data):\n",
    "    if i.get(\"image\"):\n",
    "        image = f\"<image>{mpath}/{i.get('image')}</image>\"\n",
    "        user = i['conversations'][0]['value'].replace('<image>', image)\n",
    "    else:\n",
    "        user = i['conversations'][0]['value']\n",
    "    assistant = i['conversations'][1]['value']\n",
    "    info = {\n",
    "        \"conversations\": [\n",
    "            {\n",
    "                \"from\": \"user\",\n",
    "                \"value\": user\n",
    "            },\n",
    "            {\n",
    "                \"from\": \"assistant\",\n",
    "                \"value\": assistant\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "    new_data.append(info)\n",
    "\n",
    "length = len(new_data)\n",
    "print(length)\n",
    "\n",
    "with open(\"data/665k_finetune.json\", 'w') as f:\n",
    "    f.write(json.dumps(new_data, ensure_ascii=False))\n",
    "\n",
    "print(new_data[random.randint(0, length)])\n",
    "\n",
    "new_data = random.sample(new_data, 10000)\n",
    "with open(\"data/10k_finetune.json\", 'w') as f:\n",
    "    f.write(json.dumps(new_data, ensure_ascii=False))\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}