finetune_clip_different_approach.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "k9CPHi0j-jZN"
   },
   "source": [
    "## Finetune CLIP on archaeological objects\n",
    "\n",
    "October 8. Shawn Graham\n",
    "\n",
    "This notebook downloads images from Open Context, reshapes the metadata into captions, and then uses Damian Stewart's hugginface_finetune_clip.py to retrain the `openai/clip-vit-base-patch32` model (see [this](https://github.com/damian0815/finetune-clip-huggingface/blob/main/huggingface_finetune_clip_runner.ipynb)). Other CLIP versions can be used, but so far the other ones I've tried take too much memory to be used in the free colab tier or on my m1 mac miniw 16 gb ram.\n",
    "\n",
    "The first code block under creates captions from separate metadata fields and downloads, reshapes the results. Users should use the subsequent block ('better captions') instead for the open context materials. A subsequent section grabs materials from the MET and appends it to the training data for the fine tuning. This section can be ignored if desired."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "7A_txB8l-md7",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pandas\n",
      "  Obtaining dependency information for pandas from https://files.pythonhosted.org/packages/ff/5a/c7359edec58500b35da8dc40a69ea7b0a3be48a479e1c91e8e8d0a2d9aa7/pandas-2.1.1-cp310-cp310-macosx_11_0_arm64.whl.metadata\n",
      "  Downloading pandas-2.1.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (18 kB)\n",
      "Requirement already satisfied: requests in /Users/shawngraham/mambaforge/envs/clip/lib/python3.10/site-packages (2.31.0)\n",
      "Collecting numpy>=1.22.4 (from pandas)\n",
      "  Obtaining dependency information for numpy>=1.22.4 from https://files.pythonhosted.org/packages/5c/ff/0e1f31c70495df6a1afbe98fa237f36e6fb7c5443fcb9a53f43170e5814c/numpy-1.26.0-cp310-cp310-macosx_11_0_arm64.whl.metadata\n",
      "  Downloading numpy-1.26.0-cp310-cp310-macosx_11_0_arm64.whl.metadata (53 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.5/53.5 kB\u001b[0m \u001b[31m347.8 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: python-dateutil>=2.8.2 in /Users/shawngraham/mambaforge/envs/clip/lib/python3.10/site-packages (from pandas) (2.8.2)\n",
      "Collecting pytz>=2020.1 (from pandas)\n",
      "  Obtaining dependency information for pytz>=2020.1 from https://files.pythonhosted.org/packages/32/4d/aaf7eff5deb402fd9a24a1449a8119f00d74ae9c2efa79f8ef9994261fc2/pytz-2023.3.post1-py2.py3-none-any.whl.metadata\n",
      "  Using cached pytz-2023.3.post1-py2.py3-none-any.whl.metadata (22 kB)\n",
      "Collecting tzdata>=2022.1 (from pandas)\n",
      "  Using cached tzdata-2023.3-py2.py3-none-any.whl (341 kB)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/shawngraham/mambaforge/envs/clip/lib/python3.10/site-packages (from requests) (3.3.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Users/shawngraham/mambaforge/envs/clip/lib/python3.10/site-packages (from requests) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/shawngraham/mambaforge/envs/clip/lib/python3.10/site-packages (from requests) (2.0.6)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/shawngraham/mambaforge/envs/clip/lib/python3.10/site-packages (from requests) (2023.7.22)\n",
      "Requirement already satisfied: six>=1.5 in /Users/shawngraham/mambaforge/envs/clip/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)\n",
      "Downloading pandas-2.1.1-cp310-cp310-macosx_11_0_arm64.whl (10.9 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.9/10.9 MB\u001b[0m \u001b[31m1.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0mm\n",
      "\u001b[?25hDownloading numpy-1.26.0-cp310-cp310-macosx_11_0_arm64.whl (14.0 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.0/14.0 MB\u001b[0m \u001b[31m3.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hUsing cached pytz-2023.3.post1-py2.py3-none-any.whl (502 kB)\n",
      "Installing collected packages: pytz, tzdata, numpy, pandas\n",
      "Successfully installed numpy-1.26.0 pandas-2.1.1 pytz-2023.3.post1 tzdata-2023.3\n"
     ]
    }
   ],
   "source": [
    "!pip install pandas requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## full json export from open context\n",
    "\n",
    "If the data is made available as json with just the data in the various fields, there are some shennanigans necessary to get the data into shape with uuid as image name and captions derived from those fields.\n",
    "\n",
    "If on the other hand the data is made available with the image file uri, the media uuid, and captions where some of the cidoc crm ontology fields have been used to craft more literate captions, then skip to the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "aoaWNQF_BYYp"
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import pandas as pd\n",
    "\n",
    "url = 'https://raw.githubusercontent.com/opencontext/archaeology-images-ai/main/json_data/artifact_images_w_descriptions.json'\n",
    "data = requests.get(url).json()\n",
    "df = pd.json_normalize(data)  # convert json to pandas DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 825
    },
    "id": "UksBCetGBuv6",
    "outputId": "863cf58d-4bc4-4faa-e3eb-b039b4d570e2"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>image</th>\n",
       "      <th>media__uri</th>\n",
       "      <th>image_genre</th>\n",
       "      <th>image_type</th>\n",
       "      <th>subject__item_class__label</th>\n",
       "      <th>context___1</th>\n",
       "      <th>context___2</th>\n",
       "      <th>context___3</th>\n",
       "      <th>time_range</th>\n",
       "      <th>has_type</th>\n",
       "      <th>consists_of</th>\n",
       "      <th>origin_place</th>\n",
       "      <th>has_taxonomic_identifier</th>\n",
       "      <th>has_anatomical_identification</th>\n",
       "      <th>temporal_coverage</th>\n",
       "      <th>project_specific_descriptions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https://iiif.archivelab.org/iiif/opencontext-1...</td>\n",
       "      <td>https://opencontext.org/media/a9cedbad-e25b-4f...</td>\n",
       "      <td>archaeology</td>\n",
       "      <td>artifact</td>\n",
       "      <td>Object</td>\n",
       "      <td>Asia</td>\n",
       "      <td>Turkey</td>\n",
       "      <td>Domuztepe</td>\n",
       "      <td>6500 BCE to 5500 BCE</td>\n",
       "      <td>seals (artifacts)</td>\n",
       "      <td>rock (inorganic material)</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Artifact Name: Stamp Seal \\n Material: Stone, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://iiif.archivelab.org/iiif/opencontext-1...</td>\n",
       "      <td>https://opencontext.org/media/1bbbca07-82f3-46...</td>\n",
       "      <td>archaeology</td>\n",
       "      <td>artifact</td>\n",
       "      <td>Object</td>\n",
       "      <td>Asia</td>\n",
       "      <td>Turkey</td>\n",
       "      <td>Domuztepe</td>\n",
       "      <td>6500 BCE to 5500 BCE</td>\n",
       "      <td>seals (artifacts)</td>\n",
       "      <td>soapstone</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Artifact Name: Stamp Seal \\n Material: Steatit...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://iiif.archivelab.org/iiif/opencontext-1...</td>\n",
       "      <td>https://opencontext.org/media/2062e3fa-41e2-d7...</td>\n",
       "      <td>archaeology</td>\n",
       "      <td>artifact</td>\n",
       "      <td>Object</td>\n",
       "      <td>Asia</td>\n",
       "      <td>Turkey</td>\n",
       "      <td>Domuztepe</td>\n",
       "      <td>6500 BCE to 5500 BCE</td>\n",
       "      <td>seals (artifacts)</td>\n",
       "      <td>rock (inorganic material)</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Artifact Name: Stamp Seal \\n Material: Stone, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://iiif.archivelab.org/iiif/opencontext-1...</td>\n",
       "      <td>https://opencontext.org/media/2dc18114-4ddf-7c...</td>\n",
       "      <td>archaeology</td>\n",
       "      <td>artifact</td>\n",
       "      <td>Object</td>\n",
       "      <td>Asia</td>\n",
       "      <td>Turkey</td>\n",
       "      <td>Domuztepe</td>\n",
       "      <td>6500 BCE to 5500 BCE</td>\n",
       "      <td>pendants (jewelry)</td>\n",
       "      <td>chert</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Artifact Name: Pendant \\n Material: Chert/Flint</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://iiif.archivelab.org/iiif/opencontext-1...</td>\n",
       "      <td>https://opencontext.org/media/d7e8b4e5-be3b-44...</td>\n",
       "      <td>archaeology</td>\n",
       "      <td>artifact</td>\n",
       "      <td>Object</td>\n",
       "      <td>Asia</td>\n",
       "      <td>Turkey</td>\n",
       "      <td>Domuztepe</td>\n",
       "      <td>6500 BCE to 5500 BCE</td>\n",
       "      <td>nails (fasteners)</td>\n",
       "      <td>iron (metal)</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Artifact Name: Nail \\n Material: Iron \\n Dispo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72357</th>\n",
       "      <td>https://artiraq.org/static/opencontext/pettegr...</td>\n",
       "      <td>https://opencontext.org/media/d3620d27-cb41-44...</td>\n",
       "      <td>archaeology</td>\n",
       "      <td>artifact</td>\n",
       "      <td>Object</td>\n",
       "      <td>Europe</td>\n",
       "      <td>Greece</td>\n",
       "      <td>Corinthia</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Chronotype: Fineware, Late Helladic I-IIA \\n Z...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72358</th>\n",
       "      <td>https://artiraq.org/static/opencontext/pettegr...</td>\n",
       "      <td>https://opencontext.org/media/41f03708-baa5-4d...</td>\n",
       "      <td>archaeology</td>\n",
       "      <td>artifact</td>\n",
       "      <td>Object</td>\n",
       "      <td>Europe</td>\n",
       "      <td>Greece</td>\n",
       "      <td>Corinthia</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Chronotype: Fineware, Late Helladic I-IIA \\n Z...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72359</th>\n",
       "      <td>https://artiraq.org/static/opencontext/pettegr...</td>\n",
       "      <td>https://opencontext.org/media/453e04b2-7905-4e...</td>\n",
       "      <td>archaeology</td>\n",
       "      <td>artifact</td>\n",
       "      <td>Object</td>\n",
       "      <td>Europe</td>\n",
       "      <td>Greece</td>\n",
       "      <td>Corinthia</td>\n",
       "      <td></td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Chronotype: Fineware, Late Helladic I-IIA \\n Z...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72360</th>\n",
       "      <td>https://artiraq.org/static/opencontext/interna...</td>\n",
       "      <td>https://opencontext.org/media/1840719d-2934-48...</td>\n",
       "      <td>archaeology</td>\n",
       "      <td>artifact</td>\n",
       "      <td>Object</td>\n",
       "      <td>Off World</td>\n",
       "      <td>International Space Station</td>\n",
       "      <td>Zvezda Service Module</td>\n",
       "      <td></td>\n",
       "      <td>icons (devotional images);  religions and reli...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Location: Top center \\n Item type: Icon \\n Sec...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72361</th>\n",
       "      <td>https://artiraq.org/static/opencontext/interna...</td>\n",
       "      <td>https://opencontext.org/media/5b4ea4a4-a448-4d...</td>\n",
       "      <td>archaeology</td>\n",
       "      <td>artifact</td>\n",
       "      <td>Object</td>\n",
       "      <td>Off World</td>\n",
       "      <td>International Space Station</td>\n",
       "      <td>Zvezda Service Module</td>\n",
       "      <td></td>\n",
       "      <td>photographs</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Location: Niche port side;  Niche starboard si...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>72362 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   image  \\\n",
       "0      https://iiif.archivelab.org/iiif/opencontext-1...   \n",
       "1      https://iiif.archivelab.org/iiif/opencontext-1...   \n",
       "2      https://iiif.archivelab.org/iiif/opencontext-1...   \n",
       "3      https://iiif.archivelab.org/iiif/opencontext-1...   \n",
       "4      https://iiif.archivelab.org/iiif/opencontext-1...   \n",
       "...                                                  ...   \n",
       "72357  https://artiraq.org/static/opencontext/pettegr...   \n",
       "72358  https://artiraq.org/static/opencontext/pettegr...   \n",
       "72359  https://artiraq.org/static/opencontext/pettegr...   \n",
       "72360  https://artiraq.org/static/opencontext/interna...   \n",
       "72361  https://artiraq.org/static/opencontext/interna...   \n",
       "\n",
       "                                              media__uri  image_genre  \\\n",
       "0      https://opencontext.org/media/a9cedbad-e25b-4f...  archaeology   \n",
       "1      https://opencontext.org/media/1bbbca07-82f3-46...  archaeology   \n",
       "2      https://opencontext.org/media/2062e3fa-41e2-d7...  archaeology   \n",
       "3      https://opencontext.org/media/2dc18114-4ddf-7c...  archaeology   \n",
       "4      https://opencontext.org/media/d7e8b4e5-be3b-44...  archaeology   \n",
       "...                                                  ...          ...   \n",
       "72357  https://opencontext.org/media/d3620d27-cb41-44...  archaeology   \n",
       "72358  https://opencontext.org/media/41f03708-baa5-4d...  archaeology   \n",
       "72359  https://opencontext.org/media/453e04b2-7905-4e...  archaeology   \n",
       "72360  https://opencontext.org/media/1840719d-2934-48...  archaeology   \n",
       "72361  https://opencontext.org/media/5b4ea4a4-a448-4d...  archaeology   \n",
       "\n",
       "      image_type subject__item_class__label context___1  \\\n",
       "0       artifact                     Object        Asia   \n",
       "1       artifact                     Object        Asia   \n",
       "2       artifact                     Object        Asia   \n",
       "3       artifact                     Object        Asia   \n",
       "4       artifact                     Object        Asia   \n",
       "...          ...                        ...         ...   \n",
       "72357   artifact                     Object      Europe   \n",
       "72358   artifact                     Object      Europe   \n",
       "72359   artifact                     Object      Europe   \n",
       "72360   artifact                     Object   Off World   \n",
       "72361   artifact                     Object   Off World   \n",
       "\n",
       "                       context___2            context___3  \\\n",
       "0                           Turkey              Domuztepe   \n",
       "1                           Turkey              Domuztepe   \n",
       "2                           Turkey              Domuztepe   \n",
       "3                           Turkey              Domuztepe   \n",
       "4                           Turkey              Domuztepe   \n",
       "...                            ...                    ...   \n",
       "72357                       Greece              Corinthia   \n",
       "72358                       Greece              Corinthia   \n",
       "72359                       Greece              Corinthia   \n",
       "72360  International Space Station  Zvezda Service Module   \n",
       "72361  International Space Station  Zvezda Service Module   \n",
       "\n",
       "                 time_range  \\\n",
       "0      6500 BCE to 5500 BCE   \n",
       "1      6500 BCE to 5500 BCE   \n",
       "2      6500 BCE to 5500 BCE   \n",
       "3      6500 BCE to 5500 BCE   \n",
       "4      6500 BCE to 5500 BCE   \n",
       "...                     ...   \n",
       "72357                         \n",
       "72358                         \n",
       "72359                         \n",
       "72360                         \n",
       "72361                         \n",
       "\n",
       "                                                has_type  \\\n",
       "0                                      seals (artifacts)   \n",
       "1                                      seals (artifacts)   \n",
       "2                                      seals (artifacts)   \n",
       "3                                     pendants (jewelry)   \n",
       "4                                      nails (fasteners)   \n",
       "...                                                  ...   \n",
       "72357                                               None   \n",
       "72358                                               None   \n",
       "72359                                               None   \n",
       "72360  icons (devotional images);  religions and reli...   \n",
       "72361                                        photographs   \n",
       "\n",
       "                     consists_of origin_place has_taxonomic_identifier  \\\n",
       "0      rock (inorganic material)         None                     None   \n",
       "1                      soapstone         None                     None   \n",
       "2      rock (inorganic material)         None                     None   \n",
       "3                          chert         None                     None   \n",
       "4                   iron (metal)         None                     None   \n",
       "...                          ...          ...                      ...   \n",
       "72357                       None         None                     None   \n",
       "72358                       None         None                     None   \n",
       "72359                       None         None                     None   \n",
       "72360                       None         None                     None   \n",
       "72361                       None         None                     None   \n",
       "\n",
       "      has_anatomical_identification temporal_coverage  \\\n",
       "0                              None              None   \n",
       "1                              None              None   \n",
       "2                              None              None   \n",
       "3                              None              None   \n",
       "4                              None              None   \n",
       "...                             ...               ...   \n",
       "72357                          None              None   \n",
       "72358                          None              None   \n",
       "72359                          None              None   \n",
       "72360                          None              None   \n",
       "72361                          None              None   \n",
       "\n",
       "                           project_specific_descriptions  \n",
       "0      Artifact Name: Stamp Seal \\n Material: Stone, ...  \n",
       "1      Artifact Name: Stamp Seal \\n Material: Steatit...  \n",
       "2      Artifact Name: Stamp Seal \\n Material: Stone, ...  \n",
       "3        Artifact Name: Pendant \\n Material: Chert/Flint  \n",
       "4      Artifact Name: Nail \\n Material: Iron \\n Dispo...  \n",
       "...                                                  ...  \n",
       "72357  Chronotype: Fineware, Late Helladic I-IIA \\n Z...  \n",
       "72358  Chronotype: Fineware, Late Helladic I-IIA \\n Z...  \n",
       "72359  Chronotype: Fineware, Late Helladic I-IIA \\n Z...  \n",
       "72360  Location: Top center \\n Item type: Icon \\n Sec...  \n",
       "72361  Location: Niche port side;  Niche starboard si...  \n",
       "\n",
       "[72362 rows x 16 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.rename(columns={'image_file__uri': 'image'}, inplace=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aCpRPIpFCWhY",
    "outputId": "f381388d-f7e4-4553-fcc7-bc08ad4890d6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "An error occurred while fetching: https://iiif.archivelab.org/iiif/opencontext-22-d-9-11-4-4-p-1jpg/full/675,/0/default.jpg\n",
      "An error occurred while fetching: https://iiif.archivelab.org/iiif/opencontext-24-19710499bwjpg/full/675,/0/default.jpg\n",
      "An error occurred while fetching: https://iiif.archivelab.org/iiif/opencontext-22-c-1-1093-25-4-p-2jpg/full/675,/0/default.jpg\n",
      "An error occurred while fetching: https://iiif.archivelab.org/iiif/opencontext-24-19720119chgjpg/full/675,/0/default.jpg\n",
      "An error occurred while fetching: https://iiif.archivelab.org/iiif/opencontext-24-20000037sideajpg/full/675,/0/default.jpg\n",
      "An error occurred while fetching: https://iiif.archivelab.org/iiif/opencontext-22-f-1-1021-1118-1-p-2jpg/full/675,/0/default.jpg\n",
      "An error occurred while fetching: https://iiif.archivelab.org/iiif/opencontext-22-d-6-102-2-19-p-6jpg/full/675,/0/default.jpg\n",
      "An error occurred while fetching: https://iiif.archivelab.org/iiif/opencontext-24-88-125bottomjpg/full/675,/0/default.jpg\n",
      "An error occurred while fetching: https://iiif.archivelab.org/iiif/opencontext-1-photo2-dt-419/full/650,/0/default.jpg\n",
      "An error occurred while fetching: https://iiif.archivelab.org/iiif/opencontext-14-b200918-5jpg/full/675,/0/default.jpg\n",
      "An error occurred while fetching: https://iiif.archivelab.org/iiif/opencontext-24-19710749bwbjpg/full/675,/0/default.jpg\n",
      "An error occurred while fetching: https://iiif.archivelab.org/iiif/opencontext-24-19820026bhgjpg/full/675,/0/default.jpg\n",
      "An error occurred while fetching: https://iiif.archivelab.org/iiif/opencontext-1-dscn2252jpg/full/675,/0/default.jpg\n",
      "An error occurred while fetching: https://artiraq.org/static/opencontext/poggio-civitate/preview/photos/20020015PROFILE.jpg\n",
      "An error occurred while fetching: https://iiif.archivelab.org/iiif/opencontext-22-a-9-30-6-1-p-1jpg/full/675,/0/default.jpg\n",
      "An error occurred while fetching: https://artiraq.org/static/opencontext/poggio-civitate/preview/photos/20070344BACK.jpg\n",
      "Downloaded images in directory 'images'. Number of url errors: 16\n",
      "Downloaded images in directory 'testing'. Number of url errors: 0\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import random\n",
    "import requests\n",
    "\n",
    "# Shuffle your DataFrame\n",
    "df = df.sample(frac=1, random_state=76) #67 for the first 500; will a different seed give me different pics & I can collate results?\n",
    "\n",
    "# Split the data into training and testing datasets\n",
    "train_images = df.iloc[:1000]\n",
    "test_images = df.iloc[1000:1100]\n",
    "\n",
    "def download_images(df, dir_name):\n",
    "    os.makedirs(dir_name, exist_ok=True)\n",
    "    url_errors=[]\n",
    "    for _, row in df.iterrows():\n",
    "        url = row['image']\n",
    "        # Get the extension for the image file from the URL\n",
    "        extension = url.split('.')[-1]\n",
    "        # Get the UUID for the media file\n",
    "        media_uuid = row['media__uri'].split('/')[-1]\n",
    "        file_name = f'{media_uuid}.{extension}'\n",
    "        file_path = os.path.join(dir_name, file_name)\n",
    "\n",
    "        try:\n",
    "            response = requests.get(url, timeout=5)\n",
    "            response.raise_for_status()\n",
    "        except (requests.exceptions.RequestException, requests.exceptions.Timeout):\n",
    "            print(f'An error occurred while fetching: {url}')\n",
    "            url_errors.append(url)\n",
    "            continue\n",
    "\n",
    "        with open(file_path, 'wb') as img_file:\n",
    "            img_file.write(response.content)\n",
    "    \n",
    "    print(f\"Downloaded images in directory '{dir_name}'. Number of url errors: {len(url_errors)}\")\n",
    "\n",
    "download_images(train_images, 'images') # downloads training images to 'images' folder\n",
    "download_images(test_images, 'testing') # downloads testing images to 'testing' folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2ncX_hG2BfT3",
    "outputId": "dc7c9827-9976-4ef5-a93e-8339942b0da3"
   },
   "outputs": [],
   "source": [
    "import re\n",
    "import string\n",
    "def create_metadata(df, dir_name):\n",
    "    # Make a copy of the dataframe to avoid modifying the original one\n",
    "    df = df.copy()\n",
    "\n",
    "    # fill NaN values with empty string\n",
    "    df.fillna('', inplace=True)\n",
    "\n",
    "    # append values of required columns into new 'caption' column\n",
    "    df['caption'] = 'A photograph of ' + df['consists_of'].astype(str) \\\n",
    "                    + ', ' + df['project_specific_descriptions'].astype(str) \\\n",
    "                    + ' dating to ' + df['time_range'].astype(str) \\\n",
    "                    + ' from ' + df['context___1'].astype(str) \\\n",
    "                    + ', ' + df['context___2'].astype(str) \\\n",
    "                    + ', ' + df['context___3'].astype(str)\n",
    "    df['caption'] = df['caption'].replace('\\n', ' ')\n",
    "    df['caption'] = df['caption'].replace('False', ' ')\n",
    "    df['caption'] = df['caption'].replace('True', ' ')\n",
    "    # Remove all other punctuation\n",
    "    df['caption'] = df['caption'].apply(lambda x: re.sub(r'[{}]'.format(string.punctuation), ' ', x)).str.strip()\n",
    "\n",
    "    # Rewrite 'image' column to just contain the filename\n",
    "    df['image'] = df.apply(lambda row: f\"{row['media__uri'].split('/')[-1]}.{row['image'].split('.').pop()}\", axis=1)\n",
    "\n",
    "    # reshaping data to contain only 'image' and 'caption'\n",
    "    df = df[['image', 'caption']]\n",
    "    df.loc[:, 'image'] = dir_name + '/' + df['image'].astype(str)\n",
    "\n",
    "    return df\n",
    "\n",
    "df = create_metadata(train_images, 'images')\n",
    "with open('train.json', 'w') as file:\n",
    "    df.to_json(file, orient='records', lines=True)\n",
    "\n",
    "testdf = create_metadata(test_images, 'testing')\n",
    "with open('test.json', 'w') as file:\n",
    "    testdf.to_json(file, orient='records', lines=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data from Open Context with better captions\n",
    "\n",
    "If the export data looks like this:\n",
    "\n",
    "```json\n",
    "{\n",
    "        \"image_file__uri\":\"https:\\/\\/iiif.archivelab.org\\/iiif\\/opencontext-14-b2009121-5jpg\\/full\\/675,\\/0\\/default.jpg\",\n",
    "        \"media__uuid\":\"6fb98a27-a3a9-465e-6804-84e09f23b704\",\n",
    "        \"media__uri\":\"https:\\/\\/opencontext.org\\/media\\/6fb98a27-a3a9-465e-6804-84e09f23b704\",\n",
    "        \"caption\":\"An image of an archaeological artifact found at Tell en-Nasbeh, a place in Palestinian Authority which is more generally located in Asia. The artifact mainly consists of ceramic (material). Additional attributes that describe the artifact include: Catalog: Object \\n Catalog Type: Archaeology \\n Collection: Tell en-Nasbeh Collection \\n Cultural Period: Iron IIC \\n Conservation Condition: Good \\n Stratum: 3C-3A \\n Primary Location: Holbrook Hall, Pacific School of Religion \\n Object Name: Loom Weight \\n Object Sub-category: Ceramic -- Tool \\n Subjects and Themes: Weaving and textiles \\n Category Type: Ceramic \\n Excav. Ceramic Type: M15 \\n Photo Number: 254 \\n Material: Ceramic \\n Decoration: None \\n Completeness: Whole \\n Manufacture: Handmade\"\n",
    "    }\n",
    "```\n",
    "then use the following code snippet to download images and create the training/testing json datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Download error for URL https://artiraq.org/static/opencontext/poggio-civitate/preview/photos/19820174BACK.jpg\n",
      "HTTP Error 404: Not Found\n",
      "Download error for URL https://artiraq.org/static/opencontext/poggio-civitate/preview/photos/20020015HEAD.jpg\n",
      "HTTP Error 404: Not Found\n",
      "Download error for URL https://iiif.archivelab.org/iiif/opencontext-16-250jpg/full/675,/0/default.jpg\n",
      "HTTP Error 404: NOT FOUND\n",
      "Download error for URL https://artiraq.org/static/opencontext/poggio-civitate/preview/photos/20020004PROFILE.jpg\n",
      "HTTP Error 404: Not Found\n",
      "Download error for URL https://artiraq.org/static/opencontext/poggio-civitate/preview/photos/20030122FRONT.jpg\n",
      "HTTP Error 404: Not Found\n",
      "Download error for URL https://artiraq.org/static/opencontext/poggio-civitate/preview/photos/20020028BOTTOM.jpg\n",
      "HTTP Error 404: Not Found\n",
      "Download error for URL https://iiif.archivelab.org/iiif/opencontext-16-258jpg/full/675,/0/default.jpg\n",
      "HTTP Error 404: NOT FOUND\n",
      "Download error for URL https://iiif.archivelab.org/iiif/opencontext-16-275jpg/full/675,/0/default.jpg\n",
      "HTTP Error 404: NOT FOUND\n",
      "Download error for URL https://artiraq.org/static/opencontext/poggio-civitate/preview/photos/20000005FRONT.jpg\n",
      "HTTP Error 404: Not Found\n",
      "Download error for URL https://artiraq.org/static/opencontext/poggio-civitate/preview/photos/20080087HEAD2.jpg\n",
      "HTTP Error 404: Not Found\n",
      "Download error for URL https://artiraq.org/static/opencontext/poggio-civitate/preview/photos/19780209BOTTOM.jpg\n",
      "HTTP Error 404: Not Found\n",
      "Download error for URL https://artiraq.org/static/opencontext/poggio-civitate/preview/photos/19790199TOP.jpg\n",
      "HTTP Error 404: Not Found\n",
      "Download error for URL https://artiraq.org/static/opencontext/poggio-civitate/preview/photos/20030050FRONT.jpg\n",
      "HTTP Error 404: Not Found\n",
      "Download error for URL https://artiraq.org/static/opencontext/poggio-civitate/preview/photos/20020023HEAD.jpg\n",
      "HTTP Error 404: Not Found\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import json\n",
    "import requests\n",
    "import pandas as pd\n",
    "from urllib.parse import urlparse\n",
    "from urllib.request import urlretrieve\n",
    "from urllib.error import HTTPError, URLError\n",
    "from sklearn.model_selection import train_test_split\n",
    "import concurrent.futures\n",
    "\n",
    "# Load JSON from remote URL\n",
    "url = \"https://raw.githubusercontent.com/opencontext/archaeology-images-ai/main/json_data/artifact_images_w_sentence_captions.json\"\n",
    "response = requests.get(url)\n",
    "data = response.json()\n",
    "\n",
    "# Randomly select records\n",
    "df = pd.DataFrame(data)\n",
    "train_df, rem_df = train_test_split(df, train_size=5000, random_state=42)\n",
    "test_df = rem_df.sample(500, random_state=42)\n",
    "\n",
    "def download_and_rename(row, folder):\n",
    "    os.makedirs(folder, exist_ok=True)\n",
    "    uri = row['image_file__uri']\n",
    "    # Check if uri exists and is a string\n",
    "    if uri and isinstance(uri, str):\n",
    "        uuid = row['media__uuid']\n",
    "        caption = row['caption']\n",
    "        parse_object = urlparse(uri)\n",
    "        _, ext = os.path.splitext(parse_object.path)\n",
    "        # Make sure uuid and ext are strings\n",
    "        if not isinstance(uuid, str):\n",
    "            uuid = str(uuid)\n",
    "        if isinstance(ext, bytes):\n",
    "            ext = ext.decode(\"utf-8\") \n",
    "        new_image_name = uuid + ext\n",
    "        new_image_path = os.path.join(folder, new_image_name)\n",
    "\n",
    "        try:\n",
    "            urlretrieve(uri, new_image_path)\n",
    "            return {\"image\": new_image_path, \"caption\": caption}\n",
    "\n",
    "        except (HTTPError, URLError) as error:\n",
    "            print(f\"Download error for URL {uri}\")\n",
    "            print(error)\n",
    "            return None\n",
    "    else:\n",
    "        return None\n",
    "\n",
    "# Writing to 'jsonl' files\n",
    "def write_to_jsonl(new_data, jsonl_file):\n",
    "    with open(jsonl_file, 'w') as file:\n",
    "        for json_dict in new_data:\n",
    "            line = json.dumps(json_dict)\n",
    "            file.write(line + \"\\n\")\n",
    "\n",
    "# Process train and test data\n",
    "with concurrent.futures.ThreadPoolExecutor() as executor:\n",
    "    train_data = list(executor.map(download_and_rename, [row for _, row in train_df.iterrows()], ['images']*len(train_df)))\n",
    "    test_data = list(executor.map(download_and_rename, [row for _, row in test_df.iterrows()], ['testing']*len(test_df)))\n",
    "\n",
    "# Write train/test data to jsonl files\n",
    "write_to_jsonl(train_data, 'train.json')\n",
    "write_to_jsonl(test_data, 'test.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add Other Sources of Imagery to Complement?\n",
    "\n",
    "Let's try the met"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install jsonlines\n",
    "!pip install retry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "import jsonlines\n",
    "import random\n",
    "from concurrent.futures import ThreadPoolExecutor\n",
    "\n",
    "# Function to fetch object data\n",
    "def fetch_object_data(object_id):\n",
    "    object_response = requests.get(f\"{base_url}objects/{object_id}\")\n",
    "    return object_response.json()\n",
    "\n",
    "# Define base URL for the Met's API\n",
    "base_url = 'https://collectionapi.metmuseum.org/public/collection/v1/'\n",
    "\n",
    "# Define our search term\n",
    "search_term = 'archaeology'\n",
    "\n",
    "allowed_departments = [\"Ancient Near Eastern Art\", \"Egyptian Art\", \"Greek and Roman Art\"]\n",
    "\n",
    "# Generate the search URL\n",
    "search_url = f\"{base_url}search?q={search_term}\"\n",
    "\n",
    "# Make the GET request to the Met's API search endpoint\n",
    "response = requests.get(search_url)\n",
    "\n",
    "# Parse the response as JSON\n",
    "data = response.json()\n",
    "\n",
    "# Get a random sample of 100 object IDs, if there are at least 100 object IDs. \n",
    "# Otherwise, get all object IDs.\n",
    "object_ids_sample = random.sample(data['objectIDs'], min(1000, len(data['objectIDs'])))\n",
    "\n",
    "# Open the jsonlines file in write mode\n",
    "with jsonlines.open('METoutput.json', mode='w') as writer:\n",
    "    # Use a ThreadPoolExecutor for parallel requests\n",
    "    with ThreadPoolExecutor(max_workers=5) as executor:\n",
    "        # Fetch all object data in parallel\n",
    "        for object_data in executor.map(fetch_object_data, object_ids_sample):\n",
    "            # If object's department in allowed departments and there's an image for this object\n",
    "            if (object_data.get('department') in allowed_departments) and object_data['primaryImage']:\n",
    "                # Create a list with all components of the caption\n",
    "                caption_components = [\n",
    "                    object_data['title'],\n",
    "                    f\"a {object_data['objectName']}\" if object_data.get('objectName') else None,\n",
    "                    f\"from the {object_data['culture']}\" if object_data.get('culture') else None,\n",
    "                    f\"dating to the {object_data['period']}\" if object_data.get('period') else None,\n",
    "                    object_data['dynasty'] if object_data.get('dynasty') else None,\n",
    "                    object_data['reign'] if object_data.get('reign') else None,\n",
    "                    f\"({object_data['objectDate']})\" if object_data.get('objectDate') else None,\n",
    "                    f\"created by {object_data['artistDisplayName']}\" if object_data.get('artistDisplayName') else None,\n",
    "                    f\"in {object_data['country']}\" if object_data.get('country') else None,\n",
    "                    object_data['region'] if object_data.get('region') else None\n",
    "                ]\n",
    "\n",
    "                # Remove None elements from the list\n",
    "                caption_components = [component for component in caption_components if component is not None]\n",
    "\n",
    "                # Create the caption\n",
    "                caption = ', '.join(caption_components) + '.'\n",
    "\n",
    "                # Create the record\n",
    "                record = {\n",
    "                    'image': object_data['primaryImage'],\n",
    "                    'caption': caption\n",
    "                }\n",
    "\n",
    "                # Write to jsonlines file\n",
    "                writer.write(record)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import jsonlines\n",
    "import urllib.request\n",
    "from concurrent.futures import ThreadPoolExecutor\n",
    "from sklearn.model_selection import train_test_split\n",
    "import requests\n",
    "from retry import retry\n",
    "\n",
    "# Function to download images\n",
    "@retry(tries=3, delay=2)\n",
    "def download_image(image_url, local_path):\n",
    "    try:\n",
    "        response = requests.get(image_url, stream=True)\n",
    "        response.raise_for_status()\n",
    "        with open(local_path, 'wb') as f:\n",
    "            f.write(response.content)\n",
    "    except requests.exceptions.RequestException as err:\n",
    "        print (\"Requests Error-URL {0}: {1}\".format(image_url,str(err)))\n",
    "        raise Exception(err)\n",
    "\n",
    "def process_lines(lines, dataset):\n",
    "    # List to store records\n",
    "    records = []\n",
    "\n",
    "    # Create the directory if it doesn't exist\n",
    "    os.makedirs(dataset, exist_ok=True)\n",
    "\n",
    "    for line in lines:\n",
    "        # Parse the line as JSON\n",
    "        data = json.loads(line)\n",
    "\n",
    "        # Define the local path\n",
    "        image_url = data['image']\n",
    "        local_filename = image_url.split('/')[-1]  # Use the last part of the URL as the filename\n",
    "        local_path = os.path.join(dataset, local_filename)\n",
    "\n",
    "        # Append this task to the list\n",
    "        records.append((image_url, local_path, data['caption']))\n",
    "\n",
    "    # Create ThreadPoolExecutor\n",
    "    with ThreadPoolExecutor(max_workers=10) as executor:\n",
    "        # Download images in parallel\n",
    "        executor.map(lambda x: download_image(x[0], x[1]), records)\n",
    "\n",
    "    # Open the corresponding jsonl file in write mode\n",
    "    with jsonlines.open(f'{dataset}_output.json', mode='w') as writer:\n",
    "        # Write records to file\n",
    "        for _, local_path, caption in records:\n",
    "            record = {\n",
    "                'image': local_path,\n",
    "                'caption': caption\n",
    "            }\n",
    "            writer.write(record)\n",
    "\n",
    "# Read lines from METoutput.jsonl file\n",
    "with open('METoutput.json', 'r') as f:\n",
    "    lines = f.readlines()\n",
    "\n",
    "# Split into train and test sets\n",
    "train_lines, test_lines = train_test_split(lines, test_size=0.20)\n",
    "\n",
    "# Process training and test sets\n",
    "process_lines(train_lines, 'METtrain')\n",
    "process_lines(test_lines, 'METtest')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open 'train.json' in append mode and 'METtrain_output.jsonl' in read mode\n",
    "with open('train.json', 'a') as train_file, open('METtrain_output.json', 'r') as met_file:\n",
    "    # Iterate over the lines in met_file\n",
    "    for line in met_file:\n",
    "        # Write each line to train_file\n",
    "        train_file.write(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine Tune Clip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6lBEQ2l3FWzW"
   },
   "outputs": [],
   "source": [
    "!pip install torchvision datasets Pillow\n",
    "!pip install -q git+https://github.com/huggingface/transformers\n",
    "!pip install accelerate -U"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8VtV89LzEtFd",
    "outputId": "d086f6fa-05a6-4ad9-9074-aa9a3c8b6501"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "acc4ef54ca8d48f1a74fdf3a4310aced",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "218415957aad4a53be87dfddebad6663",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Extracting data files:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9148425c58a8452c8b75fc09443e059f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "first image: images/5118cd46-6af6-4e71-8043-fa73020a418e.JPG, caption: 'An image of an archaeological artifact found at PKAP Survey Area, a place in Cyprus which is more generally located in Europe. Additional attributes that describe the artifact include: Chronotype: Phocaean Ware 10'\n"
     ]
    }
   ],
   "source": [
    "# test loading it back in\n",
    "from datasets import load_dataset\n",
    "dataset = load_dataset(\"json\", data_files=\"train.json\")\n",
    "print(f\"first image: {dataset['train'][0]['image']}, caption: '{dataset['train'][0]['caption']}'\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "eSYAUygoEUcB",
    "outputId": "8debe8e9-2965-4947-9dfe-1d6c80069cc3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'finetune-clip-huggingface'...\n",
      "remote: Enumerating objects: 19, done.\u001b[K\n",
      "remote: Counting objects: 100% (19/19), done.\u001b[K\n",
      "remote: Compressing objects: 100% (14/14), done.\u001b[K\n",
      "remote: Total 19 (delta 6), reused 17 (delta 4), pack-reused 0\u001b[K\n",
      "Receiving objects: 100% (19/19), 13.79 KiB | 336.00 KiB/s, done.\n",
      "Resolving deltas: 100% (6/6), done.\n"
     ]
    }
   ],
   "source": [
    "!git clone https://github.com/damian0815/finetune-clip-huggingface.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "kEDIyKCxIja1"
   },
   "outputs": [],
   "source": [
    "!mkdir results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "0sqCIk9rHLS9"
   },
   "outputs": [],
   "source": [
    "repo_id =  \"openai/clip-vit-base-patch32\" # this was the clip version for stable diffusion 1.5\n",
    "#repo_id = \"laion/CLIP-ViT-H-14-laion2B-s32B-b79K\" # this was the clip version for stable diffusion 2.0 onwards\n",
    "# however, using it requires more memory than I have available. More than what's available free tier google colab too.\n",
    "output_folder = \"results\"\n",
    "batch_size = 100\n",
    "num_train_epochs = 5\n",
    "out_json = \"train.json\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9WWt_a0kHRkn",
    "outputId": "ed3e3a6d-a7a2-4d25-8a3c-185eca396caf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finetuning openai/clip-vit-base-patch32 for 5 epochs with batch size 100, and then saving output to results.\n",
      "10/11/2023 13:22:24 - WARNING - __main__ - Process rank: 0, device: mps, n_gpu: 1distributed training: True, 16-bits training: False\n",
      "Filter: 100%|██████████████████████| 5220/5220 [00:00<00:00, 6050.89 examples/s]\n",
      "Running tokenizer on train dataset: 100%|█| 5204/5204 [00:00<00:00, 12214.67 exa\n",
      "Parameter 'transform'=<function main.<locals>.transform_images at 0x287878160> of the transform datasets.arrow_dataset.Dataset.set_format couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\n",
      "10/11/2023 13:22:28 - WARNING - datasets.fingerprint - Parameter 'transform'=<function main.<locals>.transform_images at 0x287878160> of the transform datasets.arrow_dataset.Dataset.set_format couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\n",
      "  6%|██▍                                     | 16/265 [04:51<1:14:13, 17.89s/it]"
     ]
    }
   ],
   "source": [
    "print(f\"Finetuning {repo_id} for {num_train_epochs} epochs with batch size {batch_size}, and then saving output to {output_folder}.\")\n",
    "!python -W ignore finetune-clip-huggingface/huggingface_finetune_clip.py \\\n",
    "    --output_dir {output_folder} \\\n",
    "    --model_name_or_path {repo_id} \\\n",
    "    --train_file {out_json} \\\n",
    "    --image_column image \\\n",
    "    --overwrite_output_dir=True \\\n",
    "    --max_seq_length=77 \\\n",
    "    --num_train_epochs={num_train_epochs} \\\n",
    "    --caption_column caption \\\n",
    "    --remove_unused_columns=False \\\n",
    "    --do_train \\\n",
    "    --per_device_train_batch_size={batch_size} \\\n",
    "    --learning_rate=\"5e-5\" --warmup_steps=\"2\" --weight_decay 0.2\n",
    "print(\"--\\nDONE\")\n",
    "print(f\"If it worked, trained data should be in {output_folder}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ehZmOm-SefWX",
    "outputId": "19576645-9020-46ad-d7cf-c247fd611799"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "updating: results/pytorch_model.bin (deflated 7%)\n",
      "updating: results/config.json (deflated 46%)\n"
     ]
    }
   ],
   "source": [
    "!zip -r archaeai.zip results/pytorch_model.bin results/config.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 17
    },
    "id": "I9bjpAP1hz2-",
    "outputId": "9c6379c6-236b-4bcc-8f4d-f5f823c4808d"
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "    async function download(id, filename, size) {\n",
       "      if (!google.colab.kernel.accessAllowed) {\n",
       "        return;\n",
       "      }\n",
       "      const div = document.createElement('div');\n",
       "      const label = document.createElement('label');\n",
       "      label.textContent = `Downloading \"${filename}\": `;\n",
       "      div.appendChild(label);\n",
       "      const progress = document.createElement('progress');\n",
       "      progress.max = size;\n",
       "      div.appendChild(progress);\n",
       "      document.body.appendChild(div);\n",
       "\n",
       "      const buffers = [];\n",
       "      let downloaded = 0;\n",
       "\n",
       "      const channel = await google.colab.kernel.comms.open(id);\n",
       "      // Send a message to notify the kernel that we're ready.\n",
       "      channel.send({})\n",
       "\n",
       "      for await (const message of channel.messages) {\n",
       "        // Send a message to notify the kernel that we're ready.\n",
       "        channel.send({})\n",
       "        if (message.buffers) {\n",
       "          for (const buffer of message.buffers) {\n",
       "            buffers.push(buffer);\n",
       "            downloaded += buffer.byteLength;\n",
       "            progress.value = downloaded;\n",
       "          }\n",
       "        }\n",
       "      }\n",
       "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
       "      const a = document.createElement('a');\n",
       "      a.href = window.URL.createObjectURL(blob);\n",
       "      a.download = filename;\n",
       "      div.appendChild(a);\n",
       "      a.click();\n",
       "      div.remove();\n",
       "    }\n",
       "  "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "download(\"download_232a83a1-c6e5-420b-a68e-47c014cbf00c\", \"archaeai.zip\", 560957788)"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from google.colab import files\n",
    "files.download(\"archaeai.zip\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GPT4's 'insight' re the warning in the training script:\n",
    "\n",
    "> That warning occurs because Python's `pickle` module is not able to serialize the function passed to `.map()`. In this case, the function is `transform_images`.\n",
    "\n",
    "> Because the function `transform_images` is defined within the `main()` function, it technically includes references to all the local variables in `main()`, making it a closure. Closures are not serializable with `pickle`.\n",
    "\n",
    "> A way around this would be to move the `transform_images` function out of the scope of the `main()` function. This means the function would be more of a \"global\" function rather than a local one, and it should remove the warning.\n",
    "\n",
    "> Here's how you could modify it:\n",
    "\n",
    "```python\n",
    "def transform_images(examples, image_column, image_transformations):\n",
    "    images = [read_image(image_file, mode=ImageReadMode.RGB) for image_file in examples[image_column]]\n",
    "    examples[\"pixel_values\"] = [image_transformations(image) for image in images]\n",
    "    return examples\n",
    "```\n",
    "\n",
    "> And then within your main function, you can use it within the `.map()` function as:\n",
    "\n",
    "```python\n",
    "train_dataset.set_transform(lambda examples: transform_images(examples, image_column, image_transformations))\n",
    "eval_dataset.set_transform(lambda examples: transform_images(examples, image_column, image_transformations))\n",
    "```\n",
    "\n",
    ">By passing the parameters explicitly and moving the function definition outside of the `main` function, it ensures that the function can be serialized and cached properly."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}