added quantization notebook

docs/72_quantization/quantization.ipynb

@@ -0,0 +1,227 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "96fd1423-4a3f-43ae-9803-bd8e2110cb93",
+   "metadata": {},
+   "source": [
+    "# Quantization\n",
+    "\n",
+    "In this notebook we demonstrate how models can be quantized to save memory. Note that the quantized model is not just smaller but may also perform worse.\n",
+    "\n",
+    "Read more\n",
+    "* [Quantization in Huggingface Transformers documentation](https://huggingface.co/docs/transformers/main/en/quantization/overview)\n",
+    "* [Quantization using bitsandbytes](https://huggingface.co/docs/transformers/main/en/quantization/bitsandbytes)\n",
+    "* [Blog post about 8-bit quantization using bitsandbytes](https://huggingface.co/blog/hf-bitsandbytes-integration)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "a02ffeb1-1f8d-4051-9bf1-1658edce16b9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\n",
+    "from utilities import calculate_model_memory_in_gb\n",
+    "import torch"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "17369f45-2bd3-45bd-8172-29d8761f809e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model_name = \"google/gemma-2b-it\"\n",
+    "attn_implementation = \"eager\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8aeedc5a-3ed3-4abc-b864-b8a03ee4671e",
+   "metadata": {},
+   "source": [
+    "This is the very normal way to load a model from Huggingface. Note that we specify to store the model in RAM, and not in GPU memory. This makes sense as we do not plan to run the model, and CPU typically has access to more memory. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "1282fc16-0737-4c6c-a32c-cca9546175c2",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "`config.hidden_act` is ignored, you should use `config.hidden_activation` instead.\n",
+      "Gemma's activation function will be set to `gelu_pytorch_tanh`. Please, use\n",
+      "`config.hidden_activation` if you want to override this behaviour.\n",
+      "See https://github.com/huggingface/transformers/pull/29402 for more details.\n"
+     ]
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "79e5f5a054c9469f9f88b23e7c7fb962",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "model = AutoModelForCausalLM.from_pretrained(\n",
+    "    model_name,\n",
+    "    device_map=\"cpu\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e04fb7ef-db4f-41b2-bd85-9a5c5b23a832",
+   "metadata": {},
+   "source": [
+    "We can then determine the model size in memory:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "eeb96995-2889-4637-b80b-534bf28dddc3",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "9.336219787597656"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "calculate_model_memory_in_gb(model)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f80b3b70-e6d1-4e64-b82e-8eec72adf696",
+   "metadata": {},
+   "source": [
+    "## 8-bit quantization\n",
+    "\n",
+    "We will now load the model again with a defined 8-bit quantization configuration."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "5073ee43-de9d-49b4-9f58-3419a02ad8c9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "bnb_config = BitsAndBytesConfig(\n",
+    "    load_in_8bit=True\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "843c1267-3224-46c4-9d20-65fd375f5896",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "67b0f9a20d2149b99c7ea59304af5675",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "quantized_model = AutoModelForCausalLM.from_pretrained(\n",
+    "    model_name,\n",
+    "    quantization_config=bnb_config,\n",
+    "    device_map=\"cpu\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "1dad8eb5-39ae-4ae8-86f5-51c1e2d9e3c3",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "4.668109893798828"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "calculate_model_memory_in_gb(quantized_model)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "196a61ae-2008-474f-8ccc-1b4b04b0da54",
+   "metadata": {},
+   "source": [
+    "## Exercise\n",
+    "Explore alternative [Quantization configurations](https://huggingface.co/docs/transformers/main/en/main_classes/quantization#transformers.BitsAndBytesConfig) and try to make the model as small as possible."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e7176b15-0bfe-4f36-9ec7-7b166eb08157",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/72_quantization/utilities.py

@@ -0,0 +1,21 @@
+def get_folder_size_in_gb(directory):
+    """
+    Inspects a folder recursively and returns it size in gigabytes.
+    """
+    import os
+    from os.path import join, getsize
+    from os import walk
+    total_size = 0
+    for dirpath, dirnames, filenames in walk(directory):
+        for filename in filenames:
+            file_path = join(dirpath, filename)
+            total_size += getsize(file_path)
+    return total_size / (1024 ** 3)
+
+def calculate_model_memory_in_gb(model):
+    """
+    Inspects a pytorch model and returns its size in memory in gigabytes.
+    """
+    total_size_in_bytes = sum(p.numel() * p.element_size() for p in model.parameters())
+    size_in_mb = total_size_in_bytes / (1024 ** 3)
+    return size_in_mb

docs/_toc.yml

@@ -135,6 +135,7 @@ parts:
       - file: 71_fine_tuning_hf/merging_model.ipynb
       - file: 71_fine_tuning_hf/test_model.ipynb
       - file: 71_fine_tuning_hf/hf_data_upload.ipynb
+      - file: 72_quantization/quantization.ipynb
 
     - file: 80_benchmarking_llms/readme.md
       sections: