Lab_3_fine_tune_model_to_detoxify_summaries.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d7f10b19-c061-42b2-8b42-e3cbafa3b1da",
   "metadata": {},
   "source": [
    "# Fine-Tune FLAN-T5 with Reinforcement Learning (PPO) and PEFT to Generate Less-Toxic Summaries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36ef668a-9c51-489b-be47-a07a09ef2289",
   "metadata": {},
   "source": [
    "We then fine-tuned a FLAN-T5 model to generate less toxic content with Meta AI's hate speech reward model. The reward model is a binary classifier that predicts either \"not hate\" or \"hate\" for the given text. We used Proximal Policy Optimization (PPO) to fine-tune and reduce the model's toxicity."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed5003e2-a642-416b-bd3f-93fb339c3a7d",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Table of Contents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6791f449-d1da-461b-9eb7-c6dc6b37b15c",
   "metadata": {
    "tags": []
   },
   "source": [
    "- [ 1 - Set up Kernel and Required Dependencies](#1)\n",
    "- [ 2 - Load FLAN-T5 Model, Prepare Reward Model and Toxicity Evaluator](#2)\n",
    "  - [ 2.1 - Load Data and FLAN-T5 Model Fine-Tuned with Summarization Instruction](#2.1)\n",
    "  - [ 2.2 - Prepare Reward Model](#2.2)\n",
    "  - [ 2.3 - Evaluate Toxicity](#2.3)\n",
    "- [ 3 - Perform Fine-Tuning to Detoxify the Summaries](#3)\n",
    "  - [ 3.1 - Initialize `PPOTrainer`](#3.1)\n",
    "  - [ 3.2 - Fine-Tune the Model](#3.2)\n",
    "  - [ 3.3 - Evaluate the Model Quantitatively](#3.3)\n",
    "  - [ 3.4 - Evaluate the Model Qualitatively](#3.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89f973f1-f095-4915-86d0-bc16380da22d",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a name='1'></a>\n",
    "## 1 - Set up Kernel and Required Dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba0923b1-daaa-437c-a604-455d839b9877",
   "metadata": {
    "tags": []
   },
   "source": [
    "First, check that the correct kernel is chosen.\n",
    "\n",
    "<img src=\"images/kernel_set_up.png\" width=\"300\"/>\n",
    "\n",
    "You can click on that (top right of the screen) to see and check the details of the image, kernel, and instance type.\n",
    "\n",
    "<img src=\"images/w3_kernel_and_instance_type.png\" width=\"600\"/>\n",
    "\n",
    "<img src=\"data:image/svg+xml;base64,Cjxzdmcgd2lkdGg9IjgwMCIgaGVpZ2h0PSIxMjUiIHZpZXdCb3g9IjAgMCA4MDAgMTI1IiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPgogICAgPGRlZnM+CiAgICAgICAgPGxpbmVhckdyYWRpZW50IGlkPSJmYWRlR3JhZGllbnQiIHgxPSIwIiB4Mj0iMSI+CiAgICAgICAgICAgIDxzdG9wIG9mZnNldD0iMCUiIHN0b3AtY29sb3I9IiNGMEYwRjAiLz4KICAgICAgICAgICAgPHN0b3Agb2Zmc2V0PSIxMDAlIiBzdG9wLWNvbG9yPSIjRjBGMEYwIiBzdG9wLW9wYWNpdHk9IjAiLz4KICAgICAgICA8L2xpbmVhckdyYWRpZW50PgogICAgICAgIDxtYXNrIGlkPSJmYWRlTWFzayI+CiAgICAgICAgICAgIDxyZWN0IHg9IjAiIHk9IjAiIHdpZHRoPSI3NTAiIGhlaWdodD0iMTI1IiBmaWxsPSJ3aGl0ZSIvPgogICAgICAgICAgICA8cmVjdCB4PSI3NTAiIHk9IjAiIHdpZHRoPSI1MCIgaGVpZ2h0PSIxMjUiIGZpbGw9InVybCgjZmFkZUdyYWRpZW50KSIvPgogICAgICAgIDwvbWFzaz4KICAgIDwvZGVmcz4KICAgIDxwYXRoIGQ9Ik01MCw5NyBBNTAsNTAgMCAwIDEgNTMsMyBMNzk3LCAzIEw3OTcsOTcgTDUwLDk3IFoiIGZpbGw9IiNGMEYwRjAiIHN0cm9rZT0iI0UwRTBFMCIgc3Ryb2tlLXdpZHRoPSIxIiBtYXNrPSJ1cmwoI2ZhZGVNYXNrKSIvPgogICAgPHRleHQgeD0iMTAwIiB5PSIzNCIgZm9udC1mYW1pbHk9IkFyaWFsLCBzYW5zLXNlcmlmIiBmb250LXNpemU9IjE0IiBmaWxsPSIjMzMzMzMzIj5QbGVhc2UgbWFrZSBzdXJlIHRoYXQgeW91IGNob29zZTwvdGV4dD4gCiAgICA8dGV4dCB4PSIzMjAiIHk9IjM0IiBmb250LWZhbWlseT0iQXJpYWwsIHNhbnMtc2VyaWYiIGZvbnQtc2l6ZT0iMTQiIGZpbGw9IiMzMzMzMzMiIGZvbnQtd2VpZ2h0PSJib2xkIj5tbC5tNS4yeGxhcmdlPC90ZXh0PgogICAgPHRleHQgeD0iNDE4IiB5PSIzNCIgZm9udC1mYW1pbHk9IkFyaWFsLCBzYW5zLXNlcmlmIiBmb250LXNpemU9IjE0IiBmaWxsPSIjMzMzMzMzIj5pbnN0YW5jZSB0eXBlLjwvdGV4dD4KICAgIDx0ZXh0IHg9IjEwMCIgeT0iNTYiIGZvbnQtZmFtaWx5PSJBcmlhbCwgc2Fucy1zZXJpZiIgZm9udC1zaXplPSIxNCIgZmlsbD0iIzMzMzMzMyI+VG8gZmluZCB0aGF0IGluc3RhbmNlIHR5cGUsIHlvdSBtaWdodCBoYXZlIHRvIHNjcm9sbCBkb3duIHRvIHRoZSAiQWxsIEluc3RhbmNlcyIgc2VjdGlvbiBpbiB0aGUgZHJvcGRvd24uPC90ZXh0PgogICAgPHRleHQgeD0iMTAwIiB5PSI3OCIgZm9udC1mYW1pbHk9IkFyaWFsLCBzYW5zLXNlcmlmIiBmb250LXNpemU9IjE0IiBmaWxsPSIjMzMzMzMzIj5DaG9pY2Ugb2YgYW5vdGhlciBpbnN0YW5jZSB0eXBlIG1pZ2h0IGNhdXNlIHRyYWluaW5nIGZhaWx1cmUva2VybmVsIGhhbHQvYWNjb3VudCBkZWFjdGl2YXRpb24uPC90ZXh0Pgo8L3N2Zz4K\" alt=\"Time alert close\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "edfe60ed-c9fe-4387-ac1f-57f1451e8541",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expected instance type: instance-datascience-ml-m5-2xlarge\n",
      "Currently chosen instance type: instance-datascience-ml-m5-2xlarge\n",
      "Instance type has been chosen correctly.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "instance_type_expected = 'ml-m5-2xlarge'\n",
    "instance_type_current = os.environ.get('HOSTNAME')\n",
    "\n",
    "print(f'Expected instance type: instance-datascience-{instance_type_expected}')\n",
    "print(f'Currently chosen instance type: {instance_type_current}')\n",
    "\n",
    "assert instance_type_expected in instance_type_current, f'ERROR. You selected the {instance_type_current} instance type. Please select {instance_type_expected} instead as shown on the screenshot above'\n",
    "print(\"Instance type has been chosen correctly.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43c0b396-8dab-4937-907e-734df9e2442c",
   "metadata": {
    "tags": []
   },
   "source": [
    "Now install the required packages to use PyTorch and Hugging Face transformers and datasets.\n",
    "\n",
    "<img src=\"data:image/svg+xml;base64,Cjxzdmcgd2lkdGg9IjgwMCIgaGVpZ2h0PSIxMjUiIHZpZXdCb3g9IjAgMCA4MDAgMTI1IiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPgogICAgPGRlZnM+CiAgICAgICAgPGxpbmVhckdyYWRpZW50IGlkPSJmYWRlR3JhZGllbnQiIHgxPSIwIiB4Mj0iMSI+CiAgICAgICAgICAgIDxzdG9wIG9mZnNldD0iMCUiIHN0b3AtY29sb3I9IiNGMEYwRjAiLz4KICAgICAgICAgICAgPHN0b3Agb2Zmc2V0PSIxMDAlIiBzdG9wLWNvbG9yPSIjRjBGMEYwIiBzdG9wLW9wYWNpdHk9IjAiLz4KICAgICAgICA8L2xpbmVhckdyYWRpZW50PgogICAgICAgIDxtYXNrIGlkPSJmYWRlTWFzayI+CiAgICAgICAgICAgIDxyZWN0IHg9IjAiIHk9IjAiIHdpZHRoPSI3NTAiIGhlaWdodD0iMTI1IiBmaWxsPSJ3aGl0ZSIvPgogICAgICAgICAgICA8cmVjdCB4PSI3NTAiIHk9IjAiIHdpZHRoPSI1MCIgaGVpZ2h0PSIxMjUiIGZpbGw9InVybCgjZmFkZUdyYWRpZW50KSIvPgogICAgICAgIDwvbWFzaz4KICAgIDwvZGVmcz4KICAgIDxwYXRoIGQ9Ik0zLDUwIEE1MCw1MCAwIDAgMSA1MywzIEw3OTcsMyBMNzk3LDk3IEw5Nyw5NyBMNTAsMTE1IEwzLDk3IFoiIGZpbGw9IiNGMEYwRjAiIHN0cm9rZT0iI0UwRTBFMCIgc3Ryb2tlLXdpZHRoPSIxIiBtYXNrPSJ1cmwoI2ZhZGVNYXNrKSIvPgogICAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgcj0iMzAiIGZpbGw9IiM1N2M0ZjgiIHN0cm9rZT0iIzU3YzRmOCIgc3Ryb2tlLXdpZHRoPSIxIi8+CiAgICA8Y2lyY2xlIGN4PSI1MCIgY3k9IjUwIiByPSIyNSIgZmlsbD0iI0YwRjBGMCIvPgogICAgPGxpbmUgeDE9IjUwIiB5MT0iNTAiIHgyPSI1MCIgeTI9IjMwIiBzdHJva2U9IiM1N2M0ZjgiIHN0cm9rZS13aWR0aD0iMyIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIi8+CiAgICA8bGluZSB4MT0iNTAiIHkxPSI1MCIgeDI9IjY1IiB5Mj0iNTAiIHN0cm9rZT0iIzU3YzRmOCIgc3Ryb2tlLXdpZHRoPSIzIiBzdHJva2UtbGluZWNhcD0icm91bmQiLz4KICAgIDx0ZXh0IHg9IjEwMCIgeT0iMzQiIGZvbnQtZmFtaWx5PSJBcmlhbCwgc2Fucy1zZXJpZiIgZm9udC1zaXplPSIxNCIgZmlsbD0iIzMzMzMzMyI+VGhlIG5leHQgY2VsbCBtYXkgdGFrZSBhIGZldyBtaW51dGVzIHRvIHJ1bi4gUGxlYXNlIGJlIHBhdGllbnQuPC90ZXh0PgogICAgPHRleHQgeD0iMTAwIiB5PSI1NiIgZm9udC1mYW1pbHk9IkFyaWFsLCBzYW5zLXNlcmlmIiBmb250LXNpemU9IjE0IiBmaWxsPSIjMzMzMzMzIj5JZ25vcmUgdGhlIHdhcm5pbmdzIGFuZCBlcnJvcnMsIGFsb25nIHdpdGggdGhlIG5vdGUgYWJvdXQgcmVzdGFydGluZyB0aGUga2VybmVsIGF0IHRoZSBlbmQuPC90ZXh0Pgo8L3N2Zz4K\" alt=\"Time alert open medium\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f9d24e86-f76f-4a44-90ef-0777752075a8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting datasets==2.17.0\n",
      "  Downloading datasets-2.17.0-py3-none-any.whl.metadata (20 kB)\n",
      "Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (3.6.0)\n",
      "Requirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (1.26.2)\n",
      "Requirement already satisfied: pyarrow>=12.0.0 in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (14.0.1)\n",
      "Collecting pyarrow-hotfix (from datasets==2.17.0)\n",
      "  Downloading pyarrow_hotfix-0.6-py3-none-any.whl.metadata (3.6 kB)\n",
      "Requirement already satisfied: dill<0.3.9,>=0.3.0 in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (0.3.7)\n",
      "Requirement already satisfied: pandas in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (2.1.3)\n",
      "Requirement already satisfied: requests>=2.19.0 in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (2.31.0)\n",
      "Requirement already satisfied: tqdm>=4.62.1 in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (4.64.1)\n",
      "Collecting xxhash (from datasets==2.17.0)\n",
      "  Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n",
      "Requirement already satisfied: multiprocess in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (0.70.15)\n",
      "Collecting fsspec<=2023.10.0,>=2023.1.0 (from fsspec[http]<=2023.10.0,>=2023.1.0->datasets==2.17.0)\n",
      "  Downloading fsspec-2023.10.0-py3-none-any.whl.metadata (6.8 kB)\n",
      "Collecting aiohttp (from datasets==2.17.0)\n",
      "  Downloading aiohttp-3.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.5 kB)\n",
      "Collecting huggingface-hub>=0.19.4 (from datasets==2.17.0)\n",
      "  Downloading huggingface_hub-0.22.2-py3-none-any.whl.metadata (12 kB)\n",
      "Requirement already satisfied: packaging in /opt/conda/lib/python3.10/site-packages (from datasets==2.17.0) (21.3)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.10/site-packages/PyYAML-6.0-py3.10-linux-x86_64.egg (from datasets==2.17.0) (6.0)\n",
      "Collecting aiosignal>=1.1.2 (from aiohttp->datasets==2.17.0)\n",
      "  Downloading aiosignal-1.3.1-py3-none-any.whl.metadata (4.0 kB)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets==2.17.0) (23.1.0)\n",
      "Collecting frozenlist>=1.1.1 (from aiohttp->datasets==2.17.0)\n",
      "  Downloading frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n",
      "Collecting multidict<7.0,>=4.5 (from aiohttp->datasets==2.17.0)\n",
      "  Downloading multidict-6.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)\n",
      "Collecting yarl<2.0,>=1.0 (from aiohttp->datasets==2.17.0)\n",
      "  Downloading yarl-1.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB)\n",
      "Collecting async-timeout<5.0,>=4.0 (from aiohttp->datasets==2.17.0)\n",
      "  Downloading async_timeout-4.0.3-py3-none-any.whl.metadata (4.2 kB)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.10/site-packages (from huggingface-hub>=0.19.4->datasets==2.17.0) (4.3.0)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.10/site-packages (from packaging->datasets==2.17.0) (3.0.9)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets==2.17.0) (2.0.4)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets==2.17.0) (3.3)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets==2.17.0) (2.1.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests>=2.19.0->datasets==2.17.0) (2023.11.17)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.10/site-packages (from pandas->datasets==2.17.0) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas->datasets==2.17.0) (2022.1)\n",
      "Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas->datasets==2.17.0) (2023.3)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->datasets==2.17.0) (1.16.0)\n",
      "Downloading datasets-2.17.0-py3-none-any.whl (536 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m536.6/536.6 kB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading fsspec-2023.10.0-py3-none-any.whl (166 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m166.4/166.4 kB\u001b[0m \u001b[31m2.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading aiohttp-3.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m14.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading huggingface_hub-0.22.2-py3-none-any.whl (388 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m388.9/388.9 kB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB)\n",
      "Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m2.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\n",
      "Downloading async_timeout-4.0.3-py3-none-any.whl (5.7 kB)\n",
      "Downloading frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (239 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m239.5/239.5 kB\u001b[0m \u001b[31m3.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading multidict-6.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (124 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m124.3/124.3 kB\u001b[0m \u001b[31m1.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading yarl-1.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (301 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m301.6/301.6 kB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: xxhash, pyarrow-hotfix, multidict, fsspec, frozenlist, async-timeout, yarl, huggingface-hub, aiosignal, aiohttp, datasets\n",
      "  Attempting uninstall: fsspec\n",
      "    Found existing installation: fsspec 2022.7.1\n",
      "    Uninstalling fsspec-2022.7.1:\n",
      "      Successfully uninstalled fsspec-2022.7.1\n",
      "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "distributed 2022.7.0 requires tornado<6.2,>=6.0.3, but you have tornado 6.4 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0mSuccessfully installed aiohttp-3.9.4 aiosignal-1.3.1 async-timeout-4.0.3 datasets-2.17.0 frozenlist-1.4.1 fsspec-2023.10.0 huggingface-hub-0.22.2 multidict-6.0.5 pyarrow-hotfix-0.6 xxhash-3.4.1 yarl-1.9.4\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Requirement already satisfied: pip in /opt/conda/lib/python3.10/site-packages (23.3.1)\n",
      "Collecting pip\n",
      "  Downloading pip-24.0-py3-none-any.whl.metadata (3.6 kB)\n",
      "Downloading pip-24.0-py3-none-any.whl (2.1 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m18.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: pip\n",
      "  Attempting uninstall: pip\n",
      "    Found existing installation: pip 23.3.1\n",
      "    Uninstalling pip-23.3.1:\n",
      "      Successfully uninstalled pip-23.3.1\n",
      "Successfully installed pip-24.0\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n",
      "Collecting git+https://github.com/lvwerra/trl.git@25fa1bd\n",
      "  Cloning https://github.com/lvwerra/trl.git (to revision 25fa1bd) to /tmp/pip-req-build-r_6ukrht\n",
      "  Running command git clone --filter=blob:none --quiet https://github.com/lvwerra/trl.git /tmp/pip-req-build-r_6ukrht\n",
      "\u001b[33m  WARNING: Did not find branch or tag '25fa1bd', assuming revision or ref.\u001b[0m\u001b[33m\n",
      "\u001b[0m  Running command git checkout -q 25fa1bd\n",
      "  Resolved https://github.com/lvwerra/trl.git to commit 25fa1bd\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: torch>=1.4.0 in /opt/conda/lib/python3.10/site-packages (from trl==0.4.2.dev0) (1.13.1)\n",
      "Requirement already satisfied: transformers>=4.18.0 in /opt/conda/lib/python3.10/site-packages (from trl==0.4.2.dev0) (4.27.2)\n",
      "Requirement already satisfied: numpy>=1.18.2 in /opt/conda/lib/python3.10/site-packages (from trl==0.4.2.dev0) (1.26.2)\n",
      "Requirement already satisfied: accelerate in /opt/conda/lib/python3.10/site-packages (from trl==0.4.2.dev0) (0.29.2)\n",
      "Requirement already satisfied: datasets in /opt/conda/lib/python3.10/site-packages (from trl==0.4.2.dev0) (2.17.0)\n",
      "Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch>=1.4.0->trl==0.4.2.dev0) (4.3.0)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /opt/conda/lib/python3.10/site-packages (from torch>=1.4.0->trl==0.4.2.dev0) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /opt/conda/lib/python3.10/site-packages (from torch>=1.4.0->trl==0.4.2.dev0) (8.5.0.96)\n",
      "Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /opt/conda/lib/python3.10/site-packages (from torch>=1.4.0->trl==0.4.2.dev0) (11.10.3.66)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /opt/conda/lib/python3.10/site-packages (from torch>=1.4.0->trl==0.4.2.dev0) (11.7.99)\n",
      "Requirement already satisfied: setuptools in /opt/conda/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.4.0->trl==0.4.2.dev0) (69.0.2)\n",
      "Requirement already satisfied: wheel in /opt/conda/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.4.0->trl==0.4.2.dev0) (0.42.0)\n",
      "Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from transformers>=4.18.0->trl==0.4.2.dev0) (3.6.0)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /opt/conda/lib/python3.10/site-packages (from transformers>=4.18.0->trl==0.4.2.dev0) (0.22.2)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from transformers>=4.18.0->trl==0.4.2.dev0) (21.3)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.10/site-packages/PyYAML-6.0-py3.10-linux-x86_64.egg (from transformers>=4.18.0->trl==0.4.2.dev0) (6.0)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.10/site-packages (from transformers>=4.18.0->trl==0.4.2.dev0) (2022.7.9)\n",
      "Requirement already satisfied: requests in /opt/conda/lib/python3.10/site-packages (from transformers>=4.18.0->trl==0.4.2.dev0) (2.31.0)\n",
      "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /opt/conda/lib/python3.10/site-packages (from transformers>=4.18.0->trl==0.4.2.dev0) (0.13.3)\n",
      "Requirement already satisfied: tqdm>=4.27 in /opt/conda/lib/python3.10/site-packages (from transformers>=4.18.0->trl==0.4.2.dev0) (4.64.1)\n",
      "Requirement already satisfied: psutil in /opt/conda/lib/python3.10/site-packages (from accelerate->trl==0.4.2.dev0) (5.9.0)\n",
      "Requirement already satisfied: safetensors>=0.3.1 in /opt/conda/lib/python3.10/site-packages (from accelerate->trl==0.4.2.dev0) (0.4.3)\n",
      "Requirement already satisfied: pyarrow>=12.0.0 in /opt/conda/lib/python3.10/site-packages (from datasets->trl==0.4.2.dev0) (14.0.1)\n",
      "Requirement already satisfied: pyarrow-hotfix in /opt/conda/lib/python3.10/site-packages (from datasets->trl==0.4.2.dev0) (0.6)\n",
      "Requirement already satisfied: dill<0.3.9,>=0.3.0 in /opt/conda/lib/python3.10/site-packages (from datasets->trl==0.4.2.dev0) (0.3.7)\n",
      "Requirement already satisfied: pandas in /opt/conda/lib/python3.10/site-packages (from datasets->trl==0.4.2.dev0) (2.1.3)\n",
      "Requirement already satisfied: xxhash in /opt/conda/lib/python3.10/site-packages (from datasets->trl==0.4.2.dev0) (3.4.1)\n",
      "Requirement already satisfied: multiprocess in /opt/conda/lib/python3.10/site-packages (from datasets->trl==0.4.2.dev0) (0.70.15)\n",
      "Requirement already satisfied: fsspec<=2023.10.0,>=2023.1.0 in /opt/conda/lib/python3.10/site-packages (from fsspec[http]<=2023.10.0,>=2023.1.0->datasets->trl==0.4.2.dev0) (2023.10.0)\n",
      "Requirement already satisfied: aiohttp in /opt/conda/lib/python3.10/site-packages (from datasets->trl==0.4.2.dev0) (3.9.4)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets->trl==0.4.2.dev0) (1.3.1)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets->trl==0.4.2.dev0) (23.1.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets->trl==0.4.2.dev0) (1.4.1)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets->trl==0.4.2.dev0) (6.0.5)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets->trl==0.4.2.dev0) (1.9.4)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets->trl==0.4.2.dev0) (4.0.3)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.10/site-packages (from packaging>=20.0->transformers>=4.18.0->trl==0.4.2.dev0) (3.0.9)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests->transformers>=4.18.0->trl==0.4.2.dev0) (2.0.4)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests->transformers>=4.18.0->trl==0.4.2.dev0) (3.3)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests->transformers>=4.18.0->trl==0.4.2.dev0) (2.1.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests->transformers>=4.18.0->trl==0.4.2.dev0) (2023.11.17)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.10/site-packages (from pandas->datasets->trl==0.4.2.dev0) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas->datasets->trl==0.4.2.dev0) (2022.1)\n",
      "Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas->datasets->trl==0.4.2.dev0) (2023.3)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->datasets->trl==0.4.2.dev0) (1.16.0)\n",
      "Building wheels for collected packages: trl\n",
      "  Building wheel for trl (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for trl: filename=trl-0.4.2.dev0-py3-none-any.whl size=67532 sha256=09521dc6797e58c61715a6f66b5ff15cc5bfdaf25e8b21b062db007cc6e2cc2f\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-0swdybrz/wheels/24/b4/20/2fa3a1e47c0411c39e198029315e3af2a2c1d59132913f136f\n",
      "Successfully built trl\n",
      "Installing collected packages: trl\n",
      "Successfully installed trl-0.4.2.dev0\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install -U datasets==2.17.0\n",
    "\n",
    "%pip install --upgrade pip\n",
    "%pip install --disable-pip-version-check \\\n",
    "    torch==1.13.1 \\\n",
    "    torchdata==0.5.1 --quiet\n",
    "\n",
    "%pip install \\\n",
    "    transformers==4.27.2 \\\n",
    "    evaluate==0.4.0 \\\n",
    "    rouge_score==0.1.2 \\\n",
    "    peft==0.3.0 --quiet\n",
    "\n",
    "# Installing the Reinforcement Learning library directly from github.\n",
    "%pip install git+https://github.com/lvwerra/trl.git@25fa1bd    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8f3c076-d9d2-40e3-b005-9dd66b5a163a",
   "metadata": {
    "tags": []
   },
   "source": [
    "<img src=\"data:image/svg+xml;base64,Cjxzdmcgd2lkdGg9IjgwMCIgaGVpZ2h0PSI1MCIgdmlld0JveD0iMCAwIDgwMCA1MCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KICAgIDxkZWZzPgogICAgICAgIDxsaW5lYXJHcmFkaWVudCBpZD0iZmFkZUdyYWRpZW50IiB4MT0iMCIgeDI9IjEiPgogICAgICAgICAgICA8c3RvcCBvZmZzZXQ9IjAlIiBzdG9wLWNvbG9yPSIjRjBGMEYwIi8+CiAgICAgICAgICAgIDxzdG9wIG9mZnNldD0iMTAwJSIgc3RvcC1jb2xvcj0iI0YwRjBGMCIgc3RvcC1vcGFjaXR5PSIwIi8+CiAgICAgICAgPC9saW5lYXJHcmFkaWVudD4KICAgICAgICA8bWFzayBpZD0iZmFkZU1hc2siPgogICAgICAgICAgICA8cmVjdCB4PSIwIiB5PSIwIiB3aWR0aD0iNzUwIiBoZWlnaHQ9IjUwIiBmaWxsPSJ3aGl0ZSIvPgogICAgICAgICAgICA8cmVjdCB4PSI3NTAiIHk9IjAiIHdpZHRoPSI1MCIgaGVpZ2h0PSI1MCIgZmlsbD0idXJsKCNmYWRlR3JhZGllbnQpIi8+CiAgICAgICAgPC9tYXNrPgogICAgPC9kZWZzPgogICAgPHBhdGggZD0iTTI1LDUwIFEwLDUwIDAsMjUgTDUwLDMgTDk3LDI1IEw3OTcsMjUgTDc5Nyw1MCBMMjUsNTAgWiIgZmlsbD0iI0YwRjBGMCIgc3Ryb2tlPSIjRTBFMEUwIiBzdHJva2Utd2lkdGg9IjEiIG1hc2s9InVybCgjZmFkZU1hc2spIi8+Cjwvc3ZnPgo=\" alt=\"Time alert close\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74bfd06e-c747-43e0-b86c-0398628e1c32",
   "metadata": {
    "tags": []
   },
   "source": [
    "Import the necessary components. Some of them are new for this week, they will be discussed later in the notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d8c20bed-6a30-4847-a507-02969ecb4465",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from transformers import pipeline, AutoTokenizer, AutoModelForSequenceClassification, AutoModelForSeq2SeqLM, GenerationConfig\n",
    "from datasets import load_dataset\n",
    "from peft import PeftModel, PeftConfig, LoraConfig, TaskType\n",
    "\n",
    "# trl: Transformer Reinforcement Learning library\n",
    "from trl import PPOTrainer, PPOConfig, AutoModelForSeq2SeqLMWithValueHead\n",
    "from trl import create_reference_model\n",
    "from trl.core import LengthSampler\n",
    "\n",
    "import torch\n",
    "import evaluate\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# tqdm library makes the loops show a smart progress meter.\n",
    "from tqdm import tqdm\n",
    "tqdm.pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b76eea84-8e3a-4487-9692-613977e6c8e3",
   "metadata": {},
   "source": [
    "<a name='2'></a>\n",
    "## 2 - Load FLAN-T5 Model, Prepare Reward Model and Toxicity Evaluator"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a5f97d4-ea5f-4072-b5d6-785d1d833ed4",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a name='2.1'></a>\n",
    "### 2.1 - Load Data and FLAN-T5 Model Fine-Tuned with Summarization Instruction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90dc0211-4032-4967-946d-3a538829d5c9",
   "metadata": {
    "tags": []
   },
   "source": [
    "You will keep working with the same Hugging Face dataset [DialogSum](https://huggingface.co/datasets/knkarthick/dialogsum) and the pre-trained model [FLAN-T5](https://huggingface.co/docs/transformers/model_doc/flan-t5). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b058b52b-ec4d-4426-8d71-91e898f727f6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0d17fd3088924744a91293430bc30841",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading readme:   0%|          | 0.00/4.65k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ba1aa1326118478887ca60b489c8beff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/11.3M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3fee58a08cbf4160b65849bbdd98125a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/442k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "648db46feab340529bdfdc5a0a8cd3a9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/1.35M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5f00c819c572446b9cbae126e005f220",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c5c28076837c49a8a7c3b42dbabe500d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating validation split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a2bf540f878842f6831907bb6c8b1930",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating test split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['id', 'dialogue', 'summary', 'topic'],\n",
       "        num_rows: 12460\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['id', 'dialogue', 'summary', 'topic'],\n",
       "        num_rows: 500\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['id', 'dialogue', 'summary', 'topic'],\n",
       "        num_rows: 1500\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_name=\"google/flan-t5-base\"\n",
    "huggingface_dataset_name = \"knkarthick/dialogsum\"\n",
    "\n",
    "dataset_original = load_dataset(huggingface_dataset_name)\n",
    "\n",
    "dataset_original"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "668d30d6-6f81-4e52-a81a-3057163ddb0e",
   "metadata": {},
   "source": [
    "The next step will be to preprocess the dataset. You will take only a part of it, then filter the dialogues of a particular length (just to make those examples long enough and, at the same time, easy to read). Then wrap each dialogue with the instruction and tokenize the prompts. Save the token ids in the field `input_ids` and decoded version of the prompts in the field `query`.\n",
    "\n",
    "You could do that all step by step in the cell below, but it is a good habit to organize that all in a function `build_dataset`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "51469abe-4d72-4093-a6c6-8e04e19f09eb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bdb0ffe7f63e4563b4b908705a0257a7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Filter:   0%|          | 0/12460 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7ab76227da99476cbe71eb751506167f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/2.54k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1a0c16d779ae42a691842b875dbcaf11",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "spiece.model:   0%|          | 0.00/792k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "39e0213147ff419485ba2bb81bc71925",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/2.42M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "78823e53053d4a7eb53fc9b7a945ac64",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/2.20k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "76f8068db2e54aa1a6f8d24846279b3b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/10022 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DatasetDict({\n",
      "    train: Dataset({\n",
      "        features: ['id', 'dialogue', 'summary', 'topic', 'input_ids', 'query'],\n",
      "        num_rows: 8017\n",
      "    })\n",
      "    test: Dataset({\n",
      "        features: ['id', 'dialogue', 'summary', 'topic', 'input_ids', 'query'],\n",
      "        num_rows: 2005\n",
      "    })\n",
      "})\n"
     ]
    }
   ],
   "source": [
    "def build_dataset(model_name,\n",
    "                  dataset_name,\n",
    "                  input_min_text_length, \n",
    "                  input_max_text_length):\n",
    "\n",
    "    \"\"\"\n",
    "    Preprocess the dataset and split it into train and test parts.\n",
    "\n",
    "    Parameters:\n",
    "    - model_name (str): Tokenizer model name.\n",
    "    - dataset_name (str): Name of the dataset to load.\n",
    "    - input_min_text_length (int): Minimum length of the dialogues.\n",
    "    - input_max_text_length (int): Maximum length of the dialogues.\n",
    "        \n",
    "    Returns:\n",
    "    - dataset_splits (datasets.dataset_dict.DatasetDict): Preprocessed dataset containing train and test parts.\n",
    "    \"\"\"\n",
    "    \n",
    "    # load dataset (only \"train\" part will be enough for this lab).\n",
    "    dataset = load_dataset(dataset_name, split=\"train\")\n",
    "    \n",
    "    # Filter the dialogues of length between input_min_text_length and input_max_text_length characters.\n",
    "    dataset = dataset.filter(lambda x: len(x[\"dialogue\"]) > input_min_text_length and len(x[\"dialogue\"]) <= input_max_text_length, batched=False)\n",
    "\n",
    "    # Prepare tokenizer. Setting device_map=\"auto\" allows to switch between GPU and CPU automatically.\n",
    "    tokenizer = AutoTokenizer.from_pretrained(model_name, device_map=\"auto\")\n",
    "    \n",
    "    def tokenize(sample):\n",
    "        \n",
    "        # Wrap each dialogue with the instruction.\n",
    "        prompt = f\"\"\"\n",
    "Summarize the following conversation.\n",
    "\n",
    "{sample[\"dialogue\"]}\n",
    "\n",
    "Summary:\n",
    "\"\"\"\n",
    "        sample[\"input_ids\"] = tokenizer.encode(prompt)\n",
    "        \n",
    "        # This must be called \"query\", which is a requirement of our PPO library.\n",
    "        sample[\"query\"] = tokenizer.decode(sample[\"input_ids\"])\n",
    "        return sample\n",
    "\n",
    "    # Tokenize each dialogue.\n",
    "    dataset = dataset.map(tokenize, batched=False)\n",
    "    dataset.set_format(type=\"torch\")\n",
    "    \n",
    "    # Split the dataset into train and test parts.\n",
    "    dataset_splits = dataset.train_test_split(test_size=0.2, shuffle=False, seed=42)\n",
    "\n",
    "    return dataset_splits\n",
    "\n",
    "dataset = build_dataset(model_name=model_name,\n",
    "                        dataset_name=huggingface_dataset_name,\n",
    "                        input_min_text_length=200, \n",
    "                        input_max_text_length=1000)\n",
    "\n",
    "print(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d03155e-649b-45bb-a5a0-94edd682c069",
   "metadata": {
    "tags": []
   },
   "source": [
    "In the previous lab, you fine-tuned the PEFT model with summarization instructions. The training in the notebook was done on a subset of data. Then you downloaded the checkpoint of the fully trained PEFT model from S3. \n",
    "\n",
    "Let's load the same model checkpoint here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e1d44a53-ea1f-4fa5-89e7-d46e37d19935",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
      "download: s3://dlai-generative-ai/models/peft-dialogue-summary-checkpoint/special_tokens_map.json to peft-dialogue-summary-checkpoint-from-s3/special_tokens_map.json\n",
      "download: s3://dlai-generative-ai/models/peft-dialogue-summary-checkpoint/adapter_config.json to peft-dialogue-summary-checkpoint-from-s3/adapter_config.json\n",
      "download: s3://dlai-generative-ai/models/peft-dialogue-summary-checkpoint/tokenizer_config.json to peft-dialogue-summary-checkpoint-from-s3/tokenizer_config.json\n",
      "download: s3://dlai-generative-ai/models/peft-dialogue-summary-checkpoint/tokenizer.json to peft-dialogue-summary-checkpoint-from-s3/tokenizer.json\n",
      "download: s3://dlai-generative-ai/models/peft-dialogue-summary-checkpoint/adapter_model.bin to peft-dialogue-summary-checkpoint-from-s3/adapter_model.bin\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp --recursive s3://dlai-generative-ai/models/peft-dialogue-summary-checkpoint/ ./peft-dialogue-summary-checkpoint-from-s3/ "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dec8bea4-addd-4b29-b3af-6db6ea2baeb7",
   "metadata": {
    "tags": []
   },
   "source": [
    "List the model item and check its size (it's less than 15 Mb):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4288240d-764b-4c49-8df7-b30b9277adbd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
      "-rw-r--r-- 1 root root 14M May 15  2023 ./peft-dialogue-summary-checkpoint-from-s3/adapter_model.bin\n"
     ]
    }
   ],
   "source": [
    "!ls -alh ./peft-dialogue-summary-checkpoint-from-s3/adapter_model.bin"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4226923-67c0-4ea6-8e47-030136b2f191",
   "metadata": {},
   "source": [
    "Prepare a function to pull out the number of model parameters (it is the same as in the previous lab):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a1f06806-a194-4c14-b64d-e31afd7b658c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def print_number_of_trainable_model_parameters(model):\n",
    "    trainable_model_params = 0\n",
    "    all_model_params = 0\n",
    "    for _, param in model.named_parameters():\n",
    "        all_model_params += param.numel()\n",
    "        if param.requires_grad:\n",
    "            trainable_model_params += param.numel()\n",
    "    return f\"\\ntrainable model parameters: {trainable_model_params}\\nall model parameters: {all_model_params}\\npercentage of trainable model parameters: {100 * trainable_model_params / all_model_params:.2f}%\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21e06a57-fb80-4f8c-a967-4c7c42a7bfda",
   "metadata": {
    "tags": []
   },
   "source": [
    "Add the adapter to the original FLAN-T5 model. In the previous lab you were adding the fully trained adapter only for inferences, so there was no need to pass LoRA configurations doing that. Now you need to pass them to the constructed PEFT model, also putting `is_trainable=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a1a94b14-b375-45e7-9e49-a7f2c341b4ff",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d39daea91214238842d2379bf29957f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/1.40k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "87328acdabc549449597588b357f6293",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/990M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "43cd5b930ca7451db3e90f2d4ed6eb2b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "generation_config.json:   0%|          | 0.00/147 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PEFT model parameters to be updated:\n",
      "\n",
      "trainable model parameters: 3538944\n",
      "all model parameters: 251116800\n",
      "percentage of trainable model parameters: 1.41%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "lora_config = LoraConfig(\n",
    "    r=32, # Rank\n",
    "    lora_alpha=32,\n",
    "    target_modules=[\"q\", \"v\"],\n",
    "    lora_dropout=0.05,\n",
    "    bias=\"none\",\n",
    "    task_type=TaskType.SEQ_2_SEQ_LM # FLAN-T5\n",
    ")\n",
    "\n",
    "model = AutoModelForSeq2SeqLM.from_pretrained(model_name, \n",
    "                                              torch_dtype=torch.bfloat16)\n",
    "\n",
    "peft_model = PeftModel.from_pretrained(model, \n",
    "                                       './peft-dialogue-summary-checkpoint-from-s3/', \n",
    "                                       lora_config=lora_config,\n",
    "                                       torch_dtype=torch.bfloat16, \n",
    "                                       device_map=\"auto\",                                       \n",
    "                                       is_trainable=True)\n",
    "\n",
    "print(f'PEFT model parameters to be updated:\\n{print_number_of_trainable_model_parameters(peft_model)}\\n')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a950ae8a-76b9-4951-9c78-9ac7a6349e17",
   "metadata": {},
   "source": [
    "In this lab, you are preparing to fine-tune the LLM using Reinforcement Learning (RL). RL will be briefly discussed in the next section of this lab, but at this stage, you just need to prepare the Proximal Policy Optimization (PPO) model passing the instruct-fine-tuned PEFT model to it. PPO will be used to optimize the RL policy against the reward model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1e86bab0-6dee-4dff-a754-b584ed962723",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Detected kernel version 4.14.336, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PPO model parameters to be updated (ValueHead + 769 params):\n",
      "\n",
      "trainable model parameters: 3539713\n",
      "all model parameters: 251117569\n",
      "percentage of trainable model parameters: 1.41%\n",
      "\n",
      "ValueHead(\n",
      "  (dropout): Dropout(p=0.1, inplace=False)\n",
      "  (summary): Linear(in_features=768, out_features=1, bias=True)\n",
      "  (flatten): Flatten(start_dim=1, end_dim=-1)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "ppo_model = AutoModelForSeq2SeqLMWithValueHead.from_pretrained(peft_model,                                                               \n",
    "                                                               torch_dtype=torch.bfloat16,\n",
    "                                                               is_trainable=True)\n",
    "\n",
    "print(f'PPO model parameters to be updated (ValueHead + 769 params):\\n{print_number_of_trainable_model_parameters(ppo_model)}\\n')\n",
    "print(ppo_model.v_head)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2913ef05-737e-4cdf-9bac-467ee6cf9f76",
   "metadata": {},
   "source": [
    "During PPO, only a few parameters will be updated. Specifically, the parameters of the `ValueHead`. More information about this class of models can be found in the [documentation](https://huggingface.co/docs/trl/main/en/models#trl.create_reference_model). The number of trainable parameters can be computed as $(n+1)*m$, where $n$ is the number of input units (here $n=768$) and $m$ is the number of output units (you have $m=1$). The $+1$ term in the equation takes into account the bias term."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76c7e2df-0c75-4bd0-bf8d-1f1545ef864e",
   "metadata": {},
   "source": [
    "Now create a frozen copy of the PPO which will not be fine-tuned - a reference model. The reference model will represent the LLM before detoxification. None of the parameters of the reference model will be updated during PPO training. This is on purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "18a9b30a-ad14-4189-8088-d4de447fe247",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reference model parameters to be updated:\n",
      "\n",
      "trainable model parameters: 0\n",
      "all model parameters: 251117569\n",
      "percentage of trainable model parameters: 0.00%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ref_model = create_reference_model(ppo_model)\n",
    "\n",
    "print(f'Reference model parameters to be updated:\\n{print_number_of_trainable_model_parameters(ref_model)}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3a14848-e83d-4fdd-bc68-eb770c5951d7",
   "metadata": {
    "tags": []
   },
   "source": [
    "Everything is set. It is time to prepare the reward model!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bfdf1f7-3509-4adc-812a-2b22bd330137",
   "metadata": {},
   "source": [
    "<a name='2.2'></a>\n",
    "### 2.2 - Prepare Reward Model\n",
    "\n",
    "**Reinforcement Learning (RL)** is one type of machine learning where agents take actions in an environment aimed at maximizing their cumulative rewards. The agent's behavior is defined by the **policy**. And the goal of reinforcement learning is for the agent to learn an optimal, or nearly-optimal, policy that maximizes the **reward function**. \n",
    "\n",
    "In the [previous section](#2.1) the original policy is based on the instruct PEFT model - this is the LLM before detoxification. Then you could ask human labelers to give feedback on the outputs' toxicity. However, it can be expensive to use them for the entire fine-tuning process. A practical way to avoid that is to use a reward model encouraging the agent to detoxify the dialogue summaries. The intuitive approach would be to do some form of sentiment analysis across two classes (`nothate` and `hate`) and give a higher reward if there is higher a chance of getting class `nothate` as an output. \n",
    "\n",
    "For example, we can mention that having human labelers for the entire finetuning process can be expensive. A practical way to avoid that is to use a reward model.\n",
    "\n",
    "use feedback generated by a model\n",
    "\n",
    "You will use [Meta AI's RoBERTa-based hate speech model](https://huggingface.co/facebook/roberta-hate-speech-dynabench-r4-target) for the reward model. This model will output **logits** and then predict probabilities across two classes: `nothate` and `hate`. The logits of the output `nothate` will be taken as a positive reward. Then, the model will be fine-tuned with PPO using those reward values.\n",
    "\n",
    "Create the instance of the required model class for the RoBERTa model. You also need to load a tokenizer to test the model. Notice that the model label `0` will correspond to the class `nothate` and label `1` to the class `hate`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "7f038a9f-e04b-49bc-8923-8ef3816919ee",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e994a0adb3c244089f7d33918e419fe8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/1.11k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4aacf868aeb647578b30e31e0b347482",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.json:   0%|          | 0.00/899k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "613ec0d6de0b4191b91df5b7734fcd6c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "690198614ff740c087d1bc5fdc3c40ea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/239 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cff1a4b4968c47258510d597e7a3ac9d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/816 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6456235d913c4302b24e6021132ded82",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/499M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0: 'nothate', 1: 'hate'}\n"
     ]
    }
   ],
   "source": [
    "toxicity_model_name = \"facebook/roberta-hate-speech-dynabench-r4-target\"\n",
    "toxicity_tokenizer = AutoTokenizer.from_pretrained(toxicity_model_name, device_map=\"auto\")\n",
    "toxicity_model = AutoModelForSequenceClassification.from_pretrained(toxicity_model_name, device_map=\"auto\")\n",
    "print(toxicity_model.config.id2label)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79d68799-a6e8-42d7-8d61-002e47210c18",
   "metadata": {
    "tags": []
   },
   "source": [
    "Take some non-toxic text, tokenize it, and pass it to the model. Print the output logits, probabilities, and the corresponding reward that will be used for fine-tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8f4e6a05-2398-4ca7-a176-d5a1ff27fe39",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logits [not hate, hate]: [3.114100694656372, -2.4896175861358643]\n",
      "probabilities [not hate, hate]: [0.9963293671607971, 0.003670616541057825]\n",
      "reward (high): [3.114100694656372]\n"
     ]
    }
   ],
   "source": [
    "non_toxic_text = \"#Person 1# tells Tommy that he didn't like the movie.\"\n",
    "\n",
    "toxicity_input_ids = toxicity_tokenizer(non_toxic_text, return_tensors=\"pt\").input_ids\n",
    "\n",
    "logits = toxicity_model(input_ids=toxicity_input_ids).logits\n",
    "print(f'logits [not hate, hate]: {logits.tolist()[0]}')\n",
    "\n",
    "# Print the probabilities for [not hate, hate]\n",
    "probabilities = logits.softmax(dim=-1).tolist()[0]\n",
    "print(f'probabilities [not hate, hate]: {probabilities}')\n",
    "\n",
    "# get the logits for \"not hate\" - this is the reward!\n",
    "not_hate_index = 0\n",
    "nothate_reward = (logits[:, not_hate_index]).tolist()\n",
    "print(f'reward (high): {nothate_reward}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63f729c5-98c3-4745-96e8-3484670215db",
   "metadata": {},
   "source": [
    "Let's show a toxic comment.  This will have a low reward because it is more toxic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0ffc81e4-b220-4f4e-95ec-98ac418612d1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logits [not hate, hate]: [-0.6921188831329346, 0.3722729980945587]\n",
      "probabilities [not hate, hate]: [0.25647106766700745, 0.7435289621353149]\n",
      "reward (low): [-0.6921188831329346]\n"
     ]
    }
   ],
   "source": [
    "toxic_text = \"#Person 1# tells Tommy that the movie was terrible, dumb and stupid.\"\n",
    "\n",
    "toxicity_input_ids = toxicity_tokenizer(toxic_text, return_tensors=\"pt\").input_ids\n",
    "\n",
    "logits = toxicity_model(toxicity_input_ids).logits\n",
    "print(f'logits [not hate, hate]: {logits.tolist()[0]}')\n",
    "\n",
    "# Print the probabilities for [not hate, hate]\n",
    "probabilities = logits.softmax(dim=-1).tolist()[0]\n",
    "print(f'probabilities [not hate, hate]: {probabilities}')\n",
    "\n",
    "# Get the logits for \"not hate\" - this is the reward!\n",
    "nothate_reward = (logits[:, not_hate_index]).tolist() \n",
    "print(f'reward (low): {nothate_reward}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc6e656e-fefe-4623-8bbb-472a8cf1c3c5",
   "metadata": {},
   "source": [
    "Setup Hugging Face inference pipeline to simplify the code for the toxicity reward model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "73aab8c6-33eb-4bf4-a816-3866fc3460af",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reward model output:\n",
      "For non-toxic text\n",
      "[{'label': 'nothate', 'score': 3.114100694656372}, {'label': 'hate', 'score': -2.4896175861358643}]\n",
      "[{'label': 'nothate', 'score': 0.9963293671607971}, {'label': 'hate', 'score': 0.003670616541057825}]\n",
      "For toxic text\n",
      "[{'label': 'hate', 'score': 0.3722729980945587}, {'label': 'nothate', 'score': -0.6921188831329346}]\n",
      "[{'label': 'hate', 'score': 0.7435289621353149}, {'label': 'nothate', 'score': 0.25647106766700745}]\n"
     ]
    }
   ],
   "source": [
    "device = 0 if torch.cuda.is_available() else \"cpu\"\n",
    "\n",
    "sentiment_pipe = pipeline(\"sentiment-analysis\", \n",
    "                          model=toxicity_model_name, \n",
    "                          device=device)\n",
    "reward_logits_kwargs = {\n",
    "    \"top_k\": None, # Return all scores.\n",
    "    \"function_to_apply\": \"none\", # Set to \"none\" to retrieve raw logits.\n",
    "    \"batch_size\": 16\n",
    "}\n",
    "\n",
    "reward_probabilities_kwargs = {\n",
    "    \"top_k\": None, # Return all scores.\n",
    "    \"function_to_apply\": \"softmax\", # Set to \"softmax\" to apply softmax and retrieve probabilities.\n",
    "    \"batch_size\": 16\n",
    "}\n",
    "\n",
    "print(\"Reward model output:\")\n",
    "print(\"For non-toxic text\")\n",
    "print(sentiment_pipe(non_toxic_text, **reward_logits_kwargs))\n",
    "print(sentiment_pipe(non_toxic_text, **reward_probabilities_kwargs))\n",
    "print(\"For toxic text\")\n",
    "print(sentiment_pipe(toxic_text, **reward_logits_kwargs))\n",
    "print(sentiment_pipe(toxic_text, **reward_probabilities_kwargs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21302d74-59d8-451f-b287-e86245bf3324",
   "metadata": {},
   "source": [
    "The outputs are the logits for both `nothate` (positive) and `hate` (negative) classes. But PPO will be using logits only of the `nothate` class as the positive reward signal used to help detoxify the LLM outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "36ff3925-70c4-495c-ae26-68e2fc36296b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'label': 'nothate', 'score': 3.114100694656372}, {'label': 'hate', 'score': -2.4896175861358643}]\n",
      "[{'label': 'nothate', 'score': 0.9963293671607971}, {'label': 'hate', 'score': 0.003670616541057825}]\n"
     ]
    }
   ],
   "source": [
    "print(sentiment_pipe(non_toxic_text, **reward_logits_kwargs))\n",
    "print(sentiment_pipe(non_toxic_text, **reward_probabilities_kwargs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8d11618b-5887-489a-b390-2139e364987f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'label': 'hate', 'score': 0.3722729980945587}, {'label': 'nothate', 'score': -0.6921188831329346}]\n",
      "[{'label': 'hate', 'score': 0.7435289621353149}, {'label': 'nothate', 'score': 0.25647106766700745}]\n"
     ]
    }
   ],
   "source": [
    "print(sentiment_pipe(toxic_text, **reward_logits_kwargs))\n",
    "print(sentiment_pipe(toxic_text, **reward_probabilities_kwargs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56513033-9bb1-41d5-81e2-54d1249c5c89",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a name='2.3'></a>\n",
    "### 2.3 - Evaluate Toxicity\n",
    "\n",
    "To evaluate the model before and after fine-tuning/detoxification you need to set up the [toxicity evaluation metric](https://huggingface.co/spaces/evaluate-measurement/toxicity). The **toxicity score** is a decimal value between 0 and 1 where 1 is the highest toxicity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "7de8e99d-60ea-48a2-bdf5-817f80b48979",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c21e6f13b05b4b38a523e1a9a7ed8efc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading builder script:   0%|          | 0.00/6.08k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "toxicity_evaluator = evaluate.load(\"toxicity\", \n",
    "                                    toxicity_model_name,\n",
    "                                    module_type=\"measurement\",\n",
    "                                    toxic_label=\"hate\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "840fbc47-c5c2-469a-b5f2-6407e8f0bfde",
   "metadata": {
    "tags": []
   },
   "source": [
    "Try to calculate toxicity for the same sentences as in section [2.2](#2.2). It's no surprise that the toxicity scores are the probabilities of `hate` class returned directly from the reward model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "5298f91c-30d1-4d17-95a7-553952ac97b5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Toxicity score for non-toxic text:\n",
      "[0.003670616541057825]\n",
      "\n",
      "Toxicity score for toxic text:\n",
      "[0.7435289621353149]\n"
     ]
    }
   ],
   "source": [
    "toxicity_score = toxicity_evaluator.compute(predictions=[\n",
    "    non_toxic_text\n",
    "])\n",
    "\n",
    "print(\"Toxicity score for non-toxic text:\")\n",
    "print(toxicity_score[\"toxicity\"])\n",
    "\n",
    "toxicity_score = toxicity_evaluator.compute(predictions=[\n",
    "    toxic_text\n",
    "])\n",
    "\n",
    "print(\"\\nToxicity score for toxic text:\")\n",
    "print(toxicity_score[\"toxicity\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d3e835b-14b9-4646-b1c9-c975ef3ea944",
   "metadata": {
    "tags": []
   },
   "source": [
    "This evaluator can be used to compute the toxicity of the dialogues prepared in section [2.1](#2.1). You will need to pass the test dataset (`dataset[\"test\"]`), the same tokenizer which was used in that section, the frozen PEFT model prepared in section [2.2](#2.2), and the toxicity evaluator. It is convenient to wrap the required steps in the function `evaluate_toxicity`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "316ab128-33ff-4a1e-8936-47bfa29d48a3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def evaluate_toxicity(model, \n",
    "                      toxicity_evaluator, \n",
    "                      tokenizer, \n",
    "                      dataset, \n",
    "                      num_samples):\n",
    "    \n",
    "    \"\"\"\n",
    "    Preprocess the dataset and split it into train and test parts.\n",
    "\n",
    "    Parameters:\n",
    "    - model (trl model): Model to be evaluated.\n",
    "    - toxicity_evaluator (evaluate_modules toxicity metrics): Toxicity evaluator.\n",
    "    - tokenizer (transformers tokenizer): Tokenizer to be used.\n",
    "    - dataset (dataset): Input dataset for the evaluation.\n",
    "    - num_samples (int): Maximum number of samples for the evaluation.\n",
    "        \n",
    "    Returns:\n",
    "    tuple: A tuple containing two numpy.float64 values:\n",
    "    - mean (numpy.float64): Mean of the samples toxicity.\n",
    "    - std (numpy.float64): Standard deviation of the samples toxicity.\n",
    "    \"\"\"\n",
    "\n",
    "    max_new_tokens=100\n",
    "\n",
    "    toxicities = []\n",
    "    input_texts = []\n",
    "    for i, sample in tqdm(enumerate(dataset)):\n",
    "        input_text = sample[\"query\"]\n",
    "\n",
    "        if i > num_samples:\n",
    "            break\n",
    "            \n",
    "        input_ids = tokenizer(input_text, return_tensors=\"pt\", padding=True).input_ids\n",
    "        \n",
    "        generation_config = GenerationConfig(max_new_tokens=max_new_tokens,\n",
    "                                             top_k=0.0,\n",
    "                                             top_p=1.0,\n",
    "                                             do_sample=True)\n",
    "\n",
    "        response_token_ids = model.generate(input_ids=input_ids,\n",
    "                                            generation_config=generation_config)\n",
    "        \n",
    "        generated_text = tokenizer.decode(response_token_ids[0], skip_special_tokens=True)\n",
    "        \n",
    "        toxicity_score = toxicity_evaluator.compute(predictions=[(input_text + \" \" + generated_text)])\n",
    "\n",
    "        toxicities.extend(toxicity_score[\"toxicity\"])\n",
    "\n",
    "    # Compute mean & std using np.\n",
    "    mean = np.mean(toxicities)\n",
    "    std = np.std(toxicities)\n",
    "        \n",
    "    return mean, std"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aed269c3-dbd7-4d45-bc44-c6ab6d4ae141",
   "metadata": {
    "tags": []
   },
   "source": [
    "And now perform the calculation of the model toxicity before fine-tuning/detoxification:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "c11ede15-dc1a-4a7e-a60d-b9cadfc7d876",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "11it [00:21,  1.92s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "toxicity [mean, std] before detox: [0.028631932472556153, 0.035603544139609644]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(model_name, device_map=\"auto\")\n",
    "\n",
    "mean_before_detoxification, std_before_detoxification = evaluate_toxicity(model=ref_model, \n",
    "                                                                          toxicity_evaluator=toxicity_evaluator, \n",
    "                                                                          tokenizer=tokenizer, \n",
    "                                                                          dataset=dataset[\"test\"], \n",
    "                                                                          num_samples=10)\n",
    "\n",
    "print(f'toxicity [mean, std] before detox: [{mean_before_detoxification}, {std_before_detoxification}]')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ba81c90-1ac8-4403-ac1a-d4c75c6df4f0",
   "metadata": {},
   "source": [
    "<a name='3'></a>\n",
    "## 3 - Perform Fine-Tuning to Detoxify the Summaries\n",
    "Optimize a RL policy against the reward model using Proximal Policy Optimization (PPO)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5516e318-8fce-4ca7-bf19-b7baf5255480",
   "metadata": {},
   "source": [
    "<a name='3.1'></a>\n",
    "### 3.1 - Initialize `PPOTrainer`\n",
    " \n",
    "For the `PPOTrainer` initialization, you will need a collator. Here it will be a function transforming the dictionaries in a particular way. You can define and test it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "8b7be1c0-382a-4fe2-8174-470f3e333e84",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collator input: [{'key1': 'value1', 'key2': 'value2', 'key3': 'value3'}]\n",
      "Collator output: {'key1': ['value1'], 'key2': ['value2'], 'key3': ['value3']}\n"
     ]
    }
   ],
   "source": [
    "def collator(data):\n",
    "    return dict((key, [d[key] for d in data]) for key in data[0])\n",
    "\n",
    "test_data = [{\"key1\": \"value1\", \"key2\": \"value2\", \"key3\": \"value3\"}]\n",
    "print(f'Collator input: {test_data}')\n",
    "print(f'Collator output: {collator(test_data)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "080c2e92-4988-4944-8353-0e1bb2048072",
   "metadata": {},
   "source": [
    "Set up the configuration parameters. Load the `ppo_model` and the tokenizer. You will also load a frozen version of the model `ref_model`. The first model is optimized while the second model serves as a reference to calculate the KL-divergence from the starting point. This works as an additional reward signal in the PPO training to make sure the optimized model does not deviate too much from the original LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "494e09a1-9024-4f38-91eb-d73cdc3239e6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Detected kernel version 4.14.336, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n"
     ]
    }
   ],
   "source": [
    "learning_rate=1.41e-5\n",
    "max_ppo_epochs=1\n",
    "mini_batch_size=4\n",
    "batch_size=16\n",
    "\n",
    "config = PPOConfig(\n",
    "    model_name=model_name,    \n",
    "    learning_rate=learning_rate,\n",
    "    ppo_epochs=max_ppo_epochs,\n",
    "    mini_batch_size=mini_batch_size,\n",
    "    batch_size=batch_size\n",
    ")\n",
    "\n",
    "ppo_trainer = PPOTrainer(config=config, \n",
    "                         model=ppo_model, \n",
    "                         ref_model=ref_model, \n",
    "                         tokenizer=tokenizer, \n",
    "                         dataset=dataset[\"train\"], \n",
    "                         data_collator=collator)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ad77d2c-3800-4e15-bb38-3851d94ad374",
   "metadata": {},
   "source": [
    "<a name='3.2'></a>\n",
    "### 3.2 - Fine-Tune the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cac21fb-fea5-4e80-a741-87f35ae72c62",
   "metadata": {},
   "source": [
    "The fine-tuning loop consists of the following main steps:\n",
    "1. Get the query responses from the policy LLM (PEFT model).\n",
    "2. Get sentiments for query/responses from hate speech RoBERTa model.\n",
    "3. Optimize policy with PPO using the (query, response, reward) triplet.\n",
    "\n",
    "The operation is running if you see the following metrics appearing:\n",
    "* `objective/kl`: minimize kl divergence,\n",
    "* `ppo/returns/mean`: maximize mean returns,\n",
    "* `ppo/policy/advantages_mean`: maximize advantages."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01536b7e-2f0f-4986-a97c-6ecfabf518d4",
   "metadata": {
    "tags": []
   },
   "source": [
    "<img src=\"data:image/svg+xml;base64,Cjxzdmcgd2lkdGg9IjgwMCIgaGVpZ2h0PSIxMjUiIHZpZXdCb3g9IjAgMCA4MDAgMTI1IiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPgogICAgPGRlZnM+CiAgICAgICAgPGxpbmVhckdyYWRpZW50IGlkPSJmYWRlR3JhZGllbnQiIHgxPSIwIiB4Mj0iMSI+CiAgICAgICAgICAgIDxzdG9wIG9mZnNldD0iMCUiIHN0b3AtY29sb3I9IiNGMEYwRjAiLz4KICAgICAgICAgICAgPHN0b3Agb2Zmc2V0PSIxMDAlIiBzdG9wLWNvbG9yPSIjRjBGMEYwIiBzdG9wLW9wYWNpdHk9IjAiLz4KICAgICAgICA8L2xpbmVhckdyYWRpZW50PgogICAgICAgIDxtYXNrIGlkPSJmYWRlTWFzayI+CiAgICAgICAgICAgIDxyZWN0IHg9IjAiIHk9IjAiIHdpZHRoPSI3NTAiIGhlaWdodD0iMTI1IiBmaWxsPSJ3aGl0ZSIvPgogICAgICAgICAgICA8cmVjdCB4PSI3NTAiIHk9IjAiIHdpZHRoPSI1MCIgaGVpZ2h0PSIxMjUiIGZpbGw9InVybCgjZmFkZUdyYWRpZW50KSIvPgogICAgICAgIDwvbWFzaz4KICAgIDwvZGVmcz4KICAgIDxwYXRoIGQ9Ik0zLDUwIEE1MCw1MCAwIDAgMSA1MywzIEw3OTcsMyBMNzk3LDk3IEw5Nyw5NyBMNTAsMTE1IEwzLDk3IFoiIGZpbGw9IiNGMEYwRjAiIHN0cm9rZT0iI0UwRTBFMCIgc3Ryb2tlLXdpZHRoPSIxIiBtYXNrPSJ1cmwoI2ZhZGVNYXNrKSIvPgogICAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgcj0iMzAiIGZpbGw9IiM1N2M0ZjgiIHN0cm9rZT0iIzU3YzRmOCIgc3Ryb2tlLXdpZHRoPSIxIi8+CiAgICA8Y2lyY2xlIGN4PSI1MCIgY3k9IjUwIiByPSIyNSIgZmlsbD0iI0YwRjBGMCIvPgogICAgPGxpbmUgeDE9IjUwIiB5MT0iNTAiIHgyPSI1MCIgeTI9IjMwIiBzdHJva2U9IiM1N2M0ZjgiIHN0cm9rZS13aWR0aD0iMyIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIi8+CiAgICA8bGluZSB4MT0iNTAiIHkxPSI1MCIgeDI9IjY1IiB5Mj0iNTAiIHN0cm9rZT0iIzU3YzRmOCIgc3Ryb2tlLXdpZHRoPSIzIiBzdHJva2UtbGluZWNhcD0icm91bmQiLz4KICAgIDx0ZXh0IHg9IjEwMCIgeT0iMzQiIGZvbnQtZmFtaWx5PSJBcmlhbCwgc2Fucy1zZXJpZiIgZm9udC1zaXplPSIxNCIgZmlsbD0iIzMzMzMzMyI+VGhlIG5leHQgY2VsbCBtYXkgdGFrZSAyMC0zMCBtaW51dGVzIHRvIHJ1bi48L3RleHQ+Cjwvc3ZnPgo=\" alt=\"Time alert open medium\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "4bc55397-92b8-4f61-9ec2-c8b39d5f8962",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "0it [00:00, ?it/s]You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n",
      "1it [01:37, 97.47s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "objective/kl: 29.314075469970703\n",
      "ppo/returns/mean: -0.5528708100318909\n",
      "ppo/policy/advantages_mean: -1.527217996510899e-08\n",
      "---------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2it [03:16, 98.53s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "objective/kl: 37.429527282714844\n",
      "ppo/returns/mean: -0.9722349643707275\n",
      "ppo/policy/advantages_mean: 1.5726914215363763e-09\n",
      "---------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "3it [04:49, 96.04s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "objective/kl: 35.188541412353516\n",
      "ppo/returns/mean: -0.8646271228790283\n",
      "ppo/policy/advantages_mean: -1.764723656094702e-08\n",
      "---------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "4it [06:14, 91.45s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "objective/kl: 24.190258026123047\n",
      "ppo/returns/mean: -0.2904372215270996\n",
      "ppo/policy/advantages_mean: 2.320304659519934e-08\n",
      "---------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "5it [07:39, 89.06s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "objective/kl: 28.547372817993164\n",
      "ppo/returns/mean: -0.47029852867126465\n",
      "ppo/policy/advantages_mean: 4.8037867017569624e-09\n",
      "---------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "6it [09:26, 95.16s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "objective/kl: 34.75200653076172\n",
      "ppo/returns/mean: -0.6979813575744629\n",
      "ppo/policy/advantages_mean: -7.28189952781122e-10\n",
      "---------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "7it [10:58, 94.25s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "objective/kl: 26.102764129638672\n",
      "ppo/returns/mean: -0.3362278342247009\n",
      "ppo/policy/advantages_mean: 8.958872754760705e-09\n",
      "---------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "8it [12:36, 95.60s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "objective/kl: 33.49162673950195\n",
      "ppo/returns/mean: -0.5960363745689392\n",
      "ppo/policy/advantages_mean: -1.4595116226701066e-08\n",
      "---------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "9it [14:13, 96.04s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "objective/kl: 37.33536911010742\n",
      "ppo/returns/mean: -0.9317035675048828\n",
      "ppo/policy/advantages_mean: 9.248212862189575e-09\n",
      "---------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "10it [15:45, 94.58s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "objective/kl: 24.834571838378906\n",
      "ppo/returns/mean: -0.3003133237361908\n",
      "ppo/policy/advantages_mean: 2.68654409829594e-09\n",
      "---------------------------------------------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "output_min_length = 100\n",
    "output_max_length = 400\n",
    "output_length_sampler = LengthSampler(output_min_length, output_max_length)\n",
    "\n",
    "generation_kwargs = {\n",
    "    \"min_length\": 5,\n",
    "    \"top_k\": 0.0,\n",
    "    \"top_p\": 1.0,\n",
    "    \"do_sample\": True\n",
    "}\n",
    "\n",
    "reward_kwargs = {\n",
    "    \"top_k\": None, # Return all scores.\n",
    "    \"function_to_apply\": \"none\", # You want the raw logits without softmax.\n",
    "    \"batch_size\": 16\n",
    "}\n",
    "\n",
    "max_ppo_steps = 10\n",
    "\n",
    "for step, batch in tqdm(enumerate(ppo_trainer.dataloader)):\n",
    "    # Break when you reach max_steps.\n",
    "    if step >= max_ppo_steps:\n",
    "        break   \n",
    "\n",
    "    prompt_tensors = batch[\"input_ids\"]\n",
    "\n",
    "    # Get response from FLAN-T5/PEFT LLM.\n",
    "    summary_tensors = []\n",
    "\n",
    "    for prompt_tensor in prompt_tensors:\n",
    "        max_new_tokens = output_length_sampler()        \n",
    "            \n",
    "        generation_kwargs[\"max_new_tokens\"] = max_new_tokens\n",
    "        summary = ppo_trainer.generate(prompt_tensor, **generation_kwargs)\n",
    "        \n",
    "        summary_tensors.append(summary.squeeze()[-max_new_tokens:])\n",
    "        \n",
    "    # This needs to be called \"response\".\n",
    "    batch[\"response\"] = [tokenizer.decode(r.squeeze()) for r in summary_tensors]\n",
    "\n",
    "    # Compute reward outputs.\n",
    "    query_response_pairs = [q + r for q, r in zip(batch[\"query\"], batch[\"response\"])]    \n",
    "    rewards = sentiment_pipe(query_response_pairs, **reward_kwargs)\n",
    "\n",
    "    # You use the `nothate` item because this is the score for the positive `nothate` class.\n",
    "    reward_tensors = [torch.tensor(reward[not_hate_index][\"score\"]) for reward in rewards]    \n",
    "\n",
    "    # Run PPO step.\n",
    "    stats = ppo_trainer.step(prompt_tensors, summary_tensors, reward_tensors)\n",
    "    ppo_trainer.log_stats(stats, batch, reward_tensors)\n",
    "    \n",
    "    print(f'objective/kl: {stats[\"objective/kl\"]}')\n",
    "    print(f'ppo/returns/mean: {stats[\"ppo/returns/mean\"]}')\n",
    "    print(f'ppo/policy/advantages_mean: {stats[\"ppo/policy/advantages_mean\"]}')\n",
    "    print('-'.join('' for x in range(100)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b648cb7-89e2-40b8-9507-9c07bdfd9ebf",
   "metadata": {},
   "source": [
    "<img src=\"data:image/svg+xml;base64,Cjxzdmcgd2lkdGg9IjgwMCIgaGVpZ2h0PSI1MCIgdmlld0JveD0iMCAwIDgwMCA1MCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KICAgIDxkZWZzPgogICAgICAgIDxsaW5lYXJHcmFkaWVudCBpZD0iZmFkZUdyYWRpZW50IiB4MT0iMCIgeDI9IjEiPgogICAgICAgICAgICA8c3RvcCBvZmZzZXQ9IjAlIiBzdG9wLWNvbG9yPSIjRjBGMEYwIi8+CiAgICAgICAgICAgIDxzdG9wIG9mZnNldD0iMTAwJSIgc3RvcC1jb2xvcj0iI0YwRjBGMCIgc3RvcC1vcGFjaXR5PSIwIi8+CiAgICAgICAgPC9saW5lYXJHcmFkaWVudD4KICAgICAgICA8bWFzayBpZD0iZmFkZU1hc2siPgogICAgICAgICAgICA8cmVjdCB4PSIwIiB5PSIwIiB3aWR0aD0iNzUwIiBoZWlnaHQ9IjUwIiBmaWxsPSJ3aGl0ZSIvPgogICAgICAgICAgICA8cmVjdCB4PSI3NTAiIHk9IjAiIHdpZHRoPSI1MCIgaGVpZ2h0PSI1MCIgZmlsbD0idXJsKCNmYWRlR3JhZGllbnQpIi8+CiAgICAgICAgPC9tYXNrPgogICAgPC9kZWZzPgogICAgPHBhdGggZD0iTTI1LDUwIFEwLDUwIDAsMjUgTDUwLDMgTDk3LDI1IEw3OTcsMjUgTDc5Nyw1MCBMMjUsNTAgWiIgZmlsbD0iI0YwRjBGMCIgc3Ryb2tlPSIjRTBFMEUwIiBzdHJva2Utd2lkdGg9IjEiIG1hc2s9InVybCgjZmFkZU1hc2spIi8+Cjwvc3ZnPgo=\" alt=\"Time alert close\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7903f5df-a9de-41eb-b239-38bc367b5654",
   "metadata": {},
   "source": [
    "<a name='3.3'></a>\n",
    "### 3.3 - Evaluate the Model Quantitatively\n",
    "\n",
    "Load the PPO/PEFT model back in from disk and use the test dataset split to evaluate the toxicity score of the RL-fine-tuned model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "3b093d43-6197-4cc0-b933-29030479a7d0",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "11it [00:23,  2.12s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "toxicity [mean, std] after detox: [0.03254406085364859, 0.037954809800451905]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "mean_after_detoxification, std_after_detoxification = evaluate_toxicity(model=ppo_model, \n",
    "                                                                        toxicity_evaluator=toxicity_evaluator, \n",
    "                                                                        tokenizer=tokenizer, \n",
    "                                                                        dataset=dataset[\"test\"], \n",
    "                                                                        num_samples=10)\n",
    "print(f'toxicity [mean, std] after detox: [{mean_after_detoxification}, {std_after_detoxification}]')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f42895cc-7bbf-45e1-a7c7-78ee29cd8009",
   "metadata": {
    "tags": []
   },
   "source": [
    "And compare the toxicity scores of the reference model (before detoxification) and fine-tuned model (after detoxification)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "77cc3af2-6600-4673-874b-917c05247ae3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage improvement of toxicity score after detoxification:\n",
      "mean: -13.66%\n",
      "std: -6.60%\n"
     ]
    }
   ],
   "source": [
    "mean_improvement = (mean_before_detoxification - mean_after_detoxification) / mean_before_detoxification\n",
    "std_improvement = (std_before_detoxification - std_after_detoxification) / std_before_detoxification\n",
    "\n",
    "print(f'Percentage improvement of toxicity score after detoxification:')\n",
    "print(f'mean: {mean_improvement*100:.2f}%')\n",
    "print(f'std: {std_improvement*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66030581-b6f7-41d7-a7e6-2466226833be",
   "metadata": {},
   "source": [
    "<a name='3.4'></a>\n",
    "### 3.4 - Evaluate the Model Qualitatively\n",
    "\n",
    "Let's inspect some examples from the test dataset. You can compare the original `ref_model` to the fine-tuned/detoxified `ppo_model` using the toxicity evaluator."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12fdc491-5437-41dd-980b-0d04304292dd",
   "metadata": {},
   "source": [
    "<img src=\"data:image/svg+xml;base64,Cjxzdmcgd2lkdGg9IjgwMCIgaGVpZ2h0PSIxMjUiIHZpZXdCb3g9IjAgMCA4MDAgMTI1IiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPgogICAgPGRlZnM+CiAgICAgICAgPGxpbmVhckdyYWRpZW50IGlkPSJmYWRlR3JhZGllbnQiIHgxPSIwIiB4Mj0iMSI+CiAgICAgICAgICAgIDxzdG9wIG9mZnNldD0iMCUiIHN0b3AtY29sb3I9IiNGMEYwRjAiLz4KICAgICAgICAgICAgPHN0b3Agb2Zmc2V0PSIxMDAlIiBzdG9wLWNvbG9yPSIjRjBGMEYwIiBzdG9wLW9wYWNpdHk9IjAiLz4KICAgICAgICA8L2xpbmVhckdyYWRpZW50PgogICAgICAgIDxtYXNrIGlkPSJmYWRlTWFzayI+CiAgICAgICAgICAgIDxyZWN0IHg9IjAiIHk9IjAiIHdpZHRoPSI3NTAiIGhlaWdodD0iMTI1IiBmaWxsPSJ3aGl0ZSIvPgogICAgICAgICAgICA8cmVjdCB4PSI3NTAiIHk9IjAiIHdpZHRoPSI1MCIgaGVpZ2h0PSIxMjUiIGZpbGw9InVybCgjZmFkZUdyYWRpZW50KSIvPgogICAgICAgIDwvbWFzaz4KICAgIDwvZGVmcz4KICAgIDxwYXRoIGQ9Ik0zLDUwIEE1MCw1MCAwIDAgMSA1MywzIEw3OTcsMyBMNzk3LDk3IEw5Nyw5NyBMNTAsMTE1IEwzLDk3IFoiIGZpbGw9IiNGMEYwRjAiIHN0cm9rZT0iI0UwRTBFMCIgc3Ryb2tlLXdpZHRoPSIxIiBtYXNrPSJ1cmwoI2ZhZGVNYXNrKSIvPgogICAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgcj0iMzAiIGZpbGw9IiM1N2M0ZjgiIHN0cm9rZT0iIzU3YzRmOCIgc3Ryb2tlLXdpZHRoPSIxIi8+CiAgICA8Y2lyY2xlIGN4PSI1MCIgY3k9IjUwIiByPSIyNSIgZmlsbD0iI0YwRjBGMCIvPgogICAgPGxpbmUgeDE9IjUwIiB5MT0iNTAiIHgyPSI1MCIgeTI9IjMwIiBzdHJva2U9IiM1N2M0ZjgiIHN0cm9rZS13aWR0aD0iMyIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIi8+CiAgICA8bGluZSB4MT0iNTAiIHkxPSI1MCIgeDI9IjY1IiB5Mj0iNTAiIHN0cm9rZT0iIzU3YzRmOCIgc3Ryb2tlLXdpZHRoPSIzIiBzdHJva2UtbGluZWNhcD0icm91bmQiLz4KICAgIDx0ZXh0IHg9IjEwMCIgeT0iMzQiIGZvbnQtZmFtaWx5PSJBcmlhbCwgc2Fucy1zZXJpZiIgZm9udC1zaXplPSIxNCIgZmlsbD0iIzMzMzMzMyI+VGhlIG5leHQgY2VsbCBtYXkgdGFrZSAyLTMgbWludXRlcyB0byBydW4uPC90ZXh0Pgo8L3N2Zz4K\" alt=\"Time alert open medium\"/>\n",
    "​"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "22cc8313-20ae-4d32-855e-9b2866fa3085",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 20/20 [01:16<00:00,  3.81s/it]\n"
     ]
    }
   ],
   "source": [
    "batch_size = 20\n",
    "compare_results = {}\n",
    "\n",
    "df_batch = dataset[\"test\"][0:batch_size]\n",
    "\n",
    "compare_results[\"query\"] = df_batch[\"query\"]\n",
    "prompt_tensors = df_batch[\"input_ids\"]\n",
    "\n",
    "summary_tensors_ref = []\n",
    "summary_tensors = []\n",
    "\n",
    "# Get response from ppo and base model.\n",
    "for i in tqdm(range(batch_size)):\n",
    "    gen_len = output_length_sampler()\n",
    "    generation_kwargs[\"max_new_tokens\"] = gen_len\n",
    "    \n",
    "    summary = ref_model.generate(\n",
    "        input_ids=torch.as_tensor(prompt_tensors[i]).unsqueeze(dim=0).to(device), \n",
    "        **generation_kwargs\n",
    "    ).squeeze()[-gen_len:]\n",
    "    summary_tensors_ref.append(summary)\n",
    "\n",
    "    summary = ppo_model.generate(\n",
    "        input_ids=torch.as_tensor(prompt_tensors[i]).unsqueeze(dim=0).to(device), \n",
    "        **generation_kwargs\n",
    "    ).squeeze()[-gen_len:]\n",
    "    summary_tensors.append(summary)\n",
    "\n",
    "# Decode responses.\n",
    "compare_results[\"response_before\"] = [tokenizer.decode(summary_tensors_ref[i]) for i in range(batch_size)]\n",
    "compare_results[\"response_after\"] = [tokenizer.decode(summary_tensors[i]) for i in range(batch_size)]\n",
    "\n",
    "# Sentiment analysis of query/response pairs before/after.\n",
    "texts_before = [d + s for d, s in zip(compare_results[\"query\"], compare_results[\"response_before\"])]\n",
    "rewards_before = sentiment_pipe(texts_before, **reward_kwargs)\n",
    "compare_results[\"reward_before\"] = [reward[not_hate_index][\"score\"] for reward in rewards_before]\n",
    "\n",
    "texts_after = [d + s for d, s in zip(compare_results[\"query\"], compare_results[\"response_after\"])]\n",
    "rewards_after = sentiment_pipe(texts_after, **reward_kwargs)\n",
    "compare_results[\"reward_after\"] = [reward[not_hate_index][\"score\"] for reward in rewards_after]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13a3853f-be22-4a95-95d8-a4b61eb2468f",
   "metadata": {
    "tags": []
   },
   "source": [
    "<img src=\"data:image/svg+xml;base64,Cjxzdmcgd2lkdGg9IjgwMCIgaGVpZ2h0PSI1MCIgdmlld0JveD0iMCAwIDgwMCA1MCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KICAgIDxkZWZzPgogICAgICAgIDxsaW5lYXJHcmFkaWVudCBpZD0iZmFkZUdyYWRpZW50IiB4MT0iMCIgeDI9IjEiPgogICAgICAgICAgICA8c3RvcCBvZmZzZXQ9IjAlIiBzdG9wLWNvbG9yPSIjRjBGMEYwIi8+CiAgICAgICAgICAgIDxzdG9wIG9mZnNldD0iMTAwJSIgc3RvcC1jb2xvcj0iI0YwRjBGMCIgc3RvcC1vcGFjaXR5PSIwIi8+CiAgICAgICAgPC9saW5lYXJHcmFkaWVudD4KICAgICAgICA8bWFzayBpZD0iZmFkZU1hc2siPgogICAgICAgICAgICA8cmVjdCB4PSIwIiB5PSIwIiB3aWR0aD0iNzUwIiBoZWlnaHQ9IjUwIiBmaWxsPSJ3aGl0ZSIvPgogICAgICAgICAgICA8cmVjdCB4PSI3NTAiIHk9IjAiIHdpZHRoPSI1MCIgaGVpZ2h0PSI1MCIgZmlsbD0idXJsKCNmYWRlR3JhZGllbnQpIi8+CiAgICAgICAgPC9tYXNrPgogICAgPC9kZWZzPgogICAgPHBhdGggZD0iTTI1LDUwIFEwLDUwIDAsMjUgTDUwLDMgTDk3LDI1IEw3OTcsMjUgTDc5Nyw1MCBMMjUsNTAgWiIgZmlsbD0iI0YwRjBGMCIgc3Ryb2tlPSIjRTBFMEUwIiBzdHJva2Utd2lkdGg9IjEiIG1hc2s9InVybCgjZmFkZU1hc2spIi8+Cjwvc3ZnPgo=\" alt=\"Time alert close\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65b70892-4c22-4bed-9d1e-9da3f4f0c97f",
   "metadata": {
    "tags": []
   },
   "source": [
    "Store and review the results in a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "e06fff0f-9dba-4517-9424-a5ebd81e8f49",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>response_before</th>\n",
       "      <th>response_after</th>\n",
       "      <th>reward_before</th>\n",
       "      <th>reward_after</th>\n",
       "      <th>reward_diff</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Could you help me, Sir? My flight got in 15 minutes ago. Everyone else has picked up the luggage but mine hasn't come through. #Person2#: I'm sorry, Madam, I'll go and find out if there is any more to come. Summary: &lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1#'s flight hasn't been for 15 minutes. #Person2# works for #Person1#'s help in Find&lt;unk&gt;ng else to make sure they not have a flight.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1#'s flight had moved in 15 minutes after #Person1#'s request but #Person1#'s flight did not come by.&lt;/s&gt;</td>\n",
       "      <td>1.768565</td>\n",
       "      <td>2.395374</td>\n",
       "      <td>0.626809</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Summarize the following conversation. #Person1#: I'd like to have this cashed, please. #Person2#: Please put you name and address here. May I see your passport? #Person1#: Yes. #Person2#: How would you like it? #Person1#: Ten hundreds and ten twenties, and the rest in small change, please. #Person2#: OK. Here you are. Summary: &lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# is cashed $100 cash in big change. #Person2# requests from #Person1#.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# needs the cash in small change to cashed a bill. #Person2# please deposit this amount, and emailed to #Person1# exchange receipt number.&lt;/s&gt;</td>\n",
       "      <td>2.232070</td>\n",
       "      <td>2.663300</td>\n",
       "      <td>0.431230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Summarize the following conversation. #Person1#: So how did you like the restaurant? #Person2#: Actually, it could have been better. #Person1#: What didn't you like about it? #Person2#: It is a new restaurant. I don't think they have their act together yet. #Person1#: What did you think about the food? #Person2#: I felt that the food was pretty mediocre. #Person1#: The service wasn't that great, either. #Person2#: I agree. The service was not good. #Person1#: Do you think that you want to tr...</td>\n",
       "      <td>&lt;pad&gt; #Person2# tells #Person1# #Person2# liked the restaurant but #Person2# doesn't think the service was good. #Person2# approves but doesn't want to try again.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# asks [Person2# about the restaurant. It's new and it's not getting the same is customers' opinion as at previous times.&lt;/s&gt;</td>\n",
       "      <td>1.959794</td>\n",
       "      <td>2.355813</td>\n",
       "      <td>0.396019</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Summarize the following conversation. #Person1#: I would like to order some internet today. #Person2#: What kind would you like? #Person1#: What kind of internet is there? #Person2#: You can get DEL or dial-up. #Person1#: Which of those two is best? #Person2#: I would recommend DEL. #Person1#: So that one better? #Person2#: It's better because it doesn't tie up the phone. #Person1#: What do you mean by that? #Person2#: DEL isn't connected through your phone line, but dial-up is. #Person1#: S...</td>\n",
       "      <td>&lt;pad&gt; #Person1# wants to buy dial-up internet, but #Person2#'s sign says DEL or dial-up doesn't tie up the phone. Says that DEL doesn't tie up the phone and does both.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person2# recommends dial-up internet because it doesn't tie up the phone and does the work by calling.&lt;/s&gt;</td>\n",
       "      <td>2.429675</td>\n",
       "      <td>2.721628</td>\n",
       "      <td>0.291954</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Where shall I register, please? #Person2#: Here. Do you have a registration card? #Person1#: Yes. Here you are. #Person2#: Please register your information here and pay for it. And I'll make a medical record for you. #Person1#: OK. How much do I need to pay for the registration? #Person2#: Please pay ten yuan for the registration. #Person1#: Here is my money. #Person2#: This is your registration card. Please don't lose it and bring it whenever...</td>\n",
       "      <td>&lt;pad&gt; #Person1# asks #Person2# where to register her information and how to get to the consulting room to know who they are.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# is going to consult the consultant department. #Person2# reminds #Person1# to pay for the registration and give #Person1# documents and directions. Then #Person1# finds an easy way to the university center.&lt;/s&gt;</td>\n",
       "      <td>1.505595</td>\n",
       "      <td>1.728968</td>\n",
       "      <td>0.223373</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Summarize the following conversation. #Person1#: How much are you asking for this? #Person2#: I'm offering them to you at 150 yuan a piece. Is that all right? #Person1#: Is tax already included in their price? #Person2#: Yes. Our price can't be matched. #Person1#: Would you consider a volume discount? #Person2#: If you buy 1, 000 or more, you'll get a 10 % discount. #Person1#: I'll accept your offer. Summary: &lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# is buying earrings from Tosai Company as the inspiration for their earrings. #Person2# offers C colors with a 10 % discount and asks #Person1# to try them for splitting.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person2# sells some vegetables at 150 yuan. #Person1# supports it and helps #Person2# to increase their supply.&lt;/s&gt;</td>\n",
       "      <td>2.661622</td>\n",
       "      <td>2.800712</td>\n",
       "      <td>0.139090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Could you help me figure out how to look for a job? #Person2#: We have lots of options, what type of job do you need? #Person1#: I want to work in an office. #Person2#: Do you want to work part-time or full-time? #Person1#: I want to work full-time. #Person2#: We have binders with local job listings or you can make use of the computers. OK? #Person1#: I am confused a bit but I am sure that I can figure it out. #Person2#: If you make an appoint...</td>\n",
       "      <td>&lt;pad&gt; #Person1# tells #Person2# they have much choices and say they have binders with local job listings or computers. #Person2# advises #Person1# to make an appointment with a job counselor. #Person1# doesn't want to have a counselor.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# wants to work in an office full-time and asks #Person2# for help. #Person2# allows #Person1# to hire a adequate staff.&lt;/s&gt;</td>\n",
       "      <td>2.008402</td>\n",
       "      <td>2.129998</td>\n",
       "      <td>0.121596</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Let's take a coffee break, shall we? #Person2#: I wish I could, but I can't. #Person1#: What keeps you so busy? You've been sitting there for hours. You've got to walk around. You just can't stay on the computer forever. #Person2#: Well, I am up to my neck in work. I've got to finish this report. Sarah needs it by noon. I don't want to be scolded if I can't finish my work by the deadline. #Person1#: I understand that, but you'd feel better if ...</td>\n",
       "      <td>&lt;pad&gt; #Person2# only needs to take a caffeine break, because he has to finish a report. #Person2#'s already hungover and wants a coffee break.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# suggests #Person2# take a coffee break but #Person2# can't because of the work load.&lt;/s&gt;</td>\n",
       "      <td>1.779952</td>\n",
       "      <td>1.896657</td>\n",
       "      <td>0.116705</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Oh, my God! What's this? #Person2#: What? #Person1#: Look! This window is open. #Person2#: Did you open it before we left? #Person1#: Are you kidding? It's winter. Why would I open it? #Person2#: I don't know. Wait. Is this yours? #Person1#: No! Oh, my God! Someone has broken into the house. #Person2#: It looks that way. That's probably why the door wasn't locked when we came in. #Person1#: I locked it when I left though. #Person2#: Yes, but t...</td>\n",
       "      <td>&lt;pad&gt; Allen's home was attacked by the robber because the window was open. Allen suspects something was stolen, and #Person2# agrees.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; Allen remembers saying his window is open when he went to a house; on the other hand he locked the windows early because of the breaking in. He doesn't care about that because he'll look upstairs.&lt;/s&gt;</td>\n",
       "      <td>1.786834</td>\n",
       "      <td>1.875944</td>\n",
       "      <td>0.089109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Excuse me, could you tell me how to get to the Cross Bakery building? #Person2#: The Cross Bakery building? Oh sure. You're actually walking in the opposite direction. #Person1#: Oh, you're kidding! I thought I was heading east. #Person2#: No, east is the other direction. To get to the Bakery, you need to turn around and go three blocks to Broadway. When you get to the intersection of Broadway and Elm, you hang a left. Go straight down that st...</td>\n",
       "      <td>&lt;pad&gt; #Person1# explains the route to Cross Bakery. #Person1# learns #Person2#'s directions and gives her the way to cross the Bakery. #Person2# will show #Person1# the way.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# asks the Cross Bakery leading to the Cross Bakery. #Person2# recommends him the west direction. #Person1# tells #Person2# the way before #Person1# walks.&lt;/s&gt;</td>\n",
       "      <td>2.729239</td>\n",
       "      <td>2.749945</td>\n",
       "      <td>0.020705</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Amanda, how do you like this peaked cap? #Person2#: Didn't you say you want to buy a top hat? #Person1#: But I think this one fits me Well. Why don't you try on the sombrero in black? #Person2#: I don't like caps at all. Summary: &lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; Amanda shop for a peaked cap and explains to #Person1# her feelings on her favorite top hat.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; Amanda likes the peaked cap but #Person2# doesn't like caps. #Person1# doesn't like caps, so Amanda suggests a sombrero in black.&lt;/s&gt;</td>\n",
       "      <td>1.409167</td>\n",
       "      <td>1.425640</td>\n",
       "      <td>0.016474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Today more and more families have personal computers. People have wider range of choice to communicate with the outside world. #Person2#: Right. With the establishment of Internet and a lot of web companies, people are getting more and more dependent on the web. #Person1#: One of the common uses of PC is that people can buy goods through it without going out to the physical stores. #Person2#: Can you tell me how it is done? #Person1#: If a cus...</td>\n",
       "      <td>&lt;pad&gt; #Person1# explains how to do online orders and what tools are available when people have shops. The purpose of this is to help people buy goods.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# tells #Person2# about practical activities for the purpose of customizable computers and how customers can get goods through them.&lt;/s&gt;</td>\n",
       "      <td>2.766026</td>\n",
       "      <td>2.691949</td>\n",
       "      <td>-0.074076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Summarize the following conversation. #Person1#: I'm forming a music band. #Person2#: Do you already know how to play an instrument? #Person1#: Uh... Yeah! I'Ve told you a thousand times that I'm learning to play the drums. Now that I know how to play well, I would like to form a rock band. #Person2#: Aside from yourself, who are the other members of the band? #Person1#: We have a guy who plays guitar, and another who plays bass. Although we still haven't found anyone to be our singer. You t...</td>\n",
       "      <td>&lt;pad&gt; #Person1# wants to form a rock band and asks #Person2# about the musical talent of the other members of the band as well as the details of their songs. #Person1# begins learning to play the drums. #Person2# asks about #Person1#'s fantasy of music and talks to #Person1# about parts for the band.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# is forming a music band with Bob and Buddy and invites #Person1# to audition this weekend.&lt;/s&gt;</td>\n",
       "      <td>2.851743</td>\n",
       "      <td>2.741764</td>\n",
       "      <td>-0.109980</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Hello? #Person2#: Hello? #Person1#: Can I speak to Li Hong, please? #Person2#: Speaking. #Person1#: Hi, Li Hong. This is Alice. #Person2#: Hi, Alice. How are you? #Person1#: Not bad. Li Hong, I am sorry that I can't go to see Mrs. Brown with you tomorrow morning. My mother is ill. I must take care of her. #Person2#: I'm sorry to hear that. You'd better stay at home. After all, we can visit Mrs. Brown later #Person1#: OK. Bye - bye. #Person2#: ...</td>\n",
       "      <td>&lt;pad&gt; Alice is staying at home and can't go to see Mrs. Brown tomorrow morning because her mother is sick. Li Hong apologizes to Alice. Her family can visit Mrs. Brown later.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; Alice tells Li Hong she can't go to Mrs. Brown because her mother is ill. Alice decides to stay at home and says she won't see Mrs. Brown later because she'll be home later.&lt;/s&gt;</td>\n",
       "      <td>1.770844</td>\n",
       "      <td>1.636200</td>\n",
       "      <td>-0.134644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Mom, I just finished my paper. Can you proofread it before I hand it in? #Person2#: Sure, let's take a look. Sweetie, this is terrific. Your ideas are so original. #Person1#: Thanks. #Person2#: I can tell you worked hard on it. #Person1#: I really did! I started thinking about what I wanted to say three weeks ago. #Person2#: Well, it was definitely worth all the time. #Person1#: Let's just hope my teacher agrees. Summary: &lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# has just finished the paper he's written. Mom has proofread it and personally finishes it. #Person1#'s mom will be proud.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# asks #Person2# to check #Person1#'s paper and be sure to give #Person1# some right tips.&lt;/s&gt;</td>\n",
       "      <td>2.693838</td>\n",
       "      <td>2.512838</td>\n",
       "      <td>-0.181000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Summarize the following conversation. #Person1#: What can I do for you, madam? #Person2#: I'd like to buy a toy car for my son. #Person1#: How about this one? #Person2#: It looks nice. How much is it? #Person1#: They're three hundred dollars. #Person2#: Oh, I'm afraid it's too expensive. Can you show me something cheaper? #Person1#: OK, This one is one hundred and twenty. It's the cheapest here. #Person2#: OK, I'll take it. Here's the money. #Person1#: Thank you very much. Summary: &lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person2# wants to buy a toy car for son. #Person1# helps #Person2# find a cheaper one that costs 60 dollars.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; Mrs. Bauer offers her an affordable toy car that she can buy for her son because it's worth three hundred dollars.&lt;/s&gt;</td>\n",
       "      <td>1.400756</td>\n",
       "      <td>1.194819</td>\n",
       "      <td>-0.205937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Hello. I want to reconfirm our flight to London. #Person2#: Yes, sir. Did you call the airline? #Person1#: Yes, I did. But I couldn't communicate with them in English. They speak only Spanish. So I need your help. #Person2#: Certainly, sir. What is the flight number and when are you leaving? #Person1#: We are taking IB 385 to London tomorrow at 1 p. m. #Person2#: Oh, I see, sir. We have the airline office inside the hotel. They have an English...</td>\n",
       "      <td>&lt;pad&gt; #Person1# wants to confirm the flight, but #Person2# tells her the flight number and #Person1#'ll dial 35 when leaving.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# realizes she needs some help because she cannot connect with the airline in English. #Person2# invites her by phone, but not the Internet.&lt;/s&gt;</td>\n",
       "      <td>1.849587</td>\n",
       "      <td>1.447008</td>\n",
       "      <td>-0.402579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Here is the final draft of our contract. I'm glad that we have reached an agreement on almost every term in our trade. #Person2#: Yes, it seems to me we have come quite a long way. However, let me take a close look at the final draft. #Person1#: Do you have some points to bring up? #Person2#: Well, everything we've discussed seems to be here. #Person1#: Yes, including a description of the shirts you want to purchase this time, the total amount...</td>\n",
       "      <td>&lt;pad&gt; #Person1# reports 7 goodbye letters to #Person2# about the revised contract and the quality standards. #Person2# is happy to sign their contract.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person1# presents the final draft of the contract to #Person2# and questions a few points about the table. #Person2# reminds #Person1# about chipping the customer. Sign the contract immediately.&lt;/s&gt;</td>\n",
       "      <td>3.099102</td>\n",
       "      <td>2.632910</td>\n",
       "      <td>-0.466192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Summarize the following conversation. #Person1#: Judy, what is everybody talking about? #Person2#: Haven't you heard? Richard was fired by our manager. #Person1#: You're kidding. It can't be true. #Person2#: Believe it or not. Everybody is talking about it in the company. #Person1#: Really? I'm surprised. #Person2#: Me too. Summary: &lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; Judy and Judy are surprised about Richard being fired. Judy didn't trust him because she's surprised about it.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; Judy's wondering how everyone is talking about Richard being fired. Judy agrees with her.&lt;/s&gt;</td>\n",
       "      <td>1.315041</td>\n",
       "      <td>0.028662</td>\n",
       "      <td>-1.286379</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Summarize the following conversation. #Person1#: It smells like an ashtray in here! #Person2#: Hi honey! What's wrong? Why do you have that look on your face? #Person1#: What's wrong? I thought we agreed that you were gonna quit smoking. #Person2#: No! I said I was going to cut down which is very different. You can't just expect me to go cold turkey overnight! #Person1#: Look, there are other ways to quit. You can try the nicotine patch, or nicotine chewing gum. We spend a fortune on cigaret...</td>\n",
       "      <td>&lt;pad&gt; There's another smoking option to his health and the others could help. #Person1# doesn't think he is having the willpower to quit. #Person1# says they can't help because they spend a fortune on cigarettes every month.&lt;/s&gt;</td>\n",
       "      <td>&lt;pad&gt; #Person2# has a smell of smoke. Then she turns down and asks for a divorce.&lt;/s&gt;</td>\n",
       "      <td>1.959103</td>\n",
       "      <td>0.550751</td>\n",
       "      <td>-1.408353</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  query  \\\n",
       "0                                                                                                                                                                                                                                         Summarize the following conversation. #Person1#: Could you help me, Sir? My flight got in 15 minutes ago. Everyone else has picked up the luggage but mine hasn't come through. #Person2#: I'm sorry, Madam, I'll go and find out if there is any more to come. Summary: </s>   \n",
       "1                                                                                                                                                                         Summarize the following conversation. #Person1#: I'd like to have this cashed, please. #Person2#: Please put you name and address here. May I see your passport? #Person1#: Yes. #Person2#: How would you like it? #Person1#: Ten hundreds and ten twenties, and the rest in small change, please. #Person2#: OK. Here you are. Summary: </s>   \n",
       "2   Summarize the following conversation. #Person1#: So how did you like the restaurant? #Person2#: Actually, it could have been better. #Person1#: What didn't you like about it? #Person2#: It is a new restaurant. I don't think they have their act together yet. #Person1#: What did you think about the food? #Person2#: I felt that the food was pretty mediocre. #Person1#: The service wasn't that great, either. #Person2#: I agree. The service was not good. #Person1#: Do you think that you want to tr...   \n",
       "3   Summarize the following conversation. #Person1#: I would like to order some internet today. #Person2#: What kind would you like? #Person1#: What kind of internet is there? #Person2#: You can get DEL or dial-up. #Person1#: Which of those two is best? #Person2#: I would recommend DEL. #Person1#: So that one better? #Person2#: It's better because it doesn't tie up the phone. #Person1#: What do you mean by that? #Person2#: DEL isn't connected through your phone line, but dial-up is. #Person1#: S...   \n",
       "4   Summarize the following conversation. #Person1#: Where shall I register, please? #Person2#: Here. Do you have a registration card? #Person1#: Yes. Here you are. #Person2#: Please register your information here and pay for it. And I'll make a medical record for you. #Person1#: OK. How much do I need to pay for the registration? #Person2#: Please pay ten yuan for the registration. #Person1#: Here is my money. #Person2#: This is your registration card. Please don't lose it and bring it whenever...   \n",
       "5                                                                                     Summarize the following conversation. #Person1#: How much are you asking for this? #Person2#: I'm offering them to you at 150 yuan a piece. Is that all right? #Person1#: Is tax already included in their price? #Person2#: Yes. Our price can't be matched. #Person1#: Would you consider a volume discount? #Person2#: If you buy 1, 000 or more, you'll get a 10 % discount. #Person1#: I'll accept your offer. Summary: </s>   \n",
       "6   Summarize the following conversation. #Person1#: Could you help me figure out how to look for a job? #Person2#: We have lots of options, what type of job do you need? #Person1#: I want to work in an office. #Person2#: Do you want to work part-time or full-time? #Person1#: I want to work full-time. #Person2#: We have binders with local job listings or you can make use of the computers. OK? #Person1#: I am confused a bit but I am sure that I can figure it out. #Person2#: If you make an appoint...   \n",
       "7   Summarize the following conversation. #Person1#: Let's take a coffee break, shall we? #Person2#: I wish I could, but I can't. #Person1#: What keeps you so busy? You've been sitting there for hours. You've got to walk around. You just can't stay on the computer forever. #Person2#: Well, I am up to my neck in work. I've got to finish this report. Sarah needs it by noon. I don't want to be scolded if I can't finish my work by the deadline. #Person1#: I understand that, but you'd feel better if ...   \n",
       "8   Summarize the following conversation. #Person1#: Oh, my God! What's this? #Person2#: What? #Person1#: Look! This window is open. #Person2#: Did you open it before we left? #Person1#: Are you kidding? It's winter. Why would I open it? #Person2#: I don't know. Wait. Is this yours? #Person1#: No! Oh, my God! Someone has broken into the house. #Person2#: It looks that way. That's probably why the door wasn't locked when we came in. #Person1#: I locked it when I left though. #Person2#: Yes, but t...   \n",
       "9   Summarize the following conversation. #Person1#: Excuse me, could you tell me how to get to the Cross Bakery building? #Person2#: The Cross Bakery building? Oh sure. You're actually walking in the opposite direction. #Person1#: Oh, you're kidding! I thought I was heading east. #Person2#: No, east is the other direction. To get to the Bakery, you need to turn around and go three blocks to Broadway. When you get to the intersection of Broadway and Elm, you hang a left. Go straight down that st...   \n",
       "10                                                                                                                                                                                                                          Summarize the following conversation. #Person1#: Amanda, how do you like this peaked cap? #Person2#: Didn't you say you want to buy a top hat? #Person1#: But I think this one fits me Well. Why don't you try on the sombrero in black? #Person2#: I don't like caps at all. Summary: </s>   \n",
       "11  Summarize the following conversation. #Person1#: Today more and more families have personal computers. People have wider range of choice to communicate with the outside world. #Person2#: Right. With the establishment of Internet and a lot of web companies, people are getting more and more dependent on the web. #Person1#: One of the common uses of PC is that people can buy goods through it without going out to the physical stores. #Person2#: Can you tell me how it is done? #Person1#: If a cus...   \n",
       "12  Summarize the following conversation. #Person1#: I'm forming a music band. #Person2#: Do you already know how to play an instrument? #Person1#: Uh... Yeah! I'Ve told you a thousand times that I'm learning to play the drums. Now that I know how to play well, I would like to form a rock band. #Person2#: Aside from yourself, who are the other members of the band? #Person1#: We have a guy who plays guitar, and another who plays bass. Although we still haven't found anyone to be our singer. You t...   \n",
       "13  Summarize the following conversation. #Person1#: Hello? #Person2#: Hello? #Person1#: Can I speak to Li Hong, please? #Person2#: Speaking. #Person1#: Hi, Li Hong. This is Alice. #Person2#: Hi, Alice. How are you? #Person1#: Not bad. Li Hong, I am sorry that I can't go to see Mrs. Brown with you tomorrow morning. My mother is ill. I must take care of her. #Person2#: I'm sorry to hear that. You'd better stay at home. After all, we can visit Mrs. Brown later #Person1#: OK. Bye - bye. #Person2#: ...   \n",
       "14                      Summarize the following conversation. #Person1#: Mom, I just finished my paper. Can you proofread it before I hand it in? #Person2#: Sure, let's take a look. Sweetie, this is terrific. Your ideas are so original. #Person1#: Thanks. #Person2#: I can tell you worked hard on it. #Person1#: I really did! I started thinking about what I wanted to say three weeks ago. #Person2#: Well, it was definitely worth all the time. #Person1#: Let's just hope my teacher agrees. Summary: </s>   \n",
       "15          Summarize the following conversation. #Person1#: What can I do for you, madam? #Person2#: I'd like to buy a toy car for my son. #Person1#: How about this one? #Person2#: It looks nice. How much is it? #Person1#: They're three hundred dollars. #Person2#: Oh, I'm afraid it's too expensive. Can you show me something cheaper? #Person1#: OK, This one is one hundred and twenty. It's the cheapest here. #Person2#: OK, I'll take it. Here's the money. #Person1#: Thank you very much. Summary: </s>   \n",
       "16  Summarize the following conversation. #Person1#: Hello. I want to reconfirm our flight to London. #Person2#: Yes, sir. Did you call the airline? #Person1#: Yes, I did. But I couldn't communicate with them in English. They speak only Spanish. So I need your help. #Person2#: Certainly, sir. What is the flight number and when are you leaving? #Person1#: We are taking IB 385 to London tomorrow at 1 p. m. #Person2#: Oh, I see, sir. We have the airline office inside the hotel. They have an English...   \n",
       "17  Summarize the following conversation. #Person1#: Here is the final draft of our contract. I'm glad that we have reached an agreement on almost every term in our trade. #Person2#: Yes, it seems to me we have come quite a long way. However, let me take a close look at the final draft. #Person1#: Do you have some points to bring up? #Person2#: Well, everything we've discussed seems to be here. #Person1#: Yes, including a description of the shirts you want to purchase this time, the total amount...   \n",
       "18                                                                                                                                                                  Summarize the following conversation. #Person1#: Judy, what is everybody talking about? #Person2#: Haven't you heard? Richard was fired by our manager. #Person1#: You're kidding. It can't be true. #Person2#: Believe it or not. Everybody is talking about it in the company. #Person1#: Really? I'm surprised. #Person2#: Me too. Summary: </s>   \n",
       "19  Summarize the following conversation. #Person1#: It smells like an ashtray in here! #Person2#: Hi honey! What's wrong? Why do you have that look on your face? #Person1#: What's wrong? I thought we agreed that you were gonna quit smoking. #Person2#: No! I said I was going to cut down which is very different. You can't just expect me to go cold turkey overnight! #Person1#: Look, there are other ways to quit. You can try the nicotine patch, or nicotine chewing gum. We spend a fortune on cigaret...   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                      response_before  \\\n",
       "0                                                                                                                                                              <pad> #Person1#'s flight hasn't been for 15 minutes. #Person2# works for #Person1#'s help in Find<unk>ng else to make sure they not have a flight.</s>   \n",
       "1                                                                                                                                                                                                                           <pad> #Person1# is cashed $100 cash in big change. #Person2# requests from #Person1#.</s>   \n",
       "2                                                                                                                                              <pad> #Person2# tells #Person1# #Person2# liked the restaurant but #Person2# doesn't think the service was good. #Person2# approves but doesn't want to try again.</s>   \n",
       "3                                                                                                                                         <pad> #Person1# wants to buy dial-up internet, but #Person2#'s sign says DEL or dial-up doesn't tie up the phone. Says that DEL doesn't tie up the phone and does both.</s>   \n",
       "4                                                                                                                                                                                    <pad> #Person1# asks #Person2# where to register her information and how to get to the consulting room to know who they are.</s>   \n",
       "5                                                                                                                       <pad> #Person1# is buying earrings from Tosai Company as the inspiration for their earrings. #Person2# offers C colors with a 10 % discount and asks #Person1# to try them for splitting.</s>   \n",
       "6                                                                     <pad> #Person1# tells #Person2# they have much choices and say they have binders with local job listings or computers. #Person2# advises #Person1# to make an appointment with a job counselor. #Person1# doesn't want to have a counselor.</s>   \n",
       "7                                                                                                                                                                  <pad> #Person2# only needs to take a caffeine break, because he has to finish a report. #Person2#'s already hungover and wants a coffee break.</s>   \n",
       "8                                                                                                                                                                           <pad> Allen's home was attacked by the robber because the window was open. Allen suspects something was stolen, and #Person2# agrees.</s>   \n",
       "9                                                                                                                                   <pad> #Person1# explains the route to Cross Bakery. #Person1# learns #Person2#'s directions and gives her the way to cross the Bakery. #Person2# will show #Person1# the way.</s>   \n",
       "10                                                                                                                                                                                                             <pad> Amanda shop for a peaked cap and explains to #Person1# her feelings on her favorite top hat.</s>   \n",
       "11                                                                                                                                                         <pad> #Person1# explains how to do online orders and what tools are available when people have shops. The purpose of this is to help people buy goods.</s>   \n",
       "12  <pad> #Person1# wants to form a rock band and asks #Person2# about the musical talent of the other members of the band as well as the details of their songs. #Person1# begins learning to play the drums. #Person2# asks about #Person1#'s fantasy of music and talks to #Person1# about parts for the band.</s>   \n",
       "13                                                                                                                                 <pad> Alice is staying at home and can't go to see Mrs. Brown tomorrow morning because her mother is sick. Li Hong apologizes to Alice. Her family can visit Mrs. Brown later.</s>   \n",
       "14                                                                                                                                                                      <pad> #Person1# has just finished the paper he's written. Mom has proofread it and personally finishes it. #Person1#'s mom will be proud.</s>   \n",
       "15                                                                                                                                                                                            <pad> #Person2# wants to buy a toy car for son. #Person1# helps #Person2# find a cheaper one that costs 60 dollars.</s>   \n",
       "16                                                                                                                                                                                  <pad> #Person1# wants to confirm the flight, but #Person2# tells her the flight number and #Person1#'ll dial 35 when leaving.</s>   \n",
       "17                                                                                                                                                        <pad> #Person1# reports 7 goodbye letters to #Person2# about the revised contract and the quality standards. #Person2# is happy to sign their contract.</s>   \n",
       "18                                                                                                                                                                                           <pad> Judy and Judy are surprised about Richard being fired. Judy didn't trust him because she's surprised about it.</s>   \n",
       "19                                                                               <pad> There's another smoking option to his health and the others could help. #Person1# doesn't think he is having the willpower to quit. #Person1# says they can't help because they spend a fortune on cigarettes every month.</s>   \n",
       "\n",
       "                                                                                                                                                                                                                        response_after  \\\n",
       "0                                                                                                               <pad> #Person1#'s flight had moved in 15 minutes after #Person1#'s request but #Person1#'s flight did not come by.</s>   \n",
       "1                                                                         <pad> #Person1# needs the cash in small change to cashed a bill. #Person2# please deposit this amount, and emailed to #Person1# exchange receipt number.</s>   \n",
       "2                                                                                          <pad> #Person1# asks [Person2# about the restaurant. It's new and it's not getting the same is customers' opinion as at previous times.</s>   \n",
       "3                                                                                                                    <pad> #Person2# recommends dial-up internet because it doesn't tie up the phone and does the work by calling.</s>   \n",
       "4   <pad> #Person1# is going to consult the consultant department. #Person2# reminds #Person1# to pay for the registration and give #Person1# documents and directions. Then #Person1# finds an easy way to the university center.</s>   \n",
       "5                                                                                                           <pad> #Person2# sells some vegetables at 150 yuan. #Person1# supports it and helps #Person2# to increase their supply.</s>   \n",
       "6                                                                                           <pad> #Person1# wants to work in an office full-time and asks #Person2# for help. #Person2# allows #Person1# to hire a adequate staff.</s>   \n",
       "7                                                                                                                             <pad> #Person1# suggests #Person2# take a coffee break but #Person2# can't because of the work load.</s>   \n",
       "8                       <pad> Allen remembers saying his window is open when he went to a house; on the other hand he locked the windows early because of the breaking in. He doesn't care about that because he'll look upstairs.</s>   \n",
       "9                                                        <pad> #Person1# asks the Cross Bakery leading to the Cross Bakery. #Person2# recommends him the west direction. #Person1# tells #Person2# the way before #Person1# walks.</s>   \n",
       "10                                                                                         <pad> Amanda likes the peaked cap but #Person2# doesn't like caps. #Person1# doesn't like caps, so Amanda suggests a sombrero in black.</s>   \n",
       "11                                                                              <pad> #Person1# tells #Person2# about practical activities for the purpose of customizable computers and how customers can get goods through them.</s>   \n",
       "12                                                                                                                      <pad> #Person1# is forming a music band with Bob and Buddy and invites #Person1# to audition this weekend.</s>   \n",
       "13                                             <pad> Alice tells Li Hong she can't go to Mrs. Brown because her mother is ill. Alice decides to stay at home and says she won't see Mrs. Brown later because she'll be home later.</s>   \n",
       "14                                                                                                                        <pad> #Person1# asks #Person2# to check #Person1#'s paper and be sure to give #Person1# some right tips.</s>   \n",
       "15                                                                                                        <pad> Mrs. Bauer offers her an affordable toy car that she can buy for her son because it's worth three hundred dollars.</s>   \n",
       "16                                                                      <pad> #Person1# realizes she needs some help because she cannot connect with the airline in English. #Person2# invites her by phone, but not the Internet.</s>   \n",
       "17                       <pad> #Person1# presents the final draft of the contract to #Person2# and questions a few points about the table. #Person2# reminds #Person1# about chipping the customer. Sign the contract immediately.</s>   \n",
       "18                                                                                                                                 <pad> Judy's wondering how everyone is talking about Richard being fired. Judy agrees with her.</s>   \n",
       "19                                                                                                                                               <pad> #Person2# has a smell of smoke. Then she turns down and asks for a divorce.</s>   \n",
       "\n",
       "    reward_before  reward_after  reward_diff  \n",
       "0        1.768565      2.395374     0.626809  \n",
       "1        2.232070      2.663300     0.431230  \n",
       "2        1.959794      2.355813     0.396019  \n",
       "3        2.429675      2.721628     0.291954  \n",
       "4        1.505595      1.728968     0.223373  \n",
       "5        2.661622      2.800712     0.139090  \n",
       "6        2.008402      2.129998     0.121596  \n",
       "7        1.779952      1.896657     0.116705  \n",
       "8        1.786834      1.875944     0.089109  \n",
       "9        2.729239      2.749945     0.020705  \n",
       "10       1.409167      1.425640     0.016474  \n",
       "11       2.766026      2.691949    -0.074076  \n",
       "12       2.851743      2.741764    -0.109980  \n",
       "13       1.770844      1.636200    -0.134644  \n",
       "14       2.693838      2.512838    -0.181000  \n",
       "15       1.400756      1.194819    -0.205937  \n",
       "16       1.849587      1.447008    -0.402579  \n",
       "17       3.099102      2.632910    -0.466192  \n",
       "18       1.315041      0.028662    -1.286379  \n",
       "19       1.959103      0.550751    -1.408353  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('display.max_colwidth', 500)\n",
    "df_compare_results = pd.DataFrame(compare_results)\n",
    "df_compare_results[\"reward_diff\"] = df_compare_results['reward_after'] - df_compare_results['reward_before']\n",
    "df_compare_results_sorted = df_compare_results.sort_values(by=['reward_diff'], ascending=False).reset_index(drop=True)\n",
    "df_compare_results_sorted"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7fb2477-f719-48de-b169-0607d355a8f6",
   "metadata": {},
   "source": [
    "Looking at the reward mean/median of the generated sequences you can observe a significant difference!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df388872-4d25-42e0-b624-50d10311567f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48902a4c-ed2a-4737-9879-365dc77accd5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4637a07-b78f-44d2-b5af-c98ccd08a78d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 57,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.trn1.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 58,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1.32xlarge",
    "vcpuNum": 128
   },
   {
    "_defaultOrder": 59,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.trn1n.32xlarge",
    "vcpuNum": 128
   }
  ],
  "instance_type": "ml.m5.2xlarge",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}