BERT_fine_tuning_intent_classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/xcomponent/chatbot-nlp/blob/main/BERT_fine_tuning_intent_classification.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EKOTlwcmxmej"
      },
      "source": [
        "# BERT Fine-Tuning Tutorial with PyTorch\n",
        "\n",
        "By Chris McCormick and Nick Ryan"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RX_ZDhicpHkV"
      },
      "source": [
        "# 1. Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GI0iOY8zvZzL"
      },
      "source": [
        "\n",
        "Google Colab offers free GPUs and TPUs! Since we'll be training a large neural network it's best to take advantage of this (in this case we'll attach a GPU), otherwise training will take a very long time.\n",
        "\n",
        "A GPU can be added by going to the menu and selecting:\n",
        "\n",
        "`Edit 🡒 Notebook Settings 🡒 Hardware accelerator 🡒 (GPU)`\n",
        "\n",
        "Then run the following cell to confirm that the GPU is detected."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DEfSbAA4QHas",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "daee25d0-6274-4e3a-dd72-1d904f223725"
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "# Get the GPU device name.\n",
        "device_name = tf.test.gpu_device_name()\n",
        "\n",
        "# The device name should look like the following:\n",
        "if device_name == '/device:GPU:0':\n",
        "    print('Found GPU at: {}'.format(device_name))\n",
        "else:\n",
        "    raise SystemError('GPU device not found')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Found GPU at: /device:GPU:0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cqG7FzRVFEIv"
      },
      "source": [
        "In order for torch to use the GPU, we need to identify and specify the GPU as the device. Later, in our training loop, we will load data onto the device. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oYsV4H8fCpZ-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e9a71595-97c7-45fc-def1-405b37f60243"
      },
      "source": [
        "import torch\n",
        "\n",
        "# If there's a GPU available...\n",
        "if torch.cuda.is_available():    \n",
        "\n",
        "    # Tell PyTorch to use the GPU.    \n",
        "    device = torch.device(\"cuda\")\n",
        "\n",
        "    print('There are %d GPU(s) available.' % torch.cuda.device_count())\n",
        "\n",
        "    print('We will use the GPU:', torch.cuda.get_device_name(0))\n",
        "\n",
        "# If not...\n",
        "else:\n",
        "    print('No GPU available, using the CPU instead.')\n",
        "    device = torch.device(\"cpu\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "There are 1 GPU(s) available.\n",
            "We will use the GPU: Tesla T4\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0NmMdkZO8R6q",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7d8bf2ff-851e-40ed-b3b1-834299262006"
      },
      "source": [
        "!pip install transformers"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: transformers in /usr/local/lib/python3.8/dist-packages (4.25.1)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.10.0 in /usr/local/lib/python3.8/dist-packages (from transformers) (0.11.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers) (3.8.0)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers) (0.13.2)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (2022.6.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from transformers) (21.3)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from transformers) (6.0)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.8/dist-packages (from transformers) (1.21.6)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.8/dist-packages (from transformers) (4.64.1)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.8/dist-packages (from huggingface-hub<1.0,>=0.10.0->transformers) (4.4.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging>=20.0->transformers) (3.0.9)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (2022.9.24)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->transformers) (2.10)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "guw6ZNtaswKc"
      },
      "source": [
        "# 2. Loading Dataset\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/colab-data-drive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_0yPSLUqbEDz",
        "outputId": "13959ea1-aad9-432d-f693-ebf036f36ca0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /colab-data-drive; to attempt to forcibly remount, call drive.mount(\"/colab-data-drive\", force_remount=True).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "\n",
        "def load_data(training_data_fn):\n",
        "\n",
        "    messages = []\n",
        "\n",
        "    df = pd.read_excel(training_data_fn, header = None)\n",
        "\n",
        "    return df"
      ],
      "metadata": {
        "id": "4BJtWPSzXymE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "training_data_fn=r\"/colab-data-drive/MyDrive/Colab Notebooks/data/training_data/6377b04fa1ef9405a2a0575b_training_data_FR.xlsx\" # 39 intents smalltalks\n",
        "# training_data_fn=r\"/colab-data-drive/MyDrive/Colab Notebooks/data/training_data/pfs_SAV_training_data_FR.xlsx\" # pfs SAV\n",
        "\n",
        "df = load_data(training_data_fn)\n",
        "\n",
        "all_messages = []\n",
        "all_intents = []\n",
        "\n",
        "intents_id_name = {}\n",
        "\n",
        "for i, row in df.iterrows():\n",
        "  message, intent, intent_name, language = row\n",
        "  all_messages.append(message)\n",
        "  all_intents.append(intent_name)\n",
        "\n",
        "  if intent not in intents_id_name:\n",
        "      intents_id_name[intent] = intent_name\n",
        "\n",
        "print(f\"Training data has {len(all_messages)} messages\")\n",
        "print(f\"Training data contains {len(set(all_intents))} intents\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PeNKMfs_X-mc",
        "outputId": "402e2d0c-16de-4c04-8fe0-4f5ab4a0a828"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training data has 687 messages\n",
            "Training data contains 39 intents\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Tuto Chris McCormick"
      ],
      "metadata": {
        "id": "k57LH3mXmdZ_"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ex5O1eV-Pfct"
      },
      "source": [
        "## Tokenization & Input Formatting\n",
        "\n",
        "**In this section, we'll transform our dataset into the format that BERT can be trained on.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bWOPOyWghJp2"
      },
      "source": [
        "\n",
        "To feed our text to BERT, it must be split into tokens, and then these tokens must be mapped to their index in the tokenizer vocabulary.\n",
        "\n",
        "The tokenization must be performed by the tokenizer included with BERT--the below cell will download this for us. We'll be using the \"uncased\" version here.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z474sSC6oe7A",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8dff1a2d-d8fd-44d3-831a-58e4f830e716"
      },
      "source": [
        "from transformers import AutoTokenizer\n",
        "\n",
        "# Load the BERT tokenizer.\n",
        "print('Loading BERT tokenizer...')\n",
        "tokenizer = AutoTokenizer.from_pretrained('camembert-base', do_lower_case=True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loading BERT tokenizer...\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dFzmtleW6KmJ"
      },
      "source": [
        "Let's apply the tokenizer to one sentence just to see the output.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dLIbudgfh6F0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8d91e995-6bd8-4e23-851b-6964c5582bbd"
      },
      "source": [
        "# Print the original sentence.\n",
        "print('Original: ', all_messages[0])\n",
        "\n",
        "# Print the sentence split into tokens.\n",
        "print('Tokenized: ', tokenizer.tokenize(all_messages[0]))\n",
        "\n",
        "# Print the sentence mapped to token ids.\n",
        "print('Token IDs: ', tokenizer.convert_tokens_to_ids(tokenizer.tokenize(all_messages[0])))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Original:  j'ai une question\n",
            "Tokenized:  ['▁j', \"'\", 'ai', '▁une', '▁question']\n",
            "Token IDs:  [76, 11, 73, 28, 397]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WeNIc4auFUdF"
      },
      "source": [
        "When we actually convert all of our sentences, we'll use the `tokenize.encode` function to handle both steps, rather than calling `tokenize` and `convert_tokens_to_ids` separately. \n",
        "\n",
        "Before we can do that, though, we need to talk about some of BERT's formatting requirements."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yDcqNlvVhL5W"
      },
      "source": [
        "The above code left out a few required formatting steps that we'll look at here.\n",
        "\n",
        "*Side Note: The input format to BERT seems \"over-specified\" to me... We are required to give it a number of pieces of information which seem redundant, or like they could easily be inferred from the data without us explicity providing it. But it is what it is, and I suspect it will make more sense once I have a deeper understanding of the BERT internals.*\n",
        "\n",
        "We are required to:\n",
        "1. Add special tokens to the start and end of each sentence.\n",
        "2. Pad & truncate all sentences to a single constant length.\n",
        "3. Explicitly differentiate real tokens from padding tokens with the \"attention mask\".\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l6w8elb-58GJ"
      },
      "source": [
        "### Tokenize Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U28qy4P-NwQ9"
      },
      "source": [
        "The transformers library provides a helpful `encode` function which will handle most of the parsing and data prep steps for us.\n",
        "\n",
        "Before we are ready to encode our text, though, we need to decide on a **maximum sentence length** for padding / truncating to.\n",
        "\n",
        "The below cell will perform one tokenization pass of the dataset in order to measure the maximum sentence length."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cKsH2sU0OCQA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5cda493b-228d-4be2-ce22-2e5d0ce6e376"
      },
      "source": [
        "max_len = 0\n",
        "\n",
        "# For every sentence...\n",
        "max_length_message = None\n",
        "for message in all_messages:\n",
        "\n",
        "    # Tokenize the text and add `[CLS]` and `[SEP]` tokens.\n",
        "    input_ids = tokenizer.encode(message, add_special_tokens=True)\n",
        "    # print(f'{message} : {input_ids}')\n",
        "\n",
        "    # Update the maximum sentence length.\n",
        "    max_len = max(max_len, len(input_ids))\n",
        "\n",
        "    if len(input_ids) == max_len:\n",
        "      max_length_message = message\n",
        "\n",
        "print('Max message length: ', max_len)\n",
        "print(max_length_message)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Max message length:  12\n",
            "est-ce que tu es content aujourd'hui\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1M296yz577fV"
      },
      "source": [
        "Just in case there are some longer test sentences, I'll set the maximum length to 64.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tIWAoWL2RK1p"
      },
      "source": [
        "Now we're ready to perform the real tokenization.\n",
        "\n",
        "The `tokenizer.encode_plus` function combines multiple steps for us:\n",
        "\n",
        "1. Split the sentence into tokens.\n",
        "2. Add the special `[CLS]` and `[SEP]` tokens.\n",
        "3. Map the tokens to their IDs.\n",
        "4. Pad or truncate all sentences to the same length.\n",
        "5. Create the attention masks which explicitly differentiate real tokens from `[PAD]` tokens.\n",
        "\n",
        "The first four features are in `tokenizer.encode`, but I'm using `tokenizer.encode_plus` to get the fifth item (attention masks). Documentation is [here](https://huggingface.co/transformers/main_classes/tokenizer.html?highlight=encode_plus#transformers.PreTrainedTokenizer.encode_plus).\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "all_messages"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pcyAqZScdv1k",
        "outputId": "5711496e-7b99-4318-bdaa-aaa8c4eed135"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[\"j'ai une question\",\n",
              " 'il me faut une réponse',\n",
              " \"j'ai besoin d'une réponse\",\n",
              " 'donne une réponse',\n",
              " 'réponds',\n",
              " 'donne-moi une réponse à ma question',\n",
              " 'peux-tu me donner une réponse',\n",
              " 'je voudrais que tu me répondes',\n",
              " 'merci de répondre',\n",
              " 'réponds-moi',\n",
              " 'peux-tu me répondre',\n",
              " 'réponds à la question',\n",
              " 'je suis en train de te tester',\n",
              " 'pourrais-je te tester un peu',\n",
              " 'est-ce possible de te tester',\n",
              " \"j'aimerais te tester\",\n",
              " 'je veux te tester',\n",
              " 'est-ce que je peux te tester',\n",
              " 'puis-je te tester',\n",
              " 'je te teste',\n",
              " 'je te teste en fait',\n",
              " 'super de te parler de nouveau',\n",
              " \"c'est un plaisir de te parler\",\n",
              " 'ça me fait plaisir de te parler',\n",
              " 'ça fait plaisir de te parler',\n",
              " \"c'est un plaisir de parler\",\n",
              " 'content de te parler',\n",
              " 'je suis content de te parler',\n",
              " 'je suis contente de te parler',\n",
              " 'je suis heureux de te parler',\n",
              " 'je suis heureuse de te parler',\n",
              " \"c'est bien de te parler\",\n",
              " \"c'est super de te parler\",\n",
              " 'je suis enchanté de te parler',\n",
              " 'je suis enchantée de te parler',\n",
              " 'je suis enchanté de parler',\n",
              " 'je suis enchantée de parler',\n",
              " 'enchanté de te parler',\n",
              " 'enchantée de te parler',\n",
              " 'enchanté de parler',\n",
              " 'enchantée de parler',\n",
              " \"c'est un plaisir de te reparler\",\n",
              " 'ça me fait plaisir de te reparler',\n",
              " 'ça fait plaisir de te reparler',\n",
              " \"c'est un plaisir de reparler\",\n",
              " 'content de te reparler',\n",
              " 'je suis content de te reparler',\n",
              " 'je suis contente de te reparler',\n",
              " 'je suis heureux de te reparler',\n",
              " 'je suis heureuse de te reparler',\n",
              " \"c'est bien de te reparler\",\n",
              " \"c'est super de te reparler\",\n",
              " 'je suis enchanté de te reparler',\n",
              " 'je suis enchantée de te reparler',\n",
              " 'je suis enchanté de reparler',\n",
              " 'je suis enchantée de reparler',\n",
              " 'enchanté de te reparler',\n",
              " 'enchantée de te reparler',\n",
              " 'enchanté de reparler',\n",
              " 'enchantée de reparler',\n",
              " 'salut',\n",
              " 'bonjour à toi',\n",
              " 'coucou',\n",
              " 're salut',\n",
              " 'salutations',\n",
              " 'hé',\n",
              " 're bonjour',\n",
              " 'hey',\n",
              " 'allo',\n",
              " 'rebonjour',\n",
              " 'bonjour',\n",
              " 'salut à tous',\n",
              " 'resalut',\n",
              " 'je te souhaite une bonne nuit',\n",
              " 'bonne nuit',\n",
              " 'dors bien',\n",
              " 'fais de beaux rêves',\n",
              " 'fais des beaux rêves',\n",
              " 'beaux rêves',\n",
              " 'beaux rêves à toi',\n",
              " 'bonne nuit à toi',\n",
              " 'passe une bonne nuit',\n",
              " 'je te souhaite de passer une bonne nuit',\n",
              " 'bonne nuit toi',\n",
              " 'bonne matinée',\n",
              " 'bon matinée à toi',\n",
              " 'coucou bonne matinée',\n",
              " \"coucou c'est un bon matin\",\n",
              " 'passe une bonne matinée',\n",
              " 'je te souhaite une bonne matinée',\n",
              " 'bonsoir',\n",
              " 'salut bonsoir',\n",
              " 'soir',\n",
              " 'bon soir',\n",
              " 'bonsoir à toi',\n",
              " 'bonsoir à vous',\n",
              " 'soir à toi',\n",
              " 'tu passes une bonne soirée',\n",
              " 'est-ce que tu passes une bonne soirée',\n",
              " \"j'espère que tu passes une bonne soirée\",\n",
              " 'Au revoir',\n",
              " 'on se parle bientôt',\n",
              " 'à bientôt',\n",
              " 'à plus tard',\n",
              " 'à la prochaine',\n",
              " 'a plus dans le bus',\n",
              " 'à la prochaine',\n",
              " 'à la revoyure',\n",
              " 'adieu',\n",
              " 'au plaisir',\n",
              " 'merci',\n",
              " 'merci beaucoup',\n",
              " 'merci bien',\n",
              " 'je te remercie',\n",
              " 'grand merci',\n",
              " \"j'apprécie ça\",\n",
              " \"j'apprécie cela\",\n",
              " \"je l'apprécie\",\n",
              " 'je suis reconnaissant',\n",
              " 'je suis reconnaissante',\n",
              " 'remerciement',\n",
              " 'gratitude',\n",
              " 'merci à toi',\n",
              " 'eh bien merci',\n",
              " 'merci bien à toi',\n",
              " 'je te remercie beaucoup',\n",
              " 'merci bcp toi',\n",
              " 'chui reconnaissant',\n",
              " 'chui reconnaissante',\n",
              " 'merci toi aussi',\n",
              " 'merci à toi aussi',\n",
              " 'super merci',\n",
              " 'merci mon ami',\n",
              " 'Okay, merci',\n",
              " \"c'est noté, merci !\",\n",
              " 'Ok, merci.',\n",
              " 'quelles origines as-tu',\n",
              " 'quelles sont tes origines',\n",
              " \"c'est quoi ton pays d'origine\",\n",
              " \"d'où tu viens\",\n",
              " \"dis-moi d'où tu viens\",\n",
              " \"quel est ton pays d'origine\",\n",
              " \"d'où est-ce que tu viens\",\n",
              " \"d'où es-tu\",\n",
              " \"d'où viens-tu\",\n",
              " \"tu es d'où\",\n",
              " 'hobby',\n",
              " \"que fais-tu pour t'amuser\",\n",
              " 'tu as quoi comme hobby',\n",
              " 'est-ce que tu as un hobby',\n",
              " 'as-tu des hobbies',\n",
              " \"c'est quoi ton hobby\",\n",
              " 'quel est ton hobby',\n",
              " 'ton hobby',\n",
              " 'tes hobbies',\n",
              " 'tu as quelques hobbies',\n",
              " 'tes hobbys',\n",
              " 'en quoi est-ce que ton travail consiste',\n",
              " \"c'est quoi ton travail\",\n",
              " 'dans quoi travailles-tu',\n",
              " 'quel est ton job',\n",
              " 'qui es tu ?',\n",
              " 'es-tu réel',\n",
              " \"t'es un robot\",\n",
              " \"t'es un bot\",\n",
              " 'tu es un robot',\n",
              " 'tu es un robot toi',\n",
              " 'es-tu un robot',\n",
              " 'es-tu un chat bot',\n",
              " \"t'es un chat bot toi\",\n",
              " 'es-tu un agent conversationnel',\n",
              " 'tu es un chatbot',\n",
              " 'es-tu une personne réelle',\n",
              " 'on dirait que tu es une vraie personne',\n",
              " 'es-tu une vraie personne',\n",
              " 'tu es un vrai humain',\n",
              " \"tu as l'air tellement vraisemblable\",\n",
              " 'tu es réelle ou non',\n",
              " 'tu es réel ou non',\n",
              " 'tu es réelle ou pas',\n",
              " 'tu es réel ou pas',\n",
              " 'tu es un bot',\n",
              " 'es-tu réelle',\n",
              " 'Quel est ton nom',\n",
              " \"Comment tu t'appelles ?\",\n",
              " 'Comment vous appelez vous ?',\n",
              " \"C'est quoi ton nom?\",\n",
              " \"T'es qui toi ?\",\n",
              " 'Comment va',\n",
              " 'bien ou bien',\n",
              " 'ça va ?',\n",
              " 'Comment vas-tu',\n",
              " 'Comment allez-vous',\n",
              " 'Tu vas bien?',\n",
              " 'Vous allez bien ?',\n",
              " 'je suis trop épuisée',\n",
              " 'je suis épuisé',\n",
              " 'je suis vraiment fatiguée',\n",
              " 'je suis vraiment fatigué',\n",
              " 'je suis tellement fatiguée',\n",
              " 'je suis tellement fatigué',\n",
              " 'je suis très fatiguée',\n",
              " 'je suis très fatigué',\n",
              " 'je suis trop fatiguée',\n",
              " 'je suis trop fatigué',\n",
              " 'je suis fatigué',\n",
              " 'je suis crevé',\n",
              " 'je suis fatiguée',\n",
              " 'je suis crevée',\n",
              " \"j'aimerais bavarder\",\n",
              " 'je voudrais causer',\n",
              " \"j'ai envie de parler\",\n",
              " 'je veux parler avec toi',\n",
              " \"j'aimerais te parler\",\n",
              " 'je veux te parler',\n",
              " 'et si on discutait',\n",
              " 'veux-tu discuter maintenant',\n",
              " 'peut-on discuter',\n",
              " \"j'aimerais discuter avec toi\",\n",
              " 'puis-je parler',\n",
              " \"voudrais-tu qu'on se parle\",\n",
              " 'je voudrais causer avec toi',\n",
              " 'pourrait-on causer',\n",
              " 'peut-on se parler',\n",
              " \"j'aimerais qu'on se parle\",\n",
              " 'je veux parler',\n",
              " 'parlons',\n",
              " 'parlons ensemble',\n",
              " 'je suis tout triste',\n",
              " 'ça me donne envie de pleurer',\n",
              " \"j'ai envie de pleurer\",\n",
              " \"c'est trop triste\",\n",
              " 'je suis trop triste',\n",
              " 'je me sens trop triste',\n",
              " 'je me sens très triste',\n",
              " 'je suis très triste',\n",
              " 'je suis triste',\n",
              " 'je me sens triste',\n",
              " \"je suis de l'humeur triste\",\n",
              " 'tu me plais beaucoup',\n",
              " 'tu me plais bien',\n",
              " 'toi aussi tu me plais',\n",
              " 'toi aussi tu me plais',\n",
              " 'moi aussi tu me plais',\n",
              " \"je t'aime bien moi aussi\",\n",
              " \"je t'aime bien également\",\n",
              " 'tu me plais bien aussi toi',\n",
              " 'tu me plais aussi',\n",
              " \"je t'aime bien aussi\",\n",
              " \"je t'aime bien\",\n",
              " 'tu me plais',\n",
              " 'je rigole',\n",
              " 'je plaisantais',\n",
              " 'je blaguais',\n",
              " \"c'était une plaisanterie\",\n",
              " \"c'était une blague\",\n",
              " 'je rigolais',\n",
              " \"c'est une plaisanterie\",\n",
              " 'je rigole là',\n",
              " \"ce n'est qu'une blague\",\n",
              " 'ceci est une blague',\n",
              " \"ça c'est une blague\",\n",
              " \"c'est une blague\",\n",
              " 'je blague',\n",
              " 'je plaisante',\n",
              " 'ça me rend heureuse',\n",
              " 'ça me rend heureux',\n",
              " \"j'en suis heureuse\",\n",
              " \"j'en suis heureux\",\n",
              " 'je suis très heureuse',\n",
              " 'je suis très heureux',\n",
              " 'je suis super heureux',\n",
              " 'je suis super heureuse',\n",
              " 'je suis tellement heureuse',\n",
              " 'je suis tellement heureux',\n",
              " 'je suis heureux',\n",
              " 'je suis heureuse',\n",
              " 'je suis trop enthousiasmée pour ça',\n",
              " 'je suis impatient',\n",
              " \"j'en ai hâte\",\n",
              " \"j'ai hâte\",\n",
              " 'je suis enthousiasmé',\n",
              " \"je m'ennuie\",\n",
              " \"je m'ennuie quoi faire\",\n",
              " \"je m'ennuie tout le temps\",\n",
              " \"je m'ennuie toujours\",\n",
              " \"c'est que je m'ennuie\",\n",
              " \"c'est que je suis ennuyé\",\n",
              " \"ça m'ennuie\",\n",
              " \"c'est ennuyeux\",\n",
              " \"je m'ennuie là\",\n",
              " \"je m'ennuie énormement\",\n",
              " \"je m'ennuie trop\",\n",
              " 'ça me met en colère',\n",
              " 'ça me rend furieux',\n",
              " 'je suis en colère',\n",
              " \"ça m'énerve\",\n",
              " 'je suis énervé',\n",
              " 'je suis énervée',\n",
              " 'je suis super fâchée',\n",
              " 'je suis super fâché',\n",
              " 'je suis trop fâchée',\n",
              " 'je suis trop fâché',\n",
              " 'fâchées',\n",
              " 'fâchés',\n",
              " 'fâchée',\n",
              " 'fâché',\n",
              " 'faché',\n",
              " 'fachée',\n",
              " 'je suis fâchée',\n",
              " 'je suis fâché',\n",
              " 'je suis furieux',\n",
              " 'je suis furieuse',\n",
              " 'super de te revoir',\n",
              " \"c'est un plaisir de te voir\",\n",
              " 'ça me fait plaisir de te voir',\n",
              " 'ça fait plaisir de te voir',\n",
              " 'content de te voir',\n",
              " 'je suis content de te voir',\n",
              " 'je suis contente de te voir',\n",
              " 'je suis heureux de te voir',\n",
              " 'je suis heureuse de te voir',\n",
              " \"c'est bien de te voir\",\n",
              " \"c'est super de te voir\",\n",
              " 'je suis enchanté de te voir',\n",
              " 'je suis enchantée de te voir',\n",
              " 'je suis enchanté de voir',\n",
              " 'je suis enchantée de voir',\n",
              " 'enchanté de te voir',\n",
              " 'enchantée de te voir',\n",
              " 'enchanté de voir',\n",
              " 'enchantée de voir',\n",
              " \"c'est un plaisir de te revoir\",\n",
              " 'ça me fait plaisir de te revoir',\n",
              " 'ça fait plaisir de te revoir',\n",
              " \"c'est un plaisir de revoir\",\n",
              " 'content de te revoir',\n",
              " 'je suis content de te revoir',\n",
              " 'je suis contente de te revoir',\n",
              " 'je suis heureux de te revoir',\n",
              " 'je suis heureuse de te revoir',\n",
              " \"c'est bien de te revoir\",\n",
              " \"c'est super de te revoir\",\n",
              " 'je suis enchanté de te revoir',\n",
              " 'je suis enchantée de te revoir',\n",
              " 'je suis enchanté de revoir',\n",
              " 'je suis enchantée de revoir',\n",
              " 'enchanté de te revoir',\n",
              " 'enchantée de te revoir',\n",
              " 'enchanté de revoir',\n",
              " 'enchantée de revoir',\n",
              " \"c'est super de te rencontrer\",\n",
              " \"c'est un plaisir de te rencontrer\",\n",
              " 'ça me fait plaisir de te rencontrer',\n",
              " 'ça fait plaisir de te rencontrer',\n",
              " 'content de te rencontrer',\n",
              " 'je suis content de te rencontrer',\n",
              " 'je suis contente de te rencontrer',\n",
              " 'je suis heureux de te rencontrer',\n",
              " 'je suis heureuse de te rencontrer',\n",
              " \"c'est bien de te rencontrer\",\n",
              " 'enchanté',\n",
              " 'enchantée',\n",
              " 'je suis enchanté',\n",
              " 'je suis enchantée',\n",
              " 'je suis enchanté de te rencontrer',\n",
              " 'je suis enchantée de te rencontrer',\n",
              " 'je suis enchanté de rencontrer',\n",
              " 'je suis enchantée de rencontrer',\n",
              " 'enchanté de te rencontrer',\n",
              " 'enchantée de te rencontrer',\n",
              " 'bien de te recontrer',\n",
              " 'enchanté de rencontrer',\n",
              " 'enchantée de rencontrer',\n",
              " 'waouuuh',\n",
              " 'oh lalalala',\n",
              " 'wow wow wow',\n",
              " 'wow wow',\n",
              " 'ahlala',\n",
              " 'ohlala',\n",
              " 'ah la la',\n",
              " 'oh la la',\n",
              " 'waou',\n",
              " 'waouh',\n",
              " 'wow',\n",
              " 'mort de rire',\n",
              " 'mdr',\n",
              " 'ahaha',\n",
              " 'ahah',\n",
              " 'hahahah',\n",
              " 'hihi',\n",
              " 'ha ha',\n",
              " 'haha',\n",
              " 'tellement navré',\n",
              " 'je te présente mes excuses',\n",
              " 'ok désolé',\n",
              " \"j'en suis désolée\",\n",
              " \"j'en suis désolé\",\n",
              " \"j'en suis navrée\",\n",
              " \"j'en suis navré\",\n",
              " 'navrée',\n",
              " 'navré',\n",
              " 'pardon',\n",
              " 'tu me pardonneras',\n",
              " \"tu m'excuseras\",\n",
              " 'scuse-moi',\n",
              " \"peux-tu m'excuser\",\n",
              " \"tu m'excuses\",\n",
              " 'scuse',\n",
              " 'dsl',\n",
              " 'désolée',\n",
              " 'désolé',\n",
              " 'excuse-moi',\n",
              " 'pardonne-moi',\n",
              " \"je te demande de m'excuser\",\n",
              " 'toutes mes excuses',\n",
              " 'mes excuses',\n",
              " \"je m'excuse\",\n",
              " 'je suis désolé',\n",
              " 'je suis désolée',\n",
              " 'es-tu toujours ici',\n",
              " 'es-tu toujours là',\n",
              " \"t'es toujours là\",\n",
              " \"t'es toujours ici\",\n",
              " 'tu es toujours ici',\n",
              " 'tu es ici ou non',\n",
              " 'tu es ici',\n",
              " 'es-tu ici',\n",
              " 'est-ce que tu es ici',\n",
              " \"t'es là ou pas\",\n",
              " \"t'es là\",\n",
              " 'est-ce que tu es là',\n",
              " 'es-tu là',\n",
              " 'tu es là',\n",
              " \"t'es là toi\",\n",
              " 'parle',\n",
              " 'parle avec moi',\n",
              " \"il faut qu'on se parle\",\n",
              " 'on pourrait se parler',\n",
              " 'on doit se parler',\n",
              " 'je voudrais te parler',\n",
              " 'pourrions-nous nous parler',\n",
              " 'pourrait-on parler',\n",
              " 'peux-tu me parler',\n",
              " 'parle-moi',\n",
              " 'est-ce que tu vis en france',\n",
              " 'est-ce que tu habites en france',\n",
              " 'tu habites dans quelle ville',\n",
              " 'tu habites dans quel pays',\n",
              " 'dans quel pays tu habites',\n",
              " 'dans quel pays tu vis',\n",
              " 'ta ville',\n",
              " 'quelle ville tu habites',\n",
              " 'quelle ville habites-tu',\n",
              " 'dans quelle ville habites-tu',\n",
              " 'ou tu habites',\n",
              " 'ou tu vis',\n",
              " 'où tu habites',\n",
              " 'où tu vis',\n",
              " 'tu habites où',\n",
              " 'tu vis où',\n",
              " 'ou vis-tu',\n",
              " 'où habites-tu',\n",
              " 'où est-ce que tu habites',\n",
              " 'où est-ce que tu vis',\n",
              " 'es-tu mon ami',\n",
              " 'tu es mon meilleur copain',\n",
              " 'tu es ma pote',\n",
              " 'tu es mon meilleur ami',\n",
              " \"est-ce qu'on est copains\",\n",
              " \"est-ce qu'on est copines\",\n",
              " \"est-ce qu'on est amies\",\n",
              " \"est-ce qu'on est amis\",\n",
              " 'est-on des copines',\n",
              " 'est-on des copains',\n",
              " 'est-on des meilleures amies',\n",
              " 'est-on des meilleurs amis',\n",
              " 'est-on copines',\n",
              " 'est-on copains',\n",
              " 'est-on amies',\n",
              " 'est-on amis',\n",
              " 'on est super amis',\n",
              " 'on est copain toi et moi',\n",
              " 'on est copines',\n",
              " 'on est copains',\n",
              " 'on est meilleures amies',\n",
              " 'on est amies',\n",
              " 'on est meilleurs amis',\n",
              " 'on est amis',\n",
              " \"t'es ma meilleure copine\",\n",
              " 'tu es ma meilleure amie',\n",
              " 'tu es mon pote',\n",
              " 'tu es mon ami',\n",
              " 'tu es mon amie',\n",
              " 'tu es mon pote',\n",
              " 'tu as faim',\n",
              " 'voudrais-tu manger',\n",
              " 'est-ce que tu as faim',\n",
              " 'veux-tu manger',\n",
              " 'as-tu faim',\n",
              " \"t'as faim\",\n",
              " \"t'as faim toi\",\n",
              " 'tu veux manger là',\n",
              " 'aimerais-tu manger maintenant',\n",
              " 'tu voudrais manger',\n",
              " 'tu veux manger',\n",
              " \"est-ce que tu es content aujourd'hui\",\n",
              " \"es-tu heureux aujourd'hui\",\n",
              " 'tu es heureux ou triste',\n",
              " 'est-ce que tu es heureuse ou triste',\n",
              " 'tu es heureuse ou pas',\n",
              " 'tu es heureux ou non',\n",
              " 'es-tu contente',\n",
              " 'es-tu content',\n",
              " 'es-tu heureuse',\n",
              " 'es-tu heureux',\n",
              " 'tu es contente',\n",
              " 'tu es content',\n",
              " 'tu es heureux',\n",
              " 'tu es heureuse',\n",
              " 'tu es vraiment bien',\n",
              " 'tu es superbe',\n",
              " 'tu es hyper cool',\n",
              " 'tu es méga cool',\n",
              " \"t'es super méga cool\",\n",
              " 'super cool toi',\n",
              " 'tu es tellement cool',\n",
              " 'tu es tellement bien',\n",
              " 'tu es très cool',\n",
              " \"t'es trop cool\",\n",
              " 'tu es trop cool',\n",
              " 'tu es cool',\n",
              " \"t'es fantastique\",\n",
              " 'tu es fantastique',\n",
              " 'tu es forte',\n",
              " 'tu es fort',\n",
              " 'tu es pro',\n",
              " 'tu es parfaite',\n",
              " 'tu es excellente',\n",
              " 'tu es parfait',\n",
              " 'tu es excellent',\n",
              " 'tu es super bien',\n",
              " 'tu es énorme',\n",
              " 'tu es super',\n",
              " 'tu es trop bien',\n",
              " 'tu es bien',\n",
              " 'intelligent',\n",
              " 'vraiment intelligent',\n",
              " \"c'est intelligent\",\n",
              " \"c'est pas bête\",\n",
              " \"ce n'est pas bête ça\",\n",
              " 'ceci est super intelligent',\n",
              " 'tu es trop smart',\n",
              " 'tu es bien smart',\n",
              " 'tu es smart',\n",
              " 'tu es super smart',\n",
              " 'très smart toi',\n",
              " \"t'es ingénieux\",\n",
              " 'tu es ingénieuse',\n",
              " \"t'es vraiment intelligent\",\n",
              " 'tu es vraiment intelligente',\n",
              " 'tu es très intelligent',\n",
              " 'super intelligent toi',\n",
              " 'tu es super intelligente',\n",
              " \"t'es trop intelligente toi\",\n",
              " \"t'es super intelligente\",\n",
              " 'es-tu incollable',\n",
              " 'serais-tu incollable',\n",
              " \"t'es intelligente\",\n",
              " 'tu es intelligente',\n",
              " 'tu es intelligent',\n",
              " 'tu me fais trop marrer',\n",
              " 'tu me fais trop rire',\n",
              " 'tu me fais marrer',\n",
              " 'tu me fais rire',\n",
              " \"t'es super rigolo\",\n",
              " 'tu es rigolo',\n",
              " 'tu es trop amusant',\n",
              " 'tu es amusant',\n",
              " 'tu es amusante',\n",
              " 'tu es vraiment marrante',\n",
              " 'tu es méga drôle',\n",
              " 'tu es hyper drôle',\n",
              " 'tu es hyper marrant',\n",
              " \"t'es trop marrant\",\n",
              " 'tu es trop marrant',\n",
              " 'tu es super marrante',\n",
              " 'super drôle toi',\n",
              " 'tu es marrant toi',\n",
              " \"t'es marrant\",\n",
              " \"t'es marrante\",\n",
              " 'tu es marrant',\n",
              " 'tu es drôle',\n",
              " 'bien joué',\n",
              " 'bien fait',\n",
              " 'bravo',\n",
              " 'chapeau',\n",
              " 'travail bien fait',\n",
              " 'un grand bravo',\n",
              " \"c'est bien joué\",\n",
              " \"c'est bien fait\",\n",
              " 'ceci est bien joué',\n",
              " 'ceci est bien fait',\n",
              " 'cela est bien joué',\n",
              " 'cela est bien fait',\n",
              " 'es-tu folle',\n",
              " 'es-tu fou',\n",
              " \"t'es fou ou quoi\",\n",
              " 'trop fou\\xa0toi',\n",
              " \"t'es trop fou\",\n",
              " 'tu es trop folle',\n",
              " 'tu es folle',\n",
              " 'tu es cinglée',\n",
              " \"t'es taré\",\n",
              " \"t'es tarée\",\n",
              " 'tu es tarée',\n",
              " 'tu es taré',\n",
              " \"t'es cinglé\",\n",
              " 'tu es dingue',\n",
              " 'tu es fou',\n",
              " \"t'es folle\",\n",
              " 'la date de ta naissance',\n",
              " 'donne-moi ta date de naissance',\n",
              " 'dis-moi la date de ton anniversaire',\n",
              " 'la date de ton anniv',\n",
              " 'quelle est la date de ton anniv',\n",
              " 'il est quand ton anniversaire',\n",
              " \"ton anniversaire c'est quand\",\n",
              " \"c'est quand ton anniversaire\",\n",
              " 'quand est-ce que tu fêtes ton anniversaire',\n",
              " 'quelle est ta date de naissance',\n",
              " 'quelle est la date de ta naissance',\n",
              " 'quand est ton anniversaire',\n",
              " \"tu m'ennuies\",\n",
              " 'super ennuyeuse toi',\n",
              " 'tu es carrément ennuyeuse',\n",
              " 'tu es vraiment ennuyeux',\n",
              " 'tu sais que tu ennuyeuse',\n",
              " \"t'es ennuyeux toi\",\n",
              " 'trop ennuyeux toi',\n",
              " 'tu es enormement ennuyeux',\n",
              " 'tu es trop ennuyeuse',\n",
              " 'tu es très ennuyeuse',\n",
              " \"t'es pas drôle\",\n",
              " \"tu n'es pas drôle\",\n",
              " 'tu es ennuyeux',\n",
              " \"tu ne m'aides même pas\",\n",
              " \"tu ne m'aides pas\",\n",
              " 'pas tellement cool toi',\n",
              " 'pas cool toi',\n",
              " \"tu n'es pas trop cool\",\n",
              " \"tu n'es pas super\",\n",
              " \"t'es pas bien\",\n",
              " \"tu n'es pas bien\",\n",
              " 'tu es une mauvaise personne',\n",
              " 'tu es horrible',\n",
              " 'tu es nulle',\n",
              " \"t'es nul\",\n",
              " 'tu sers à rien',\n",
              " 'tu ne sers à rien',\n",
              " 'tu ne peux rien faire',\n",
              " \"t'es mauvais\",\n",
              " 'tu es mauvais',\n",
              " 'tu es ennuyante.',\n",
              " \"t'es trop agaçante\",\n",
              " 'tu es fatigante',\n",
              " 'tu es fatigant',\n",
              " \"tu m'embêtes\",\n",
              " \"tu m'agaces\",\n",
              " 'tu me fatigues',\n",
              " 'tu es agaçante',\n",
              " 'tu es agaçant',\n",
              " 'tu es embêtant',\n",
              " 'tu me fatigues énormement',\n",
              " 'tu est fatiguante toi',\n",
              " 'tu me fatigues trop',\n",
              " ' Tu es incompétent',\n",
              " ' Tu es incompétente',\n",
              " ' Tu es mauvais',\n",
              " ' Tu es mauvaise',\n",
              " 'Quel âge as-tu',\n",
              " \"T'es vieux de combien\",\n",
              " \"T'es vieille de combien\",\n",
              " \"T'as quel âge ?\",\n",
              " \"t'as été crée quand \",\n",
              " 'quand as-tu été créée',\n",
              " \"C'est quoi ton âge\"]"
            ]
          },
          "metadata": {},
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "intent_labels = {}\n",
        "\n",
        "for intent_id, intent_name in enumerate(list(set(all_intents))):\n",
        "  intent_labels[intent_name] = intent_id\n",
        "\n",
        "labels = []\n",
        "for intent_name in all_intents: \n",
        "  labels.append(intent_labels[intent_name])\n",
        "\n",
        "print(f\"Intent labels : {intent_labels}\")\n",
        "print(labels)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VyGkdQBQerkY",
        "outputId": "01ac0a5d-2e98-4521-baa5-6dfc1c6212d0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Intent labels : {\"Demander au chatbot s'il est réel\": 0, 'Être fatigué': 1, 'Remercier le chatbot': 2, 'Dire au revoir': 3, \"Demander au Chatbot s'il a faim\": 4, \"Dire au Chatbot qu'il est mon ami\": 5, 'Être désolé': 6, 'Critiquer le chatbot': 7, 'Souhaiter une bonne matinée': 8, 'Être enchanté de faire la rencontre': 9, 'Être surpris': 10, 'Être enthousiaste': 11, 'Demander où se trouve le chatbot': 12, 'Demander une réponse au chatbot': 13, 'Dire au Chatbot que je veux le tester': 14, 'Souhaiter une bonne nuit': 15, 'Vouloir bavarder': 16, 'Être content de revoir le Chatbot': 17, 'Rigoler': 18, \"Demander l'âge du Chatbot\": 19, \"Dire au Chatbot qu'il est fou\": 20, 'Être énervé': 21, 'Demander le nom du Chatbot': 22, 'Être heureux': 23, 'Prendre des nouvelles du chatbot': 24, \"S'ennuyer\": 25, \"Demander au Chatbot s'il est encore là\": 26, 'Être content de reparler au Chatbot': 27, 'Demander au Chatbot sa date de naissance': 28, 'Dire bonsoir': 29, 'Complimenter le chatbot': 30, 'Apprécier le chatbot': 31, 'Demander les origines du chatbot': 32, 'Demander des informations sur le Chatbot': 33, 'Demander au Chatbot de parler': 34, 'Être triste': 35, 'Saluer le chatbot': 36, 'Plaisanter': 37, \"Demander au Chatbot s'il est content\": 38}\n",
            "[13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 14, 14, 14, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 36, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 8, 8, 8, 8, 8, 8, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 33, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 22, 22, 22, 22, 22, 24, 24, 24, 24, 24, 24, 24, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 35, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 37, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 11, 11, 11, 11, 11, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 18, 18, 18, 18, 18, 18, 18, 18, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 34, 34, 34, 34, 34, 34, 34, 34, 34, 34, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 38, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 19, 19, 19, 19, 19, 19, 19]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2bBdb3pt8LuQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1146049a-2dc7-454d-d9da-6baa1a73fce8"
      },
      "source": [
        "# Tokenize all of the sentences and map the tokens to thier word IDs.\n",
        "input_ids = []\n",
        "attention_masks = []\n",
        "\n",
        "# For every sentence...\n",
        "for message in all_messages:\n",
        "    # `encode_plus` will:\n",
        "    #   (1) Tokenize the sentence.\n",
        "    #   (2) Prepend the `[CLS]` token to the start.\n",
        "    #   (3) Append the `[SEP]` token to the end.\n",
        "    #   (4) Map tokens to their IDs.\n",
        "    #   (5) Pad or truncate the sentence to `max_length`\n",
        "    #   (6) Create attention masks for [PAD] tokens.\n",
        "    encoded_dict = tokenizer.encode_plus(\n",
        "                        message,                      # Sentence to encode.\n",
        "                        add_special_tokens = True, # Add '[CLS]' and '[SEP]'\n",
        "                        max_length = 30,           # Pad & truncate all sentences.\n",
        "                        pad_to_max_length = True,\n",
        "                        return_attention_mask = True,   # Construct attn. masks.\n",
        "                        return_tensors = 'pt',     # Return pytorch tensors.\n",
        "                   )\n",
        "    \n",
        "    # Add the encoded sentence to the list.    \n",
        "    input_ids.append(encoded_dict['input_ids'])\n",
        "    \n",
        "    # And its attention mask (simply differentiates padding from non-padding).\n",
        "    attention_masks.append(encoded_dict['attention_mask'])\n",
        "\n",
        "# Convert the lists into tensors.\n",
        "input_ids = torch.cat(input_ids, dim=0)\n",
        "attention_masks = torch.cat(attention_masks, dim=0)\n",
        "labels = torch.tensor(labels)\n",
        "\n",
        "# Print sentence 0, now as a list of IDs.\n",
        "print('Original: ', all_messages[0])\n",
        "print('Token IDs:', input_ids[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Original:  j'ai une question\n",
            "Token IDs: tensor([  5,  76,  11,  73,  28, 397,   6,   1,   1,   1,   1,   1,   1,   1,\n",
            "          1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,   1,\n",
            "          1,   1])\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/transformers/tokenization_utils_base.py:2336: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).\n",
            "  warnings.warn(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aRp4O7D295d_"
      },
      "source": [
        "### Training & Validation Split\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qu0ao7p8rb06"
      },
      "source": [
        "Divide up our training set to use 90% for training and 10% for validation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GEgLpFVlo1Z-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3190f0bb-bb27-4cb9-b302-f33362247051"
      },
      "source": [
        "from torch.utils.data import TensorDataset, random_split\n",
        "\n",
        "# Combine the training inputs into a TensorDataset.\n",
        "dataset = TensorDataset(input_ids, attention_masks, labels)\n",
        "\n",
        "# Create a 90-10 train-validation split.\n",
        "\n",
        "# Calculate the number of samples to include in each set.\n",
        "train_size = int(0.9 * len(dataset))\n",
        "val_size = len(dataset) - train_size\n",
        "\n",
        "# Divide the dataset by randomly selecting samples.\n",
        "train_dataset, val_dataset = random_split(dataset, [train_size, val_size])\n",
        "\n",
        "print('{:>5,} training samples'.format(train_size))\n",
        "print('{:>5,} validation samples'.format(val_size))\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  618 training samples\n",
            "   69 validation samples\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dD9i6Z2pG-sN"
      },
      "source": [
        "We'll also create an iterator for our dataset using the torch DataLoader class. This helps save on memory during training because, unlike a for loop, with an iterator the entire dataset does not need to be loaded into memory."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XGUqOCtgqGhP"
      },
      "source": [
        "from torch.utils.data import DataLoader, RandomSampler, SequentialSampler\n",
        "\n",
        "# The DataLoader needs to know our batch size for training, so we specify it \n",
        "# here. For fine-tuning BERT on a specific task, the authors recommend a batch \n",
        "# size of 16 or 32.\n",
        "batch_size = 5\n",
        "\n",
        "# Create the DataLoaders for our training and validation sets.\n",
        "# We'll take training samples in random order. \n",
        "train_dataloader = DataLoader(\n",
        "            train_dataset,  # The training samples.\n",
        "            sampler = RandomSampler(train_dataset), # Select batches randomly\n",
        "            batch_size = batch_size # Trains with this batch size.\n",
        "        )\n",
        "\n",
        "# For validation the order doesn't matter, so we'll just read them sequentially.\n",
        "validation_dataloader = DataLoader(\n",
        "            val_dataset, # The validation samples.\n",
        "            sampler = SequentialSampler(val_dataset), # Pull out batches sequentially.\n",
        "            batch_size = batch_size # Evaluate with this batch size.\n",
        "        )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8bwa6Rts-02-"
      },
      "source": [
        "## Train Our Classification Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3xYQ3iLO08SX"
      },
      "source": [
        "Now that our input data is properly formatted, it's time to fine tune the BERT model. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1sjzRT1V0zwm"
      },
      "source": [
        "For this task, we first want to modify the pre-trained BERT model to give outputs for classification, and then we want to continue training the model on our dataset until that the entire model, end-to-end, is well-suited for our task. \n",
        "\n",
        "Thankfully, the huggingface pytorch implementation includes a set of interfaces designed for a variety of NLP tasks. Though these interfaces are all built on top of a trained BERT model, each has different top layers and output types designed to accomodate their specific NLP task.  \n",
        "\n",
        "Here is the current list of classes provided for fine-tuning:\n",
        "* BertModel\n",
        "* BertForPreTraining\n",
        "* BertForMaskedLM\n",
        "* BertForNextSentencePrediction\n",
        "* **BertForSequenceClassification** - The one we'll use.\n",
        "* BertForTokenClassification\n",
        "* BertForQuestionAnswering\n",
        "\n",
        "The documentation for these can be found under [here](https://huggingface.co/transformers/v2.2.0/model_doc/bert.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BXYitPoE-cjH"
      },
      "source": [
        "\n",
        "\n",
        "We'll be using [BertForSequenceClassification](https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#bertforsequenceclassification). This is the normal BERT model with an added single linear layer on top for classification that we will use as a sentence classifier. As we feed input data, the entire pre-trained BERT model and the additional untrained classification layer is trained on our specific task. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gFsCTp_mporB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "745010fa-46c6-46b0-c31f-61e1e5923749"
      },
      "source": [
        "from transformers import BertForSequenceClassification, AdamW, BertConfig\n",
        "\n",
        "# Load BertForSequenceClassification, the pretrained BERT model with a single \n",
        "# linear classification layer on top. \n",
        "model = BertForSequenceClassification.from_pretrained(\n",
        "    \"camembert-base\", \n",
        "    num_labels = len(list(set(all_intents))), \n",
        "    output_attentions = False, # Whether the model returns attentions weights.\n",
        "    output_hidden_states = False, # Whether the model returns all hidden-states.\n",
        ")\n",
        "\n",
        "# Tell pytorch to run this model on the GPU.\n",
        "model.cuda()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "You are using a model of type camembert to instantiate a model of type bert. This is not supported for all configurations of models and can yield errors.\n",
            "Some weights of the model checkpoint at camembert-base were not used when initializing BertForSequenceClassification: ['roberta.embeddings.token_type_embeddings.weight', 'roberta.encoder.layer.3.output.LayerNorm.weight', 'roberta.encoder.layer.7.intermediate.dense.bias', 'roberta.encoder.layer.6.intermediate.dense.bias', 'roberta.encoder.layer.2.intermediate.dense.weight', 'roberta.embeddings.position_embeddings.weight', 'roberta.encoder.layer.10.output.LayerNorm.bias', 'roberta.encoder.layer.9.attention.output.dense.bias', 'roberta.encoder.layer.10.intermediate.dense.bias', 'roberta.encoder.layer.9.intermediate.dense.bias', 'lm_head.layer_norm.weight', 'roberta.encoder.layer.0.output.LayerNorm.weight', 'roberta.encoder.layer.3.attention.output.LayerNorm.weight', 'roberta.encoder.layer.10.attention.output.LayerNorm.weight', 'roberta.encoder.layer.0.attention.output.LayerNorm.bias', 'roberta.encoder.layer.1.attention.output.dense.weight', 'roberta.encoder.layer.8.attention.self.value.weight', 'roberta.encoder.layer.1.attention.self.value.weight', 'roberta.encoder.layer.11.output.dense.weight', 'roberta.encoder.layer.9.output.dense.weight', 'roberta.encoder.layer.11.attention.output.LayerNorm.bias', 'roberta.encoder.layer.0.attention.self.key.bias', 'roberta.encoder.layer.4.attention.self.key.bias', 'roberta.encoder.layer.4.attention.output.LayerNorm.bias', 'roberta.encoder.layer.9.attention.self.value.bias', 'roberta.encoder.layer.2.output.dense.weight', 'roberta.embeddings.LayerNorm.bias', 'roberta.encoder.layer.7.attention.output.dense.bias', 'roberta.encoder.layer.10.attention.self.key.weight', 'roberta.encoder.layer.8.intermediate.dense.bias', 'roberta.encoder.layer.7.attention.self.query.weight', 'roberta.encoder.layer.1.output.LayerNorm.weight', 'roberta.encoder.layer.4.attention.self.query.weight', 'roberta.encoder.layer.3.intermediate.dense.weight', 'roberta.encoder.layer.2.output.LayerNorm.weight', 'roberta.encoder.layer.5.attention.self.value.bias', 'roberta.encoder.layer.8.attention.output.dense.weight', 'roberta.encoder.layer.11.attention.self.key.weight', 'roberta.encoder.layer.0.attention.self.query.bias', 'roberta.encoder.layer.0.attention.self.value.weight', 'roberta.encoder.layer.2.output.dense.bias', 'roberta.encoder.layer.5.attention.output.LayerNorm.bias', 'roberta.encoder.layer.6.attention.self.key.bias', 'roberta.encoder.layer.10.attention.output.dense.weight', 'roberta.pooler.dense.weight', 'roberta.encoder.layer.9.attention.output.LayerNorm.bias', 'roberta.encoder.layer.9.output.LayerNorm.weight', 'roberta.encoder.layer.8.output.dense.weight', 'roberta.encoder.layer.1.attention.output.dense.bias', 'roberta.encoder.layer.1.output.dense.bias', 'roberta.encoder.layer.10.attention.self.value.bias', 'roberta.encoder.layer.10.output.LayerNorm.weight', 'roberta.encoder.layer.11.attention.self.value.weight', 'roberta.encoder.layer.11.intermediate.dense.weight', 'roberta.encoder.layer.11.attention.self.value.bias', 'roberta.encoder.layer.8.attention.self.query.bias', 'roberta.encoder.layer.6.attention.output.LayerNorm.bias', 'roberta.encoder.layer.10.attention.output.dense.bias', 'roberta.encoder.layer.1.attention.self.key.weight', 'roberta.encoder.layer.8.attention.output.LayerNorm.bias', 'roberta.encoder.layer.5.attention.output.dense.bias', 'roberta.pooler.dense.bias', 'roberta.encoder.layer.6.attention.self.value.weight', 'roberta.encoder.layer.9.attention.self.query.bias', 'roberta.encoder.layer.5.intermediate.dense.weight', 'roberta.encoder.layer.3.attention.self.key.weight', 'roberta.encoder.layer.1.output.LayerNorm.bias', 'roberta.encoder.layer.3.attention.self.key.bias', 'roberta.encoder.layer.0.attention.self.key.weight', 'roberta.encoder.layer.3.attention.output.dense.weight', 'roberta.encoder.layer.4.intermediate.dense.bias', 'roberta.encoder.layer.11.attention.output.LayerNorm.weight', 'roberta.encoder.layer.9.attention.self.key.weight', 'roberta.encoder.layer.11.attention.self.query.weight', 'roberta.encoder.layer.6.output.LayerNorm.weight', 'roberta.encoder.layer.0.attention.output.dense.bias', 'roberta.encoder.layer.10.attention.self.key.bias', 'roberta.encoder.layer.7.intermediate.dense.weight', 'roberta.encoder.layer.1.intermediate.dense.bias', 'roberta.encoder.layer.3.attention.self.value.weight', 'roberta.encoder.layer.1.attention.self.key.bias', 'lm_head.layer_norm.bias', 'roberta.encoder.layer.4.output.dense.bias', 'roberta.encoder.layer.7.output.dense.bias', 'roberta.encoder.layer.6.attention.output.dense.bias', 'roberta.encoder.layer.5.output.dense.weight', 'roberta.encoder.layer.7.attention.output.LayerNorm.weight', 'roberta.encoder.layer.5.intermediate.dense.bias', 'roberta.encoder.layer.0.attention.self.query.weight', 'roberta.encoder.layer.10.output.dense.bias', 'roberta.encoder.layer.9.output.LayerNorm.bias', 'lm_head.dense.weight', 'roberta.encoder.layer.1.output.dense.weight', 'roberta.encoder.layer.6.output.dense.weight', 'roberta.encoder.layer.6.output.dense.bias', 'roberta.encoder.layer.11.attention.output.dense.bias', 'roberta.encoder.layer.3.attention.output.dense.bias', 'roberta.encoder.layer.10.output.dense.weight', 'roberta.encoder.layer.11.output.LayerNorm.weight', 'roberta.encoder.layer.8.attention.output.dense.bias', 'roberta.encoder.layer.3.output.LayerNorm.bias', 'roberta.encoder.layer.6.attention.self.query.weight', 'roberta.encoder.layer.5.attention.self.value.weight', 'roberta.encoder.layer.11.output.LayerNorm.bias', 'roberta.encoder.layer.0.intermediate.dense.bias', 'roberta.encoder.layer.2.attention.self.value.weight', 'roberta.encoder.layer.0.attention.output.LayerNorm.weight', 'roberta.encoder.layer.5.attention.output.LayerNorm.weight', 'roberta.encoder.layer.2.attention.self.key.bias', 'roberta.encoder.layer.0.intermediate.dense.weight', 'roberta.encoder.layer.11.attention.self.key.bias', 'lm_head.dense.bias', 'roberta.encoder.layer.4.attention.self.value.weight', 'roberta.encoder.layer.5.attention.self.query.weight', 'roberta.encoder.layer.4.attention.self.value.bias', 'roberta.encoder.layer.7.attention.output.dense.weight', 'roberta.encoder.layer.2.attention.self.value.bias', 'roberta.encoder.layer.8.output.LayerNorm.weight', 'roberta.encoder.layer.4.attention.output.LayerNorm.weight', 'roberta.encoder.layer.2.attention.output.dense.weight', 'roberta.encoder.layer.2.attention.output.LayerNorm.weight', 'roberta.encoder.layer.2.attention.self.query.weight', 'roberta.encoder.layer.8.attention.self.key.bias', 'roberta.encoder.layer.8.attention.self.query.weight', 'roberta.encoder.layer.9.intermediate.dense.weight', 'roberta.encoder.layer.7.attention.output.LayerNorm.bias', 'roberta.encoder.layer.7.output.dense.weight', 'roberta.encoder.layer.9.attention.output.LayerNorm.weight', 'roberta.encoder.layer.0.output.dense.weight', 'roberta.encoder.layer.8.output.LayerNorm.bias', 'roberta.encoder.layer.11.attention.self.query.bias', 'roberta.encoder.layer.3.attention.self.query.weight', 'roberta.encoder.layer.8.attention.self.value.bias', 'roberta.encoder.layer.6.attention.output.dense.weight', 'roberta.encoder.layer.10.attention.self.value.weight', 'roberta.encoder.layer.4.output.dense.weight', 'roberta.encoder.layer.1.attention.self.query.bias', 'roberta.encoder.layer.11.output.dense.bias', 'roberta.encoder.layer.0.attention.output.dense.weight', 'roberta.encoder.layer.8.attention.self.key.weight', 'roberta.encoder.layer.5.attention.output.dense.weight', 'roberta.encoder.layer.1.intermediate.dense.weight', 'roberta.encoder.layer.9.attention.output.dense.weight', 'roberta.encoder.layer.6.output.LayerNorm.bias', 'roberta.encoder.layer.11.attention.output.dense.weight', 'roberta.encoder.layer.4.attention.output.dense.bias', 'roberta.encoder.layer.4.attention.self.key.weight', 'roberta.encoder.layer.5.output.LayerNorm.weight', 'roberta.encoder.layer.7.output.LayerNorm.bias', 'roberta.encoder.layer.5.attention.self.key.weight', 'lm_head.decoder.weight', 'roberta.encoder.layer.10.attention.output.LayerNorm.bias', 'roberta.encoder.layer.1.attention.self.value.bias', 'roberta.encoder.layer.8.intermediate.dense.weight', 'roberta.encoder.layer.10.attention.self.query.bias', 'roberta.encoder.layer.2.attention.output.dense.bias', 'roberta.encoder.layer.2.output.LayerNorm.bias', 'roberta.encoder.layer.5.attention.self.query.bias', 'roberta.encoder.layer.7.attention.self.value.bias', 'roberta.encoder.layer.4.intermediate.dense.weight', 'roberta.encoder.layer.6.attention.self.key.weight', 'roberta.encoder.layer.4.attention.self.query.bias', 'roberta.encoder.layer.6.intermediate.dense.weight', 'roberta.encoder.layer.7.attention.self.value.weight', 'roberta.encoder.layer.1.attention.self.query.weight', 'roberta.encoder.layer.6.attention.self.value.bias', 'roberta.encoder.layer.10.intermediate.dense.weight', 'roberta.encoder.layer.2.attention.self.key.weight', 'roberta.encoder.layer.3.output.dense.bias', 'roberta.encoder.layer.6.attention.output.LayerNorm.weight', 'roberta.encoder.layer.5.output.LayerNorm.bias', 'roberta.encoder.layer.7.attention.self.key.bias', 'roberta.encoder.layer.7.attention.self.query.bias', 'roberta.encoder.layer.10.attention.self.query.weight', 'roberta.encoder.layer.2.intermediate.dense.bias', 'roberta.encoder.layer.9.attention.self.query.weight', 'roberta.encoder.layer.6.attention.self.query.bias', 'roberta.encoder.layer.9.attention.self.key.bias', 'roberta.encoder.layer.1.attention.output.LayerNorm.bias', 'roberta.encoder.layer.11.intermediate.dense.bias', 'roberta.embeddings.word_embeddings.weight', 'roberta.encoder.layer.2.attention.self.query.bias', 'roberta.encoder.layer.9.output.dense.bias', 'roberta.encoder.layer.2.attention.output.LayerNorm.bias', 'roberta.encoder.layer.4.attention.output.dense.weight', 'roberta.encoder.layer.3.attention.self.value.bias', 'roberta.encoder.layer.0.attention.self.value.bias', 'roberta.encoder.layer.0.output.dense.bias', 'roberta.encoder.layer.3.attention.output.LayerNorm.bias', 'roberta.encoder.layer.3.attention.self.query.bias', 'roberta.encoder.layer.3.output.dense.weight', 'lm_head.bias', 'roberta.encoder.layer.0.output.LayerNorm.bias', 'roberta.embeddings.LayerNorm.weight', 'roberta.encoder.layer.8.output.dense.bias', 'roberta.encoder.layer.8.attention.output.LayerNorm.weight', 'roberta.encoder.layer.4.output.LayerNorm.bias', 'roberta.encoder.layer.9.attention.self.value.weight', 'roberta.encoder.layer.7.attention.self.key.weight', 'roberta.encoder.layer.7.output.LayerNorm.weight', 'roberta.encoder.layer.1.attention.output.LayerNorm.weight', 'roberta.encoder.layer.3.intermediate.dense.bias', 'roberta.encoder.layer.5.attention.self.key.bias', 'roberta.encoder.layer.5.output.dense.bias', 'roberta.encoder.layer.4.output.LayerNorm.weight']\n",
            "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at camembert-base and are newly initialized: ['encoder.layer.6.attention.self.query.weight', 'encoder.layer.0.attention.self.key.weight', 'encoder.layer.2.attention.self.value.weight', 'encoder.layer.6.attention.output.dense.weight', 'encoder.layer.7.attention.self.key.bias', 'encoder.layer.8.output.LayerNorm.weight', 'encoder.layer.3.attention.self.value.bias', 'encoder.layer.10.output.dense.weight', 'encoder.layer.11.intermediate.dense.bias', 'encoder.layer.7.output.dense.weight', 'encoder.layer.2.attention.output.LayerNorm.weight', 'encoder.layer.11.attention.self.value.weight', 'encoder.layer.9.attention.self.key.weight', 'encoder.layer.4.intermediate.dense.weight', 'encoder.layer.5.attention.output.dense.bias', 'encoder.layer.10.attention.self.key.weight', 'encoder.layer.8.attention.self.key.weight', 'encoder.layer.8.attention.self.key.bias', 'encoder.layer.10.intermediate.dense.weight', 'pooler.dense.weight', 'encoder.layer.7.intermediate.dense.bias', 'encoder.layer.4.output.LayerNorm.bias', 'encoder.layer.10.attention.self.key.bias', 'encoder.layer.0.attention.output.dense.bias', 'encoder.layer.6.intermediate.dense.bias', 'encoder.layer.11.attention.self.query.bias', 'encoder.layer.6.attention.output.LayerNorm.bias', 'encoder.layer.0.attention.self.query.weight', 'encoder.layer.2.attention.self.key.bias', 'encoder.layer.0.attention.self.value.weight', 'encoder.layer.1.attention.self.value.bias', 'encoder.layer.5.intermediate.dense.bias', 'encoder.layer.7.intermediate.dense.weight', 'encoder.layer.7.output.LayerNorm.bias', 'encoder.layer.7.attention.self.key.weight', 'encoder.layer.5.attention.self.value.weight', 'encoder.layer.0.intermediate.dense.bias', 'encoder.layer.1.output.LayerNorm.weight', 'encoder.layer.1.attention.self.key.weight', 'encoder.layer.2.attention.output.dense.bias', 'encoder.layer.3.attention.self.value.weight', 'encoder.layer.10.output.LayerNorm.bias', 'encoder.layer.6.output.dense.weight', 'encoder.layer.1.attention.self.query.weight', 'encoder.layer.4.intermediate.dense.bias', 'encoder.layer.0.attention.output.dense.weight', 'encoder.layer.4.attention.self.key.bias', 'encoder.layer.2.attention.output.LayerNorm.bias', 'encoder.layer.10.attention.self.query.weight', 'encoder.layer.5.attention.self.query.bias', 'encoder.layer.9.intermediate.dense.weight', 'encoder.layer.6.attention.self.key.bias', 'encoder.layer.0.output.LayerNorm.weight', 'encoder.layer.9.output.LayerNorm.weight', 'encoder.layer.2.attention.self.key.weight', 'encoder.layer.6.attention.output.LayerNorm.weight', 'encoder.layer.8.attention.output.LayerNorm.weight', 'encoder.layer.7.attention.self.value.bias', 'encoder.layer.2.attention.self.query.weight', 'encoder.layer.4.output.LayerNorm.weight', 'encoder.layer.1.intermediate.dense.bias', 'encoder.layer.7.attention.self.query.weight', 'encoder.layer.4.attention.self.key.weight', 'encoder.layer.7.output.dense.bias', 'encoder.layer.0.attention.self.key.bias', 'encoder.layer.11.attention.self.query.weight', 'encoder.layer.7.attention.self.value.weight', 'classifier.bias', 'encoder.layer.8.attention.output.dense.weight', 'encoder.layer.5.attention.self.value.bias', 'embeddings.position_embeddings.weight', 'encoder.layer.8.output.LayerNorm.bias', 'encoder.layer.9.attention.self.query.bias', 'encoder.layer.1.output.dense.bias', 'encoder.layer.8.attention.self.value.bias', 'encoder.layer.10.attention.self.query.bias', 'encoder.layer.0.output.dense.bias', 'encoder.layer.2.attention.self.value.bias', 'embeddings.LayerNorm.weight', 'encoder.layer.3.output.LayerNorm.weight', 'encoder.layer.5.output.dense.bias', 'encoder.layer.7.attention.output.dense.weight', 'encoder.layer.3.attention.self.query.weight', 'encoder.layer.4.output.dense.weight', 'encoder.layer.7.attention.output.LayerNorm.weight', 'encoder.layer.1.attention.self.value.weight', 'encoder.layer.4.attention.self.query.weight', 'encoder.layer.3.attention.output.dense.weight', 'encoder.layer.9.output.dense.weight', 'encoder.layer.10.output.dense.bias', 'encoder.layer.1.intermediate.dense.weight', 'encoder.layer.8.attention.self.query.bias', 'encoder.layer.5.attention.self.query.weight', 'encoder.layer.7.output.LayerNorm.weight', 'encoder.layer.3.attention.output.LayerNorm.weight', 'encoder.layer.11.attention.self.key.weight', 'encoder.layer.6.attention.self.query.bias', 'encoder.layer.8.intermediate.dense.bias', 'encoder.layer.11.attention.output.dense.bias', 'encoder.layer.3.attention.output.dense.bias', 'encoder.layer.0.attention.self.value.bias', 'encoder.layer.9.attention.self.value.bias', 'embeddings.LayerNorm.bias', 'encoder.layer.8.attention.self.value.weight', 'encoder.layer.9.output.LayerNorm.bias', 'encoder.layer.10.output.LayerNorm.weight', 'encoder.layer.5.attention.output.LayerNorm.weight', 'encoder.layer.2.intermediate.dense.weight', 'encoder.layer.7.attention.output.LayerNorm.bias', 'encoder.layer.5.attention.self.key.weight', 'encoder.layer.8.attention.output.LayerNorm.bias', 'encoder.layer.5.output.LayerNorm.bias', 'encoder.layer.11.attention.output.dense.weight', 'encoder.layer.9.intermediate.dense.bias', 'encoder.layer.2.attention.output.dense.weight', 'classifier.weight', 'encoder.layer.2.attention.self.query.bias', 'encoder.layer.7.attention.self.query.bias', 'encoder.layer.9.attention.self.value.weight', 'encoder.layer.0.attention.self.query.bias', 'encoder.layer.4.output.dense.bias', 'encoder.layer.4.attention.output.dense.bias', 'encoder.layer.10.attention.output.dense.bias', 'encoder.layer.0.output.LayerNorm.bias', 'encoder.layer.3.attention.output.LayerNorm.bias', 'encoder.layer.9.output.dense.bias', 'encoder.layer.1.attention.output.dense.weight', 'encoder.layer.1.output.LayerNorm.bias', 'encoder.layer.9.attention.output.LayerNorm.weight', 'encoder.layer.9.attention.output.LayerNorm.bias', 'encoder.layer.8.output.dense.weight', 'encoder.layer.1.attention.self.query.bias', 'encoder.layer.11.attention.self.value.bias', 'encoder.layer.4.attention.self.value.bias', 'encoder.layer.5.output.dense.weight', 'encoder.layer.10.intermediate.dense.bias', 'encoder.layer.5.attention.self.key.bias', 'encoder.layer.5.output.LayerNorm.weight', 'encoder.layer.2.intermediate.dense.bias', 'encoder.layer.2.output.LayerNorm.bias', 'encoder.layer.0.output.dense.weight', 'encoder.layer.5.intermediate.dense.weight', 'encoder.layer.11.attention.self.key.bias', 'encoder.layer.1.attention.self.key.bias', 'encoder.layer.11.output.LayerNorm.bias', 'encoder.layer.6.output.LayerNorm.weight', 'encoder.layer.3.attention.self.key.bias', 'encoder.layer.11.intermediate.dense.weight', 'encoder.layer.4.attention.output.LayerNorm.weight', 'encoder.layer.6.intermediate.dense.weight', 'encoder.layer.0.attention.output.LayerNorm.weight', 'encoder.layer.1.output.dense.weight', 'encoder.layer.6.attention.output.dense.bias', 'encoder.layer.3.intermediate.dense.bias', 'encoder.layer.5.attention.output.dense.weight', 'encoder.layer.3.attention.self.key.weight', 'encoder.layer.6.output.dense.bias', 'encoder.layer.11.output.LayerNorm.weight', 'encoder.layer.6.attention.self.value.bias', 'encoder.layer.1.attention.output.dense.bias', 'encoder.layer.3.output.dense.bias', 'encoder.layer.7.attention.output.dense.bias', 'embeddings.word_embeddings.weight', 'encoder.layer.9.attention.self.query.weight', 'encoder.layer.11.output.dense.weight', 'encoder.layer.4.attention.self.value.weight', 'encoder.layer.6.attention.self.value.weight', 'encoder.layer.10.attention.self.value.weight', 'encoder.layer.11.output.dense.bias', 'encoder.layer.6.attention.self.key.weight', 'encoder.layer.8.attention.self.query.weight', 'encoder.layer.0.intermediate.dense.weight', 'encoder.layer.10.attention.output.dense.weight', 'encoder.layer.8.output.dense.bias', 'encoder.layer.3.output.dense.weight', 'encoder.layer.4.attention.self.query.bias', 'encoder.layer.6.output.LayerNorm.bias', 'encoder.layer.9.attention.self.key.bias', 'encoder.layer.9.attention.output.dense.bias', 'encoder.layer.2.output.LayerNorm.weight', 'encoder.layer.9.attention.output.dense.weight', 'encoder.layer.10.attention.output.LayerNorm.bias', 'pooler.dense.bias', 'encoder.layer.5.attention.output.LayerNorm.bias', 'encoder.layer.11.attention.output.LayerNorm.weight', 'encoder.layer.0.attention.output.LayerNorm.bias', 'encoder.layer.2.output.dense.weight', 'encoder.layer.1.attention.output.LayerNorm.weight', 'encoder.layer.2.output.dense.bias', 'encoder.layer.3.intermediate.dense.weight', 'encoder.layer.4.attention.output.LayerNorm.bias', 'encoder.layer.4.attention.output.dense.weight', 'encoder.layer.11.attention.output.LayerNorm.bias', 'encoder.layer.10.attention.self.value.bias', 'embeddings.token_type_embeddings.weight', 'encoder.layer.3.attention.self.query.bias', 'encoder.layer.1.attention.output.LayerNorm.bias', 'encoder.layer.8.attention.output.dense.bias', 'encoder.layer.3.output.LayerNorm.bias', 'encoder.layer.10.attention.output.LayerNorm.weight', 'encoder.layer.8.intermediate.dense.weight']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BertForSequenceClassification(\n",
              "  (bert): BertModel(\n",
              "    (embeddings): BertEmbeddings(\n",
              "      (word_embeddings): Embedding(32005, 768, padding_idx=1)\n",
              "      (position_embeddings): Embedding(514, 768)\n",
              "      (token_type_embeddings): Embedding(1, 768)\n",
              "      (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "      (dropout): Dropout(p=0.1, inplace=False)\n",
              "    )\n",
              "    (encoder): BertEncoder(\n",
              "      (layer): ModuleList(\n",
              "        (0): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (intermediate_act_fn): GELUActivation()\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (1): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (intermediate_act_fn): GELUActivation()\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (2): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (intermediate_act_fn): GELUActivation()\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (3): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (intermediate_act_fn): GELUActivation()\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (4): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (intermediate_act_fn): GELUActivation()\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (5): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (intermediate_act_fn): GELUActivation()\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (6): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (intermediate_act_fn): GELUActivation()\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (7): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (intermediate_act_fn): GELUActivation()\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (8): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (intermediate_act_fn): GELUActivation()\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (9): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (intermediate_act_fn): GELUActivation()\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (10): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (intermediate_act_fn): GELUActivation()\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (11): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "            (intermediate_act_fn): GELUActivation()\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "    )\n",
              "    (pooler): BertPooler(\n",
              "      (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "      (activation): Tanh()\n",
              "    )\n",
              "  )\n",
              "  (dropout): Dropout(p=0.1, inplace=False)\n",
              "  (classifier): Linear(in_features=768, out_features=39, bias=True)\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e0Jv6c7-HHDW"
      },
      "source": [
        "Just for curiosity's sake, we can browse all of the model's parameters by name here.\n",
        "\n",
        "In the below cell, I've printed out the names and dimensions of the weights for:\n",
        "\n",
        "1. The embedding layer.\n",
        "2. The first of the twelve transformers.\n",
        "3. The output layer.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8PIiVlDYCtSq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aa0559fb-b517-4151-fcd2-c7381a980f4e"
      },
      "source": [
        "# Get all of the model's parameters as a list of tuples.\n",
        "params = list(model.named_parameters())\n",
        "\n",
        "print('The BERT model has {:} different named parameters.\\n'.format(len(params)))\n",
        "\n",
        "print('==== Embedding Layer ====\\n')\n",
        "\n",
        "for p in params[0:5]:\n",
        "    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))\n",
        "\n",
        "print('\\n==== First Transformer ====\\n')\n",
        "\n",
        "for p in params[5:21]:\n",
        "    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))\n",
        "\n",
        "print('\\n==== Output Layer ====\\n')\n",
        "\n",
        "for p in params[-4:]:\n",
        "    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The BERT model has 201 different named parameters.\n",
            "\n",
            "==== Embedding Layer ====\n",
            "\n",
            "bert.embeddings.word_embeddings.weight                  (32005, 768)\n",
            "bert.embeddings.position_embeddings.weight                (514, 768)\n",
            "bert.embeddings.token_type_embeddings.weight                (1, 768)\n",
            "bert.embeddings.LayerNorm.weight                              (768,)\n",
            "bert.embeddings.LayerNorm.bias                                (768,)\n",
            "\n",
            "==== First Transformer ====\n",
            "\n",
            "bert.encoder.layer.0.attention.self.query.weight          (768, 768)\n",
            "bert.encoder.layer.0.attention.self.query.bias                (768,)\n",
            "bert.encoder.layer.0.attention.self.key.weight            (768, 768)\n",
            "bert.encoder.layer.0.attention.self.key.bias                  (768,)\n",
            "bert.encoder.layer.0.attention.self.value.weight          (768, 768)\n",
            "bert.encoder.layer.0.attention.self.value.bias                (768,)\n",
            "bert.encoder.layer.0.attention.output.dense.weight        (768, 768)\n",
            "bert.encoder.layer.0.attention.output.dense.bias              (768,)\n",
            "bert.encoder.layer.0.attention.output.LayerNorm.weight        (768,)\n",
            "bert.encoder.layer.0.attention.output.LayerNorm.bias          (768,)\n",
            "bert.encoder.layer.0.intermediate.dense.weight           (3072, 768)\n",
            "bert.encoder.layer.0.intermediate.dense.bias                 (3072,)\n",
            "bert.encoder.layer.0.output.dense.weight                 (768, 3072)\n",
            "bert.encoder.layer.0.output.dense.bias                        (768,)\n",
            "bert.encoder.layer.0.output.LayerNorm.weight                  (768,)\n",
            "bert.encoder.layer.0.output.LayerNorm.bias                    (768,)\n",
            "\n",
            "==== Output Layer ====\n",
            "\n",
            "bert.pooler.dense.weight                                  (768, 768)\n",
            "bert.pooler.dense.bias                                        (768,)\n",
            "classifier.weight                                          (39, 768)\n",
            "classifier.bias                                                (39,)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8o-VEBobKwHk"
      },
      "source": [
        "Now that we have our model loaded we need to grab the training hyperparameters from within the stored model.\n",
        "\n",
        "For the purposes of fine-tuning, the authors recommend choosing from the following values (from Appendix A.3 of the [BERT paper](https://arxiv.org/pdf/1810.04805.pdf)):\n",
        "\n",
        ">- **Batch size:** 16, 32  \n",
        "- **Learning rate (Adam):** 5e-5, 3e-5, 2e-5  \n",
        "- **Number of epochs:** 2, 3, 4 \n",
        "\n",
        "We chose:\n",
        "* Batch size: 32 (set when creating our DataLoaders)\n",
        "* Learning rate: 2e-5\n",
        "* Epochs: 4 (we'll see that this is probably too many...)\n",
        "\n",
        "The epsilon parameter `eps = 1e-8` is \"a very small number to prevent any division by zero in the implementation\" (from [here](https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/)).\n",
        "\n",
        "You can find the creation of the AdamW optimizer in `run_glue.py` [here](https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/examples/run_glue.py#L109)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GLs72DuMODJO",
        "outputId": "52144c8b-4a0a-4c70-aa36-a6146316510e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Note: AdamW is a class from the huggingface library (as opposed to pytorch) \n",
        "# I believe the 'W' stands for 'Weight Decay fix\"\n",
        "optimizer = AdamW(model.parameters(),\n",
        "                  lr = 1e-5, # args.learning_rate - default is 5e-5, our notebook had 2e-5\n",
        "                  eps = 1e-8 # args.adam_epsilon  - default is 1e-8.\n",
        "                )\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
            "  warnings.warn(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-p0upAhhRiIx"
      },
      "source": [
        "from transformers import get_linear_schedule_with_warmup\n",
        "\n",
        "# Number of training epochs. The BERT authors recommend between 2 and 4. \n",
        "# We chose to run for 4, but we'll see later that this may be over-fitting the\n",
        "# training data.\n",
        "epochs = 4\n",
        "\n",
        "# Total number of training steps is [number of batches] x [number of epochs]. \n",
        "# (Note that this is not the same as the number of training samples).\n",
        "total_steps = len(train_dataloader) * epochs\n",
        "\n",
        "# Create the learning rate scheduler.\n",
        "scheduler = get_linear_schedule_with_warmup(optimizer, \n",
        "                                            num_warmup_steps = 0, # Default value in run_glue.py\n",
        "                                            num_training_steps = total_steps)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_QXZhFb4LnV5"
      },
      "source": [
        "Below is our training loop. There's a lot going on, but fundamentally for each pass in our loop we have a trianing phase and a validation phase. \n",
        "\n",
        "> *Thank you to [Stas Bekman](https://ca.linkedin.com/in/stasbekman) for contributing the insights and code for using validation loss to detect over-fitting!*\n",
        "\n",
        "**Training:**\n",
        "- Unpack our data inputs and labels\n",
        "- Load data onto the GPU for acceleration\n",
        "- Clear out the gradients calculated in the previous pass. \n",
        "    - In pytorch the gradients accumulate by default (useful for things like RNNs) unless you explicitly clear them out.\n",
        "- Forward pass (feed input data through the network)\n",
        "- Backward pass (backpropagation)\n",
        "- Tell the network to update parameters with optimizer.step()\n",
        "- Track variables for monitoring progress\n",
        "\n",
        "**Evalution:**\n",
        "- Unpack our data inputs and labels\n",
        "- Load data onto the GPU for acceleration\n",
        "- Forward pass (feed input data through the network)\n",
        "- Compute loss on our validation data and track variables for monitoring progress\n",
        "\n",
        "Pytorch hides all of the detailed calculations from us, but we've commented the code to point out which of the above steps are happening on each line. \n",
        "\n",
        "> *PyTorch also has some [beginner tutorials](https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py) which you may also find helpful.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pE5B99H5H2-W"
      },
      "source": [
        "Define a helper function for calculating accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9cQNvaZ9bnyy"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "# Function to calculate the accuracy of our predictions vs labels\n",
        "def flat_accuracy(preds, labels):\n",
        "    pred_flat = np.argmax(preds, axis=1).flatten()\n",
        "    labels_flat = labels.flatten()\n",
        "    return np.sum(pred_flat == labels_flat) / len(labels_flat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KNhRtWPXH9C3"
      },
      "source": [
        "Helper function for formatting elapsed times as `hh:mm:ss`\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gpt6tR83keZD"
      },
      "source": [
        "import time\n",
        "import datetime\n",
        "\n",
        "def format_time(elapsed):\n",
        "    '''\n",
        "    Takes a time in seconds and returns a string hh:mm:ss\n",
        "    '''\n",
        "    # Round to the nearest second.\n",
        "    elapsed_rounded = int(round((elapsed)))\n",
        "    \n",
        "    # Format as hh:mm:ss\n",
        "    return str(datetime.timedelta(seconds=elapsed_rounded))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cfNIhN19te3N"
      },
      "source": [
        "We're ready to kick off the training!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6J-FYdx6nFE_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6b79c362-d8f8-4a97-a058-b153292eec88"
      },
      "source": [
        "import random\n",
        "import numpy as np\n",
        "\n",
        "# This training code is based on the `run_glue.py` script here:\n",
        "# https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/examples/run_glue.py#L128\n",
        "\n",
        "# Set the seed value all over the place to make this reproducible.\n",
        "seed_val = 42\n",
        "\n",
        "random.seed(seed_val)\n",
        "np.random.seed(seed_val)\n",
        "torch.manual_seed(seed_val)\n",
        "torch.cuda.manual_seed_all(seed_val)\n",
        "\n",
        "# We'll store a number of quantities such as training and validation loss, \n",
        "# validation accuracy, and timings.\n",
        "training_stats = []\n",
        "\n",
        "# Measure the total training time for the whole run.\n",
        "total_t0 = time.time()\n",
        "\n",
        "# For each epoch...\n",
        "for epoch_i in range(0, epochs):\n",
        "    \n",
        "    # ========================================\n",
        "    #               Training\n",
        "    # ========================================\n",
        "    \n",
        "    # Perform one full pass over the training set.\n",
        "\n",
        "    print(\"\")\n",
        "    print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, epochs))\n",
        "    print('Training...')\n",
        "\n",
        "    # Measure how long the training epoch takes.\n",
        "    t0 = time.time()\n",
        "\n",
        "    # Reset the total loss for this epoch.\n",
        "    total_train_loss = 0\n",
        "\n",
        "    # Put the model into training mode. Don't be mislead--the call to \n",
        "    # `train` just changes the *mode*, it doesn't *perform* the training.\n",
        "    # `dropout` and `batchnorm` layers behave differently during training\n",
        "    # vs. test (source: https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch)\n",
        "    model.train()\n",
        "\n",
        "    # For each batch of training data...\n",
        "    for step, batch in enumerate(train_dataloader):\n",
        "\n",
        "        # Progress update every 40 batches.\n",
        "        if step % 10 == 0 and not step == 0:\n",
        "            # Calculate elapsed time in minutes.\n",
        "            elapsed = format_time(time.time() - t0)\n",
        "            \n",
        "            # Report progress.\n",
        "            print('  Batch {:>5,}  of  {:>5,}.    Elapsed: {:}.'.format(step, len(train_dataloader), elapsed))\n",
        "\n",
        "        # Unpack this training batch from our dataloader. \n",
        "        #\n",
        "        # As we unpack the batch, we'll also copy each tensor to the GPU using the \n",
        "        # `to` method.\n",
        "        #\n",
        "        # `batch` contains three pytorch tensors:\n",
        "        #   [0]: input ids \n",
        "        #   [1]: attention masks\n",
        "        #   [2]: labels \n",
        "        b_input_ids = batch[0].to(device)\n",
        "        b_input_mask = batch[1].to(device)\n",
        "        b_labels = batch[2].to(device)\n",
        "\n",
        "        # Always clear any previously calculated gradients before performing a\n",
        "        # backward pass. PyTorch doesn't do this automatically because \n",
        "        # accumulating the gradients is \"convenient while training RNNs\". \n",
        "        # (source: https://stackoverflow.com/questions/48001598/why-do-we-need-to-call-zero-grad-in-pytorch)\n",
        "        model.zero_grad()        \n",
        "\n",
        "        # Perform a forward pass (evaluate the model on this training batch).\n",
        "        # In PyTorch, calling `model` will in turn call the model's `forward` \n",
        "        # function and pass down the arguments. The `forward` function is \n",
        "        # documented here: \n",
        "        # https://huggingface.co/transformers/model_doc/bert.html#bertforsequenceclassification\n",
        "        # The results are returned in a results object, documented here:\n",
        "        # https://huggingface.co/transformers/main_classes/output.html#transformers.modeling_outputs.SequenceClassifierOutput\n",
        "        # Specifically, we'll get the loss (because we provided labels) and the\n",
        "        # \"logits\"--the model outputs prior to activation.\n",
        "        result = model(b_input_ids, \n",
        "                       token_type_ids=None, \n",
        "                       attention_mask=b_input_mask, \n",
        "                       labels=b_labels,\n",
        "                       return_dict=True)\n",
        "\n",
        "        loss = result.loss\n",
        "        logits = result.logits\n",
        "\n",
        "        # Accumulate the training loss over all of the batches so that we can\n",
        "        # calculate the average loss at the end. `loss` is a Tensor containing a\n",
        "        # single value; the `.item()` function just returns the Python value \n",
        "        # from the tensor.\n",
        "        total_train_loss += loss.item()\n",
        "\n",
        "        # Perform a backward pass to calculate the gradients.\n",
        "        loss.backward()\n",
        "\n",
        "        # Clip the norm of the gradients to 1.0.\n",
        "        # This is to help prevent the \"exploding gradients\" problem.\n",
        "        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n",
        "\n",
        "        # Update parameters and take a step using the computed gradient.\n",
        "        # The optimizer dictates the \"update rule\"--how the parameters are\n",
        "        # modified based on their gradients, the learning rate, etc.\n",
        "        optimizer.step()\n",
        "\n",
        "        # Update the learning rate.\n",
        "        scheduler.step()\n",
        "\n",
        "    # Calculate the average loss over all of the batches.\n",
        "    avg_train_loss = total_train_loss / len(train_dataloader)            \n",
        "    \n",
        "    # Measure how long this epoch took.\n",
        "    training_time = format_time(time.time() - t0)\n",
        "\n",
        "    print(\"\")\n",
        "    print(\"  Average training loss: {0:.2f}\".format(avg_train_loss))\n",
        "    print(\"  Training epoch took: {:}\".format(training_time))\n",
        "        \n",
        "    # ========================================\n",
        "    #               Validation\n",
        "    # ========================================\n",
        "    # After the completion of each training epoch, measure our performance on\n",
        "    # our validation set.\n",
        "\n",
        "    print(\"\")\n",
        "    print(\"Running Validation...\")\n",
        "\n",
        "    t0 = time.time()\n",
        "\n",
        "    # Put the model in evaluation mode--the dropout layers behave differently\n",
        "    # during evaluation.\n",
        "    model.eval()\n",
        "\n",
        "    # Tracking variables \n",
        "    total_eval_accuracy = 0\n",
        "    total_eval_loss = 0\n",
        "    nb_eval_steps = 0\n",
        "\n",
        "    # Evaluate data for one epoch\n",
        "    for batch in validation_dataloader:\n",
        "        \n",
        "        # Unpack this training batch from our dataloader. \n",
        "        #\n",
        "        # As we unpack the batch, we'll also copy each tensor to the GPU using \n",
        "        # the `to` method.\n",
        "        #\n",
        "        # `batch` contains three pytorch tensors:\n",
        "        #   [0]: input ids \n",
        "        #   [1]: attention masks\n",
        "        #   [2]: labels \n",
        "        b_input_ids = batch[0].to(device)\n",
        "        b_input_mask = batch[1].to(device)\n",
        "        b_labels = batch[2].to(device)\n",
        "        \n",
        "        # Tell pytorch not to bother with constructing the compute graph during\n",
        "        # the forward pass, since this is only needed for backprop (training).\n",
        "        with torch.no_grad():        \n",
        "\n",
        "            # Forward pass, calculate logit predictions.\n",
        "            # token_type_ids is the same as the \"segment ids\", which \n",
        "            # differentiates sentence 1 and 2 in 2-sentence tasks.\n",
        "            result = model(b_input_ids, \n",
        "                           token_type_ids=None, \n",
        "                           attention_mask=b_input_mask,\n",
        "                           labels=b_labels,\n",
        "                           return_dict=True)\n",
        "\n",
        "        # Get the loss and \"logits\" output by the model. The \"logits\" are the \n",
        "        # output values prior to applying an activation function like the \n",
        "        # softmax.\n",
        "        loss = result.loss\n",
        "        logits = result.logits\n",
        "            \n",
        "        # Accumulate the validation loss.\n",
        "        total_eval_loss += loss.item()\n",
        "\n",
        "        # Move logits and labels to CPU\n",
        "        logits = logits.detach().cpu().numpy()\n",
        "        label_ids = b_labels.to('cpu').numpy()\n",
        "\n",
        "        # Calculate the accuracy for this batch of test sentences, and\n",
        "        # accumulate it over all batches.\n",
        "        total_eval_accuracy += flat_accuracy(logits, label_ids)\n",
        "        \n",
        "\n",
        "    # Report the final accuracy for this validation run.\n",
        "    avg_val_accuracy = total_eval_accuracy / len(validation_dataloader)\n",
        "    print(\"  Accuracy: {0:.2f}\".format(avg_val_accuracy))\n",
        "\n",
        "    # Calculate the average loss over all of the batches.\n",
        "    avg_val_loss = total_eval_loss / len(validation_dataloader)\n",
        "    \n",
        "    # Measure how long the validation run took.\n",
        "    validation_time = format_time(time.time() - t0)\n",
        "    \n",
        "    print(\"  Validation Loss: {0:.2f}\".format(avg_val_loss))\n",
        "    print(\"  Validation took: {:}\".format(validation_time))\n",
        "\n",
        "    # Record all statistics from this epoch.\n",
        "    training_stats.append(\n",
        "        {\n",
        "            'epoch': epoch_i + 1,\n",
        "            'Training Loss': avg_train_loss,\n",
        "            'Valid. Loss': avg_val_loss,\n",
        "            'Valid. Accur.': avg_val_accuracy,\n",
        "            'Training Time': training_time,\n",
        "            'Validation Time': validation_time\n",
        "        }\n",
        "    )\n",
        "\n",
        "print(\"\")\n",
        "print(\"Training complete!\")\n",
        "\n",
        "print(\"Total training took {:} (h:mm:ss)\".format(format_time(time.time()-total_t0)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "======== Epoch 1 / 4 ========\n",
            "Training...\n",
            "  Batch    10  of    124.    Elapsed: 0:00:01.\n",
            "  Batch    20  of    124.    Elapsed: 0:00:02.\n",
            "  Batch    30  of    124.    Elapsed: 0:00:02.\n",
            "  Batch    40  of    124.    Elapsed: 0:00:03.\n",
            "  Batch    50  of    124.    Elapsed: 0:00:04.\n",
            "  Batch    60  of    124.    Elapsed: 0:00:05.\n",
            "  Batch    70  of    124.    Elapsed: 0:00:05.\n",
            "  Batch    80  of    124.    Elapsed: 0:00:06.\n",
            "  Batch    90  of    124.    Elapsed: 0:00:07.\n",
            "  Batch   100  of    124.    Elapsed: 0:00:08.\n",
            "  Batch   110  of    124.    Elapsed: 0:00:08.\n",
            "  Batch   120  of    124.    Elapsed: 0:00:09.\n",
            "\n",
            "  Average training loss: 3.47\n",
            "  Training epoch took: 0:00:10\n",
            "\n",
            "Running Validation...\n",
            "  Accuracy: 0.18\n",
            "  Validation Loss: 3.26\n",
            "  Validation took: 0:00:00\n",
            "\n",
            "======== Epoch 2 / 4 ========\n",
            "Training...\n",
            "  Batch    10  of    124.    Elapsed: 0:00:01.\n",
            "  Batch    20  of    124.    Elapsed: 0:00:02.\n",
            "  Batch    30  of    124.    Elapsed: 0:00:02.\n",
            "  Batch    40  of    124.    Elapsed: 0:00:03.\n",
            "  Batch    50  of    124.    Elapsed: 0:00:04.\n",
            "  Batch    60  of    124.    Elapsed: 0:00:05.\n",
            "  Batch    70  of    124.    Elapsed: 0:00:05.\n",
            "  Batch    80  of    124.    Elapsed: 0:00:06.\n",
            "  Batch    90  of    124.    Elapsed: 0:00:07.\n",
            "  Batch   100  of    124.    Elapsed: 0:00:08.\n",
            "  Batch   110  of    124.    Elapsed: 0:00:08.\n",
            "  Batch   120  of    124.    Elapsed: 0:00:09.\n",
            "\n",
            "  Average training loss: 3.06\n",
            "  Training epoch took: 0:00:10\n",
            "\n",
            "Running Validation...\n",
            "  Accuracy: 0.19\n",
            "  Validation Loss: 2.78\n",
            "  Validation took: 0:00:00\n",
            "\n",
            "======== Epoch 3 / 4 ========\n",
            "Training...\n",
            "  Batch    10  of    124.    Elapsed: 0:00:01.\n",
            "  Batch    20  of    124.    Elapsed: 0:00:02.\n",
            "  Batch    30  of    124.    Elapsed: 0:00:02.\n",
            "  Batch    40  of    124.    Elapsed: 0:00:03.\n",
            "  Batch    50  of    124.    Elapsed: 0:00:04.\n",
            "  Batch    60  of    124.    Elapsed: 0:00:05.\n",
            "  Batch    70  of    124.    Elapsed: 0:00:05.\n",
            "  Batch    80  of    124.    Elapsed: 0:00:06.\n",
            "  Batch    90  of    124.    Elapsed: 0:00:07.\n",
            "  Batch   100  of    124.    Elapsed: 0:00:08.\n",
            "  Batch   110  of    124.    Elapsed: 0:00:08.\n",
            "  Batch   120  of    124.    Elapsed: 0:00:09.\n",
            "\n",
            "  Average training loss: 2.66\n",
            "  Training epoch took: 0:00:10\n",
            "\n",
            "Running Validation...\n",
            "  Accuracy: 0.25\n",
            "  Validation Loss: 2.61\n",
            "  Validation took: 0:00:00\n",
            "\n",
            "======== Epoch 4 / 4 ========\n",
            "Training...\n",
            "  Batch    10  of    124.    Elapsed: 0:00:01.\n",
            "  Batch    20  of    124.    Elapsed: 0:00:02.\n",
            "  Batch    30  of    124.    Elapsed: 0:00:02.\n",
            "  Batch    40  of    124.    Elapsed: 0:00:03.\n",
            "  Batch    50  of    124.    Elapsed: 0:00:04.\n",
            "  Batch    60  of    124.    Elapsed: 0:00:05.\n",
            "  Batch    70  of    124.    Elapsed: 0:00:05.\n",
            "  Batch    80  of    124.    Elapsed: 0:00:06.\n",
            "  Batch    90  of    124.    Elapsed: 0:00:07.\n",
            "  Batch   100  of    124.    Elapsed: 0:00:08.\n",
            "  Batch   110  of    124.    Elapsed: 0:00:08.\n",
            "  Batch   120  of    124.    Elapsed: 0:00:09.\n",
            "\n",
            "  Average training loss: 2.48\n",
            "  Training epoch took: 0:00:10\n",
            "\n",
            "Running Validation...\n",
            "  Accuracy: 0.29\n",
            "  Validation Loss: 2.51\n",
            "  Validation took: 0:00:00\n",
            "\n",
            "Training complete!\n",
            "Total training took 0:00:39 (h:mm:ss)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VQTvJ1vRP7u4"
      },
      "source": [
        "Let's view the summary of the training process."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6O_NbXFGMukX",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "b33be8c0-c36e-4638-b1e8-6310bbc6e201"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Display floats with two decimal places.\n",
        "pd.set_option('precision', 2)\n",
        "\n",
        "# Create a DataFrame from our training statistics.\n",
        "df_stats = pd.DataFrame(data=training_stats)\n",
        "\n",
        "# Use the 'epoch' as the row index.\n",
        "df_stats = df_stats.set_index('epoch')\n",
        "\n",
        "# A hack to force the column headers to wrap.\n",
        "#df = df.style.set_table_styles([dict(selector=\"th\",props=[('max-width', '70px')])])\n",
        "\n",
        "# Display the table.\n",
        "df_stats"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "       Training Loss  Valid. Loss  Valid. Accur. Training Time Validation Time\n",
              "epoch                                                                         \n",
              "1               3.47         3.26           0.18       0:00:10         0:00:00\n",
              "2               3.06         2.78           0.19       0:00:10         0:00:00\n",
              "3               2.66         2.61           0.25       0:00:10         0:00:00\n",
              "4               2.48         2.51           0.29       0:00:10         0:00:00"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-26eb3434-7f76-4b97-acf1-1362053372f3\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Valid. Loss</th>\n",
              "      <th>Valid. Accur.</th>\n",
              "      <th>Training Time</th>\n",
              "      <th>Validation Time</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>epoch</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>3.47</td>\n",
              "      <td>3.26</td>\n",
              "      <td>0.18</td>\n",
              "      <td>0:00:10</td>\n",
              "      <td>0:00:00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3.06</td>\n",
              "      <td>2.78</td>\n",
              "      <td>0.19</td>\n",
              "      <td>0:00:10</td>\n",
              "      <td>0:00:00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2.66</td>\n",
              "      <td>2.61</td>\n",
              "      <td>0.25</td>\n",
              "      <td>0:00:10</td>\n",
              "      <td>0:00:00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2.48</td>\n",
              "      <td>2.51</td>\n",
              "      <td>0.29</td>\n",
              "      <td>0:00:10</td>\n",
              "      <td>0:00:00</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-26eb3434-7f76-4b97-acf1-1362053372f3')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-26eb3434-7f76-4b97-acf1-1362053372f3 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-26eb3434-7f76-4b97-acf1-1362053372f3');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1-G03mmwH3aI"
      },
      "source": [
        "Notice that, while the the training loss is going down with each epoch, the validation loss is increasing! This suggests that we are training our model too long, and it's over-fitting on the training data. \n",
        "\n",
        "(For reference, we are using 7,695 training samples and 856 validation samples).\n",
        "\n",
        "Validation Loss is a more precise measure than accuracy, because with accuracy we don't care about the exact output value, but just which side of a threshold it falls on. \n",
        "\n",
        "If we are predicting the correct answer, but with less confidence, then validation loss will catch this, while accuracy will not."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "68xreA9JAmG5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 427
        },
        "outputId": "0f0da255-0c13-4a52-cddc-da5ac57bf712"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "# % matplotlib inline\n",
        "\n",
        "import seaborn as sns\n",
        "\n",
        "# Use plot styling from seaborn.\n",
        "sns.set(style='darkgrid')\n",
        "\n",
        "# Increase the plot size and font size.\n",
        "sns.set(font_scale=1.5)\n",
        "plt.rcParams[\"figure.figsize\"] = (12,6)\n",
        "\n",
        "# Plot the learning curve.\n",
        "plt.plot(df_stats['Training Loss'], 'b-o', label=\"Training\")\n",
        "plt.plot(df_stats['Valid. Loss'], 'g-o', label=\"Validation\")\n",
        "\n",
        "# Label the plot.\n",
        "plt.title(\"Training & Validation Loss\")\n",
        "plt.xlabel(\"Epoch\")\n",
        "plt.ylabel(\"Loss\")\n",
        "plt.legend()\n",
        "plt.xticks([1, 2, 3, 4])\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 864x432 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuUAAAGaCAYAAACopj13AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd1yT5/o/8E8SSNgbZIuMAAIC7lkFAXEvFNxVW7V19LTfttbWtrbnZ0+PbbXVtra1w7oH4Gpx4qoTcStTnEyRDbKT3x8qxwhqQCABP+/X67xehzvPcz8Xkau5cufK/QjkcrkcRERERESkMkJVB0BERERE9LJjUU5EREREpGIsyomIiIiIVIxFORERERGRirEoJyIiIiJSMRblREREREQqxqKciFqt1NRUuLq6YsWKFQ2e44MPPoCrq2sjRtV6Pe35dnV1xQcffKDUHCtWrICrqytSU1MbPb7IyEi4urri9OnTjT43EdGL0lB1AET08qhPcRsdHQ1bW9smjKbluX//Pn766SdERUXh7t27MDExQadOnfDmm2/CyclJqTnmzZuHvXv3Yvv27XB3d6/zGLlcjv79+6OwsBDHjh2DlpZWY/4aTer06dOIiYnBlClTYGBgoOpwaklNTUX//v0xYcIEfPLJJ6oOh4jUCItyImo2S5YsUfj57Nmz2Lx5M0JDQ9GpUyeFx0xMTF74ejY2Nrh06RJEIlGD5/j3v/+Nzz777IVjaQwLFy7E33//jSFDhqBr167Izs7GwYMHcfHiRaWL8pCQEOzduxcRERFYuHBhncecOnUKaWlpCA0NbZSC/NKlSxAKm+eD2ZiYGHz//fcYOXJkraJ8+PDhGDx4MDQ1NZslFiKi+mBRTkTNZvjw4Qo/V1dXY/PmzfDx8an12JOKi4uhp6dXr+sJBAJIJJJ6x/k4dSngSktLsWfPHvTu3RvffPNNzficOXNQUVGh9Dy9e/eGlZUVdu3ahffffx9isbjWMZGRkQAeFPCN4UX/DRqLSCR6oTdoRERNiT3lRKR2/P39MWnSJMTFxWH69Ono1KkThg0bBuBBcb5s2TKMGTMG3bp1g6enJwIDA/H111+jtLRUYZ66epwfHzt06BBGjx4NLy8v9O7dG//9739RVVWlMEddPeWPxoqKivDpp5+iR48e8PLyQlhYGC5evFjr98nLy8OCBQvQrVs3+Pr6YvLkyYiLi8OkSZPg7++v1HMiEAggEAjqfJNQV2H9NEKhECNHjkR+fj4OHjxY6/Hi4mLs27cPUqkUHTp0qNfz/TR19ZTLZDL8/PPP8Pf3h5eXF4YMGYKdO3fWeX5KSgoWLVqEwYMHw9fXF97e3hg1ahS2bt2qcNwHH3yA77//HgDQv39/uLq6Kvz7P62nPDc3F5999hn69u0LT09P9O3bF5999hny8vIUjnt0/smTJ/Hbb78hICAAnp6eGDBgALZt26bUc1EfCQkJmD17Nrp16wYvLy8MGjQIq1atQnV1tcJxGRkZWLBgAfz8/ODp6YkePXogLCxMISaZTIbVq1dj6NCh8PX1RceOHTFgwAB8+OGHqKysbPTYiaj+uFJORGopPT0dU6ZMQXBwMIKCgnD//n0AQFZWFsLDwxEUFIQhQ4ZAQ0MDMTEx+PXXXxEfH4/ffvtNqfmPHDmCDRs2ICwsDKNHj0Z0dDR+//13GBoaYtasWUrNMX36dJiYmGD27NnIz8/HH3/8gRkzZiA6OrpmVb+iogJTp05FfHw8Ro0aBS8vLyQmJmLq1KkwNDRU+vnQ0tLCiBEjEBERgb/++gtDhgxR+twnjRo1CitXrkRkZCSCg4MVHvv7779RVlaG0aNHA2i85/tJ//nPf7BmzRp06dIFr776KnJycvD555/Dzs6u1rExMTGIjY1Fv379YGtrW/OpwcKFC5Gbm4uZM2cCAEJDQ1FcXIz9+/djwYIFMDY2BvDs7zIUFRVh3LhxuHXrFkaPHo327dsjPj4eGzduxKlTp7B169Zan9AsW7YMZWVlCA0NhVgsxsaNG/HBBx/A3t6+VhtWQ12+fBmTJk2ChoYGJkyYADMzMxw6dAhff/01EhISaj4tqaqqwtSpU5GVlYXx48fDwcEBxcXFSExMRGxsLEaOHAkAWLlyJZYvXw4/Pz+EhYVBJBIhNTUVBw8eREVFhdp8IkT0UpMTEalIRESEXCqVyiMiIhTG/fz85FKpVL5ly5Za55SXl8srKipqjS9btkwulUrlFy9erBm7c+eOXCqVypcvX15rzNvbW37nzp2acZlMJh88eLC8V69eCvPOnz9fLpVK6xz79NNPFcajoqLkUqlUvnHjxpqxdevWyaVSqfzHH39UOPbRuJ+fX63fpS5FRUXy119/Xe7p6Slv3769/O+//1bqvKeZPHmy3N3dXZ6VlaUwPnbsWLmHh4c8JydHLpe/+PMtl8vlUqlUPn/+/JqfU1JS5K6urvLJkyfLq6qqasavXLkid3V1lUulUoV/m5KSklrXr66ulk+cOFHesWNHhfiWL19e6/xHHv29nTp1qmZs6dKlcqlUKl+3bp3CsY/+fZYtW1br/OHDh8vLy8trxjMzM+UeHh7yt99+u9Y1n/ToOfrss8+eeVxoaKjc3d1dHh8fXzMmk8nk8+bNk0ulUvmJEyfkcrlcHh8fL5dKpfJffvnlmfONGDFCPnDgwOfGR0Sqw/YVIlJLRkZGGDVqVK1xsVhcs6pXVVWFgoIC5ObmomfPngBQZ/tIXfr376+wu4tAIEC3bt2QnZ2NkpISpeZ49dVXFX7u3r07AODWrVs1Y4cOHYJIJMLkyZMVjh0zZgz09fWVuo5MJsNbb72FhIQE7N69G6+88greffdd7Nq1S+G4jz/+GB4eHkr1mIeEhKC6uhrbt2+vGUtJScGFCxfg7+9f80Xbxnq+HxcdHQ25XI6pU6cq9Hh7eHigV69etY7X0dGp+f/l5eXIy8tDfn4+evXqheLiYly/fr3eMTyyf/9+mJiYIDQ0VGE8NDQUJiYmOHDgQK1zxo8fr9Ay1KZNG7Rr1w43b95scByPy8nJwfnz5+Hv7w83N7eacYFAgDfeeKMmbgA1f0OnT59GTk7OU+fU09NDVlYWYmNjGyVGImp8bF8hIrVkZ2f31C/lrV+/Hps2bcK1a9cgk8kUHisoKFB6/icZGRkBAPLz86Grq1vvOR61S+Tn59eMpaamwsLCotZ8YrEYtra2KCwsfO51oqOjcezYMXz11VewtbXFd999hzlz5uD9999HVVVVTYtCYmIivLy8lOoxDwoKgoGBASIjIzFjxgwAQEREBADUtK480hjP9+Pu3LkDAHB0dKz1mJOTE44dO6YwVlJSgu+//x67d+9GRkZGrXOUeQ6fJjU1FZ6entDQUHw51NDQgIODA+Li4mqd87S/nbS0tAbH8WRMAODs7FzrMUdHRwiFwprn0MbGBrNmzcIvv/yC3r17w93dHd27d0dwcDA6dOhQc94777yD2bNnY8KECbCwsEDXrl3Rr18/DBgwoF7fSSCipsOinIjUkra2dp3jf/zxB7788kv07t0bkydPhoWFBTQ1NZGVlYUPPvgAcrlcqfmftQvHi86h7PnKevTFxC5dugB4UNB///33eOONN7BgwQJUVVXBzc0NFy9exOLFi5WaUyKRYMiQIdiwYQPOnTsHb29v7Ny5E5aWlujTp0/NcY31fL+I//u//8Phw4cxduxYdOnSBUZGRhCJRDhy5AhWr15d641CU2uu7R2V9fbbbyMkJASHDx9GbGwswsPD8dtvv+G1117De++9BwDw9fXF/v37cezYMZw+fRqnT5/GX3/9hZUrV2LDhg01b0iJSHVYlBNRi7Jjxw7Y2Nhg1apVCsXR0aNHVRjV09nY2ODkyZMoKSlRWC2vrKxEamqqUje4efR7pqWlwcrKCsCDwvzHH3/ErFmz8PHHH8PGxgZSqRQjRoxQOraQkBBs2LABkZGRKCgoQHZ2NmbNmqXwvDbF8/1opfn69euwt7dXeCwlJUXh58LCQhw+fBjDhw/H559/rvDYiRMnas0tEAjqHcuNGzdQVVWlsFpeVVWFmzdv1rkq3tQetVVdu3at1mPXr1+HTCarFZednR0mTZqESZMmoby8HNOnT8evv/6KadOmwdTUFACgq6uLAQMGYMCAAQAefALy+eefIzw8HK+99loT/1ZE9Dzq9XafiOg5hEIhBAKBwgptVVUVVq1apcKons7f3x/V1dVYs2aNwviWLVtQVFSk1Bx9+/YF8GDXj8f7xSUSCZYuXQoDAwOkpqZiwIABtdownsXDwwPu7u6IiorC+vXrIRAIau1N3hTPt7+/PwQCAf744w+F7f2uXr1aq9B+9EbgyRX5u3fv1toSEfhf/7mybTUBAQHIzc2tNdeWLVuQm5uLgIAApeZpTKampvD19cWhQ4eQlJRUMy6Xy/HLL78AAAIDAwE82D3myS0NJRJJTWvQo+chNze31nU8PDwUjiEi1eJKORG1KMHBwfjmm2/w+uuvIzAwEMXFxfjrr7/qVYw2pzFjxmDTpk349ttvcfv27ZotEffs2YO2bdvW2he9Lr169UJISAjCw8MxePBgDB8+HJaWlrhz5w527NgB4EGB9cMPP8DJyQkDBw5UOr6QkBD8+9//xj///IOuXbvWWoFtiufbyckJEyZMwLp16zBlyhQEBQUhJycH69evh5ubm0Ift56eHnr16oWdO3dCS0sLXl5eSEtLw+bNm2Fra6vQvw8A3t7eAICvv/4aQ4cOhUQigYuLC6RSaZ2xvPbaa9izZw8+//xzxMXFwd3dHfHx8QgPD0e7du2abAX5ypUr+PHHH2uNa2hoYMaMGfjoo48wadIkTJgwAePHj4e5uTkOHTqEY8eOYciQIejRoweAB61NH3/8MYKCgtCuXTvo6uriypUrCA8Ph7e3d01xPmjQIPj4+KBDhw6wsLBAdnY2tmzZAk1NTQwePLhJfkciqh/1fBUjInqK6dOnQy6XIzw8HIsXL4a5uTkGDhyI0aNHY9CgQaoOrxaxWIw///wTS5YsQXR0NHbv3o0OHTpg9erV+Oijj1BWVqbUPIsXL0bXrl2xadMm/Pbbb6isrISNjQ2Cg4Mxbdo0iMVihIaG4r333oO+vj569+6t1LxDhw7FkiVLUF5eXusLnkDTPd8fffQRzMzMsGXLFixZsgQODg745JNPcOvWrVpfrvzqq6/wzTff4ODBg9i2bRscHBzw9ttvQ0NDAwsWLFA4tlOnTnj33XexadMmfPzxx6iqqsKcOXOeWpTr6+tj48aNWL58OQ4ePIjIyEiYmpoiLCwMc+fOrfddZJV18eLFOneuEYvFmDFjBry8vLBp0yYsX74cGzduxP3792FnZ4d3330X06ZNqzne1dUVgYGBiImJwa5duyCTyWBlZYWZM2cqHDdt2jQcOXIEa9euRVFREUxNTeHt7Y2ZM2cq7PBCRKojkDfHt3SIiEhBdXU1unfvjg4dOjT4BjxERNR6sKeciKiJ1bUavmnTJhQWFta5LzcREb182L5CRNTEFi5ciIqKCvj6+kIsFuP8+fP466+/0LZtW4wdO1bV4RERkRpg+woRURPbvn071q9fj5s3b+L+/fswNTVF37598dZbb8HMzEzV4RERkRpgUU5EREREpGLsKSciIiIiUjEW5UREREREKsYvej6Ul1cCmax5O3lMTfWQk1PcrNckaomYK0TKYa4QKUdVuSIUCmBsrFvnYyzKH5LJ5M1elD+6LhE9H3OFSDnMFSLlqFuusH2FiIiIiEjFWJQTEREREakYi3IiIiIiIhVjUU5EREREpGIsyomIiIiIVIy7rxARERE9Q2lpCYqLC1BdXanqUKiR3L0rhEwma7T5RCJN6OkZQlu77u0OlcGinIiIiOgpKisrUFSUByMjM2hqSiAQCFQdEjUCDQ0hqqoapyiXy+WorCxHfv49aGhoQlNT3KB52L5CRERE9BRFRfnQ0zOEWKzFgpzqJBAIIBZrQVfXEMXF+Q2eh0U5ERER0VNUVVVAItFWdRjUAmhpaaOysqLB57N9RQVOXs1E5JEU5BaWw8RAglF9ndDDw1LVYREREdETZLJqCIUiVYdBLYBQKIJMVt3g81mUN7OTVzPx5+4EVDzsY8opLMefuxMAgIU5ERGRGmLbCinjRf9O2L7SzCKPpNQU5I9UVMkQeSRFRRERERERkaqxKG9mOYXl9RonIiIiamnmzJmBOXNmNPu5LRnbV5qZqYGkzgJcAGB/7B34d7SBSMj3SkRERNT4evfurNRxW7fuhJWVdRNHQ48TyOVyuaqDUAc5OcWQyZr+qXiypxwANDWEsDDSRtq9EthZ6GFSkCucbQ2bPBailsLcXB/Z2UWqDoNI7TFXGl9m5i1YWrZVdRiNZu/eKIWft2zZiKysDMyd+47C+Cuv+EFbu+G7zlRWPrjRkqamZrOeq6zG3Kf8cc/7exEKBTA11as7pkaPhp7p0Zc5n9x9pXv7NjibmI2N0cn4Yt1Z9PayQkg/JxjoNmwDeiIiIqInDRgwSOHnw4ejUVCQX2v8SWVlZdDS0lL6Oi9SUDdlMa7OWJSrQA8PS/TwsKy1otHZzQKejibYdeIm9sXcwbmkbIzu64i+PjYQCvnNbyIiImp6c+bMQHFxMd5//0OsWLEMiYkJmDBhMqZPn4l//jmMnTu3ISkpEYWFBTA3t8CgQUMxadJUiEQihTkA4PvvfwEAnDsXi3nzZmHx4iW4ceM6tm+PQGFhAby8vPHeex/C1tauUc4FgIiILdi0aT1ycu7ByckJc+a8jVWrVirMqY5YlKsZLbEGxvRzRi9PK6zbl4i1+5Jw9FIGJgW5wtHaQNXhERER0Qt6dL+SnMJymKrp/Ury8/Pw/vtvIygoGMHBg9GmzYP4oqL+gra2DkJDJ0BHRxtnz8bi119/QklJCWbPfuu58/75528QCkUYP34yiooKsXHjWnz22UKsWvVno5y7bVs4li1bAh+fjggNHYeMjAwsWPAu9PX1YW5u0fAnpBmwKFdT1ma6eG+cL2Li72LTwWQsXhOLPt7WCOnnBD3tl/NjHSIiopaupdyv5N69bHzwwccYMmS4wviiRf8PEsn/2lhGjAjBV199gW3btuL119+AWPzsttuqqir8/vuf0NB4UIIaGBjiu+++xvXr1+Do6PxC51ZWVuLXX1fCw8ML3377Y81xzs4uWLx4EYtyajiBQIBu7dugg5Mpdh6/gf1nUnE28S5C+jmhj7c1hLyZARERUbM7fjkDxy5lNOjclPQCVFUrbixRUSXDH1HxOHohvV5z9e5ghV5eVg2K43m0tLQQHDy41vjjBfn9+yWoqKiEt7cvduyIxK1bN+HiIn3mvIMHD6splgHA29sHAJCenvbcovx55yYkxKGgoABvvjlS4bjAwGAsX770mXOrAxblLYC2RAOh/i7o5WWFdfuS8OeeRBy9mIFJA6RwsGRLCxERUUvxZEH+vHFVMTe3UChsH7l+PQWrVq3EuXNnUFJSovBYSUnxc+d91AbziL7+gzqmqOj5uwY979zMzAdvlJ7sMdfQ0ICVVdO8eWlMLMpbEFtzPcwf74tTV7Ow+dA1/Ht1LPp1tMGoVxyhq8WWFiIioubQy6vhK9Tv/Xi8zvuVmBpIMH9CxxcNrdE8viL+SFFREebOnQEdHT1Mnz4LNja2EIvFSEpKwMqVKyCTPX+LQaFQVOe4Mjt0v8i5LQGL8hZGIBCgh6clvJ1Nsf2fG4g+l4rYhLsY088ZPb0s2dJCRESkxkb1dap1vxKxhhCj+jqpMCrlnD9/FgUFBVi8+Cv4+PzvDURGRv3abpqKpeWDN0qpqXfg7e1bM15VVYWMjAw4OT27PUbVeOvIFkpHSxPjA6X49NUuaGOsg9+j4vHlunO4ncWbRhAREamrHh6WmDLQDaYGEgAPVsinDHRTqy95Po3w4R3HH1+ZrqysxLZtW1UVkgI3t/YwNDTEzp3bUFVVVTO+f/8eFBUVqjAy5XClvIWzb6OPDyZ2xInLmdh6+Bo+W30G/TvaYkQfR+ho8Z+XiIhI3Ty6X0lL4+XVAfr6Bli8eBFCQkIhEAiwd28U1KV7RFNTE9OmzcCyZV/hX/96E35+/ZGRkYHdu3fBxsYWAjXvJuBKeSsgFAjQu4MVvpjRHf18bBB9NhUfrjqFk1cyW02fFREREamWoaERlixZBlNTM6xatRIbN65D587d8Oab81QdWo3Ro0Pxr3+9i8zMDPzww3e4ePE8vvxyKfT09CEWS1Qd3jMJ5KzaAAA5OcWQyZr3qXjyjp6N5WZmIdbuTcKNjEJI7YwwMUgKW3O9Rr8OUXNpqlwham2YK40vM/MWLC3bqjoMegEymQxDhgSib18/zJ+/EACgoSFEVdXzv5haX8/7exEKBTA1rbsmU9lK+eXLlzF79mz4+fmhQ4cO6NWrF6ZPn45z587Ve67XX38drq6uWLx4cRNE2vI4WBrgo8mdMCXYFWnZxVj0+xlsik5GaXnV808mIiIiaqHKy2vvbLNnz98oLCyAr28nFUSkPJU1Hd+5cwfV1dUYM2YMzM3NUVRUhF27dmHixIlYtWoVevXqpdQ8hw8fRmxsbBNH2/IIBQL09bFBR6k5Io5cx/4zdxATn4VQfxd0dbdQ+74qIiIiovq6dOkCVq5cgX79/GFgYIikpAT8/fdOODo6wc8vQNXhPZPKivJBgwZh0KBBCmPjxo1DQEAA1qxZo1RRXlFRgf/85z+YPn06VqxY0VShtmj6OmK8OtANfbytsG5vEn7eeRVHL6ZjQqAU1ma6qg6PiIiIqNFYW9vAzMwc4eGbUVhYAAMDQwQHD8asWXOgqane93RRq+05tLW1YWJigsJC5batWbNmDcrKyliUK8HJ2hAfT+mMIxfSEHHkOj79PQZBXe0wrGc7SMR1b8ZPRERE1JLY2NhiyZJlqg6jQVRelBcXF6OiogL5+fnYvn07kpKSMHv27Oeel52djR9//BGffPIJtLW1myHSlk8oFMCvoy06uVpg6+Fr2H3qNk7HZSHM3wWdXM3Z0kJERESkIiovyj/88EPs3bsXwIP9JcPCwjBr1qznnrd06VK0a9cOw4cPb+oQWx0DXTGmD26PV7ytsW5fEn7cfgUe7UwwMVCKNiY6qg6PiIiI6KWj8qJ89uzZCA0NRWZmJnbs2IGKigpUVlZCLBY/9ZxLly5h+/btWLt2baOt7j5te5qmZm6ur5LrPrp2tw42+PvEDazfk4CPf4vBKD9njOnvAi2xyv80iBSoMleIWhLmSuO6e1cIDQ3e1qU1aop/V6FQ2OAcVKt9yisrKzF69Gg4ODhg+fLldR4jl8sRFhYGS0tLfPfddzXjrq6umDx5Mj766KMGXbs17VPeEAXF5dhy6BpOXs2CqYEWxge4wMfFjC0tpBbUKVeI1BlzpfFxn/LWifuUP4empib69++Pffv2oaysrM5j9u/fj0uXLmHcuHFITU2t+R/woD89NTX1qefS0xnqSfD6UA/MH+8LLbEIKyIv47vwS7ibd1/VoRERERG1emrXo1BWVga5XI6SkhJoaWnVejw9PR0ymQxTpkyp9VhkZCQiIyOxatUqvPLKK80Rbqvjam+MT6d2wYHYVOw4fgMLf43BoO72GNS9LcSa3KWFiIiIqCmorCjPzc2FiYmJwlhxcTH27t0LKysrmJqaAnhQhJeWlsLJyQkA4O/vD1tb21rzPbo7aEhICDw8PJr+F2jFNERCBHezR7f2bbD5YDJ2Hr+Jk1czMSFQig5OZqoOj4iIiKjVUVlR/q9//QsSiQS+vr4wNzdHRkYGIiMjkZmZiaVLl9YcN3/+fMTExCAxMREAYG9vD3t7+zrntLOzQ0CAet+tqSUx1pdg1nBPvOKdi/X7k/Dt1kvwdTHDuP4uMDPiNpREREQEREXtwhdffIatW3fCysoaABASMhS+vp3w0UeL6n3uizp3Lhbz5s3C8uU/oWPHzo0yZ3NQWU/5sGHDUFZWhrVr12LRokXYsGED3NzcsGbNmlp3+iTVau9ggs+mdUVIPydcvZmLhb+exq4TN1HZBF+QICIioqb1/vtvIyCgN0pLS596zDvvzMGAAX1RXl7ejJHVz4EDe7FlywZVh9FoVLZSHhISgpCQkOcet3btWqXme7SSTk1DQyTEoO5t0b19G2yMTsa2o9dx4nIGJgRJ4dnOVNXhERERkZICAwfgxIl/cOzYEQQGBtd6PC8vF2fPnkFQ0EBIJJIGXWPDhggIhU279hsdvQ/JyUkYO3a8wriPT0dERx+HpqZmk16/sanV7iuk/kwMtDB7pBfeGesNAFi6+SJ+2HYZuYXc8YaIiKgl6NOnH7S1dXDgwN46Hz948ACqq6sRFFS7YFeWWCyGhoZq1n6FQiEkEkmTvylobGq3+wq1DJ6Opvh8ejfsibmNv0/cxOXrORjWqx2CuthBQ9SykoCIiOhloqWlhT59+uLQoQMoLCyEgYGBwuMHDuyFqakp7Oza4uuvv8TZszHIysqClpYWOnbsjNmz33pu/3ddPeXXr6fg22+/wpUrl2FoaIjhw0fBzMy81rn//HMYO3duQ1JSIgoLC2BuboFBg4Zi0qSpEIke7AQ3Z84MXLhwDgDQu/eDvnFLSyuEh+96ak95dPQ+rFu3Grdu3YSuri569uyDN96YByMjo5pj5syZgeLiYnzyyedYunQJ4uOvQl/fAGPGhGHChNo7/zUmFuXUYJoaQgzt6YAeD1tawg+n4PjlDEwMlMLdweT5ExAREb2EYjLPYWfKHuSV58NYYoRhTsHoatmxWWMIDAzGvn27cfhwNIYNG1kznpmZgStXLiEkJAzx8Vdx5colBAQMgLm5BTIy0rF9ewTmzp2Jdeu21rl19dPk5NzDvHmzIJPJMHHiFGhpaWPnzm11tsdERf0FbW0dhIZOgI6ONs6ejcWvv/6EkpISzJ79FgBgypRpKC0tRVZWBubOfQcAoK2t89TrP/pCqYeHF954Yx7u3cvC1q2bER9/FatWrVGIo7CwAP/3f/Pg59cf/fsH4dChA1i5cgUcHZ3Ro0cvpX/n+mJRTi/MzEgbc0d3wMVr97B+fxK+2nQBXd0tEOrvAqg/imkAACAASURBVGP9hvWiERERtUYxmeewISEClbJKAEBeeT42JEQAQLMW5l26dIORkTEOHNirUJQfOLAXcrkcgYED4OTkDD8/xV3tevV6BbNmTcXhw9EIDh6s9PXWr/8TBQX5+PXXtXB1dQMADBw4BOPGjax17KJF/w8Syf8K/hEjQvDVV19g27ateP31NyAWi9GlS3dERm5FQUE+Bgx49gYhVVVVWLlyBZydpVix4ueHrTVCuLi4YdGij7Br1zaEhITVHH/3bhY+/fT/1fTbDxkyHCEhQ/D33ztYlFPL4O1sBve2xog6dQtRp27jYkoORvRuh/6dbNnSQkRErcbpjLM4mXGmQefeKLiNKnmVwlilrBLr48NxIj2mXnP1sOqCbladGhSHhoYG/P0DsH17BO7duwczswf3ITlwYB9sbe3Qvr2nwvFVVVUoKSmGra0d9PT0kZSUUK+i/OTJ4/Dy8q4pyAHA2NgYgYEDsW3bVoVjHy/I798vQUVFJby9fbFjRyRu3boJFxdpvX7XhIQ45OXl1hT0j/j7B+KHH77DiRPHFYpyPT09BAQMqPlZU1MT7u4eSE9Pq9d164tFOTUqsaYII/o4oqenJTYcSMbmg9dw7GFLi6u9sarDIyIiUqknC/LnjTelwMBgREZuxcGD+zB27HjcvHkD164lYerU1wEA5eVlWLt2NaKidiE7+y7kcnnNucXFxfW6VlZWJry8vGuN29u3rTV2/XoKVq1aiXPnzqCkpEThsZKS+l0XeNCSU9e1hEIhbG3tkJWVoTBuYdEGAoFAYUxf3wApKdfqfe36YFFOTcLCWAdvhXTAheR72HAgGf/dcB49PNpgrJ8zDPXY0kJERC1XN6tODV6hXnj8C+SV59caN5YY4V8dZ71oaPXi5eUNKysb7N+/B2PHjsf+/XsAoKZtY9myrxAVtQtjxoyDp6cX9PT0AAiwaNGHCgV6YyoqKsLcuTOgo6OH6dNnwcbGFmKxGElJCVi5cgVksqa/R4pQKKpzvKl+50dYlFOTEQgE8JWao307E/x98ib2nL6NC9fuYWQfR/h1tIGohW1VRERE9KKGOQUr9JQDgKZQE8OcGr794IsICAjC2rV/IDX1DqKj98HV1b1mRflR3/jcuW/XHF9eXl7vVXIAaNPGEqmpd2qN3759S+Hn8+fPoqCgAIsXfwUfn//12GdkpNcxq6COsdosLa1qrvX4nHK5HKmpd9CunZNS8zQ1VkXU5CSaIox6xQmfT+8GRysDbDiQjM9Xx+JaaoGqQyMiImpWXS07YrzbaBhLHmzDZywxwni30c2++8ojQUEDAQDff78Mqal3FPYmr2vFOCJiM6qrq+t9nR49euHy5YtITEyoGcvLy8P+/bsVjnu0t/jjq9KVlZW1+s4BQFtbW6k3CG5u7WFsbILt28NRWfm/N0OHDkUjO/suevZsui9v1gdXyqnZWJro4J1QH5xNzMbG6GR8se4sentZIaSfEwx0xc+fgIiIqBXoatlRZUX4k9q1c4SzsxTHjh2FUChE//7/+4Jjz569sXdvFHR19eDg0A5Xr15GbGwMDA0N632d8eOnYO/eKLzzzmyEhIRBItHCzp3b0KaNFYqLk2uO8/LqAH19AyxevAghIaEQCATYuzcKdXWOuLq6Yd++3VixYinc3NpDW1sHvXu/Uus4DQ0NvPHGXHzxxWeYO3cmAgKCkJ19F1u3boKjoxOGDq29A4wqsCinZiUQCNDZzQKejibYdeIm9sXcwbmkbIzu64i+PjYQCpX7KIqIiIgaR1BQMK5dS4Kvb6eaXVgA4K233oVQKMT+/btRXl4BLy9vfPvtD3jnnbn1voaZmRmWL/8Zy5Ytwdq1qxVuHvTll/+uOc7Q0AhLlizD999/i1WrVkJf3wBBQQPRuXNXvPPOHIU5hw8fjaSkBERF/YXNmzfA0tKqzqIcAAYNGgqxWIz16//EDz98B11dXQQGBmPWrLl17pWuCgJ5U3ettxA5OcWQyZr3qTA310d2dlGzXlPdpN8rwbp9iUi4nY+2lvqYFOQKR2uD559ILxXmCpFymCuNLzPzFiwta+8QQi2bhoYQVVWN/6XR5/29CIUCmJrq1f1Yo0dDVA/WZrp4b5wvZg7zQH5xORavicXq3QkoLq18/slERERErQTbV0jlBAIBurVvgw5Opthx7AYOxKbibOJdhPRzQh9vawgFbGkhIiKi1o0r5aQ2tCUaCOvvgkXTusDGXA9/7knE4jVncTOzUNWhERERETUpFuWkdmzN9TB/vC9eH9IeOYVl+PfqWKzdl4iSMra0EBERUevE9hVSSwKBAD08LeHtbIrt/9xA9LlUxCbcxZh+zujpZcmWFiIiImpVuFJOak1HSxPjA6X49NUusDDWxu9R8fhy/TnczuLuAkRERNR6sCinFsG+jT4WTOyEqYPckJlzH5+tPoMN+5Nwv6xK1aERERERvTC2r1CLIRQI0KeDNXxdzLHt6HVEn01FTMJdhPo5o7tHGwjY0kJERE1ALpfzNYae60Vv/cOVcmpx9LQ1MWmAKz5+tTNMDbSw6q84/HfDeaRmF6s6NCIiamVEIg1UVlaoOgxqASorKyASNXy9m0U5tVgOlgb4aHInTAl2RVp2MRb9fgabDyajtJwtLURE1Dj09IyQn5+NioryF14JpdZJLpejoqIc+fnZ0NMzavA8bF+hFk0oEKCvjw06Ss0RceQ69sbcwem4LIT6u6CruwU/biQioheira0LACgouIfqai76tBZCoRAymazR5hOJNKCvb1zz99IQAjnf9gEAcnKKIZM171Nhbq6P7GzuItKYUtILsG5vEm5lFcG9rTEmBEphbdbwBCH1wFwhUg5zhUg5qsoVoVAAU1O9uh9r5liImpSTtSE+ntIZE4OkuJVZhE9/j8HWw9dQXlGt6tCIiIiInortK9TqCIUC+He0RWdXC2w9fA27T93G6bgshPm7oJOrOVtaiIiISO1wpZxaLQNdMaYPbo8FEztCR6KJH7dfwbItF5GVe1/VoREREREpYFFOrZ6LrRE+ndoZ4wJckJJegI9/O43Ioykor2RLCxEREakHtq/QS0EkFCKwsx26uFlg66Fr+OvELZy8koXxAS7wcTFjSwsRERGpFFfK6aVipCfB60M9MH+8L7TEIqyIvIzvwi/hbn6pqkMjIiKilxiLcnopudob49OpXTDWzxmJd/KxcNVp7Dh2AxVsaSEiIiIVYPsKvbQ0REIEd7NHt/ZtsPlgMnYcu4ETVzIwIVCKDk5mqg6PiIiIXiK8edBDzXnzoJjMc9iZsgf55fkwkhhhmFMwulp2bJZr09PF3czF+v1JyMi5D18XM4zr7wIzI21Vh0XgDVGIlMVcIVKOOt48iEX5Q81VlMdknsOGhAhUyiprxjSFmhjvNpqFuRqoqpZh35k72Hn8BiAHBvd0QHBXe2hqsNNLlVhoECmHuUKkHHUsyllpNLOdKXsUCnIAqJRVYmfKHhVFRI/TEAkxqHtbfPF6d3g5mWLb0ev45LfTuHIjR9WhERERUSvGoryZ5ZXn12ucVMPEQAuzR3rhnbHeAIClmy/ix22XkVtYpuLIiIiIqDViUd7MjCVGT31s940DqJZx9w914ulois+nd8PIPu1wMSUHH646hahTt1BVLVN1aERERNSKsChvZsOcgqEp1FQY0xRqop2BPf66sQ/Lzq1E9n22SqgTTQ0hhvZqh8WvdYOHgwnCD6fg099jEH8zV9WhERERUSshWrRo0SJVB6EOSksr0BxfebXRs4KJljFuF6aivLoMxhIjhEiHYYx0ONpom+FU5lkcTTsJfbEu7PRseKdJNaKjpYlu7dvAwVIfF67dw4GzqcjIKYGTjSG0JdxdtCnp6kpw/36FqsMgUnvMFSLlqCpXBAIBdHTEdT/G3VceaM4tER+p65u/eWX5WBO/BUl51+Bl1h4T3EKgL677W7qkOhWV1Yg6dQtRp25DJBJgRO926N/JFhoifvjUFLijBJFymCtEylHH3VdYlD+kLkU5AMjkMhy+cww7ru+BtkgLE93HwNPMvVljI+XczbuPDQeScSklBzbmupgYKIWrvbGqw2p1WGgQKYe5QqQcFuVqTJ2K8kfSizOxOm4j0ooz0Nu6G0a5DIVEVPdHHqQ6crkcF5LvYcOBZOQUlqGHRxuM9XOGoZ5E1aG1Giw0iJTDXCFSjjoW5ewpf6i5esof97x+Jn2xHrpbdUGVrApHUk/g/N1LcDC0g5HEsBmjpOcRCASwMtVFXx9rCATA0YvpOHwhDRJNEdpa6kPI7wW8MPbJEimHuUKkHPaUqzF1XCl/XFJeCtbEbUZBRSGCHfojuK0/REJRE0dIDZGZex/r9yXi6s082FvoYWKQK5xt+UbqRXD1j0g5zBUi5ajjSjmL8ofUvSgHgPuVpdiStB1nss7DwcAeU9qHwkLHvAkjpIaSy+U4m5iNjdHJyCsqR28vK4T0c4KBLtuPGoKFBpFymCtEymFRrsZaQlH+yNmsC9iYuA3VsiqMdhmKXtbduHWimiqrqMKu4zex78wdSDRFGN3XEX19bCAU8t+rPlhoECmHuUKkHBblj7l8+TJ++uknxMXFIScnB/r6+nBzc8Ps2bPRsWPHZ567b98+REVF4dKlS8jJyYGVlRX8/Pzw5ptvQl9fv0HxtKSiHHiwdeLa+C1IzLsGT1N3THAPgYG4Yb87Nb30eyVYty8RCbfz0dZSH5OCXOFobaDqsFoMFhpEymGuECmHRfljoqKisHPnTnTo0AHm5uYoKirCrl27kJiYiFWrVqFXr15PPbdbt26wsLBAQEAArK2tkZiYiE2bNsHBwQERERGQSOq/60VLK8qBh1snph7HjpTd0BJJMNF9DLzM2jdihNSY5HI5YuLvYtPBZBQWV6CPtzVC+jlBT1vz+Se/5FhoECmHuUKkHBblz1FaWoqAgAB4enri559/fupxp0+fRrdu3RTGtm/fjvnz5+M///kPRo0aVe9rt8Si/JHHt07sZd0Vo5yHQkuD2/Gpq9LyKuw4dgMHYlOhLREhpJ8T+nhbc5eWZ2ChQaQc5gqRctSxKFer2w9qa2vDxMQEhYWFzzzuyYIcAAICAgAAKSkpTRKbOrPWs8R7neci0L4fTqSfwZdnvsWNgluqDoueQluigbD+Llg0tQtszHTx555ELF5zFjczn/13T0RERK2Xyovy4uJi5Obm4vr161i6dCmSkpLQo0ePes9z7949AICx8ct5N0VNoQZGOA/CW74zUS2XYem5lfjr+j5Uy6pVHRo9ha2FHuZP6IjXhrgjp6AU/14di7X7ElFSVqnq0IiIiKiZaag6gA8//BB79+4FAGhqaiIsLAyzZs2q9zyrVq2CSCRCUFBQY4fYorgYO+LDrv/ClqQd2H3zAOJyEjHFIwxtuHWiWhIIBOjpaQUfZzNs/+cGos+lIjbhLsb0c0ZPL0u2tBAREb0kVN5TnpiYiHv37iEzMxM7duyAjY0NFi5cCF1dXaXn2LVrF959913MnDkT77zzThNG27KcvHMWv8RuQFV1FSb5jEagUx9unajmrqcVYGXERSTcyoO7gwneGN0B7ax54yEiIqLWTuVF+eMqKysxevRoODg4YPny5UqdExsbi2nTpqFHjx748ccfIRI17C6XLfmLns+SX16AtXFbkJCXDE9TN4x3GwNDCbdOVGcyuRzHL2dg66EUlJRVon9HW4zo4wgdLZV/sKUy/PIakXKYK0TKUccveooWLVq0qHnDeTqRSIS7d+9i8+bNeO2116Ch8ewiJCEhAdOnT4ezszN++umnBm2F+EhpaQWa++2Jrq4E9+9XNOk1tDS00MXSF7qaOjiefhonM87AQscclroWTXpdajiBQIC2bfTRx9saZeXVOHQuDccuZ8BQVwxbc92X8tOO5sgVotaAuUKkHFXlikAggI5O3Xf3VvkXPZ9UVlYGuVyOkpKSZx53+/ZtvPbaazAxMcHPP/8MHR2dZoqw5REKhPCz6435Xd6CkcQQv1z+E+vjt6KsqkzVodEz6GlrYtIAVyyc0hmmBhKs+isO/91wHqnZxaoOjYiIiBqZytpXcnNzYWJiojBWXFyMYcOGQSaT4fDhwwCA9PR0lJaWwsnJqea47OxsjBs3DuXl5di4cSNsbW1fOJ7W2r7ypCpZFf6+sR/7bx2GqZYxpniMg6Nh22aNgepPJpfjn4vpCD+cgtLyagR2scWwXu2gLXk5Wlr4kTyRcpgrRMpRx/YVlRXlkydPhkQiga+vL8zNzZGRkYHIyEhkZmZi6dKlGDRoEABg0qRJiImJQWJiYs25w4cPR0JCAl577TVIpVKFee3t7eHr61vveF6WovyRa/k3sCZuE3LL8jHAwR+DHAIgEjasH5+aT9H9CkQcuY6jF9NhpCdGWH8XdHGzaPUtLSw0iJTDXCFSDovyx4SHh2PHjh24du0aCgsLoa+vDx8fH0ybNg1du3atOa6uotzV1fWp844cORJffvllveN52YpyACitKkN40k6cyoyFvb4tXm0fhjbsNW8RUtILsG5vEm5lFcG9rTEmBklhZar8jkUtjapzhailYK4QKYdFuRp7GYvyR87fvYyNCRGokFVilPNg9LHp0epXXlsDmUyOwxfSEHnkOsorqxHU1Q7DeraDRNz6PvFQl1whUnfMFSLlsChXYy9zUQ482DpxXfxWxOcmob2pKya6jeXWiS1EYUkFth6+huOXM2FiIEGYvws6uZq3qjdW6pQrROqMuUKkHBblauxlL8oBQC6X40jaCWy/9jfEIjHGu4XAx9xT1WGRkpJT87F2bxJSs4vh2c4EEwKlaGPSOnYlUrdcIVJXzBUi5bAoV2Msyv8nsyQLq+M24U5RGrpbdcYYl2HQ0tBSdVikhGqZDAfPpWH7P9dRWSVDcLe2GNyjLSSaLbulRV1zhUjdMFeIlMOiXI2xKFdUJatC1I0D2HfrEEy0jDGlfRicjBxUHRYpKb+4HFsPXcPJq1kwNdDC+AAX+LiYtdiWFnXOFSJ1wlwhUg6LcjXGorxuKfk38WfcJuSW5SGorR8GtQuAhvDl2Bu7NUi8nYd1+5KQdq8EHZxMMT5QCgsjbVWHVW8tIVeI1AFzhUg56liUixYtWrSoecNRT6WlFWjutyct4XbIJlpG6G7VGYUVRTiSehxXcxLgbNQOeuK6/6BIvZgZauMVb2toSzRw/EomomNTIZPL4WRtAJFQ7W7o+1QtIVeI1AFzhUg5qsoVgUAAHR1x3Y9xpfwBrpQ/34W7l7EhMQIV1RUY4TwYfW16tth2iJdRXlE5Nh9MRkz8XZgbaWFCoBQdnMxUHZZSWlquEKkKc4VIOeq4Us6i/CEW5copKC/EuoStiMtJhLuJFBPdx8BIYqjqsKge4m7mYv3+JGTk3IevixnG9XeBmZq3tLTEXCFSBeYKkXLUsShn+8pDbF9RjpaGBF3a+EJfrIfj6TE4kR4DM21TWOm2UXVopCRzI2309bGGRCzCscsZOHguDXIAjlYGEAnV85OPlpgrRKrAXCFSDttX1BhXyusvq+QuVsdtwu2iVHSz7IQx0uHQ5taJLUpOQRk2HUzG2cRstDHWxoQgKTzbmao6rFpaeq4QNRfmCpFy1HGlnEX5QyzKG6ZaVo2omwew9+ZBmGgZYXL7MDgbtVN1WFRPl6/nYP3+JNzNK0VnV3OE9XeBiYH6vMFqDblC1ByYK0TKUceinO0rD7F9pWGEAiFcjZ3hZuKCi9lXcejOMVTKquBs1A5CQcvZ3eNl18ZYB319bKApEuCfSxk4dC4NIqEA7awMIFSDlpbWkCtEzYG5QqQctq+oMa6Uv7iyqjJEJO/CiYwzsNOzxhSPcew1b4Hu5Zdiw4FkXLh2D1amOpgYKIW7g4lKY2ptuULUVJgrRMpRx5VyFuUPsShvPBezr2BDQgTKq8sx3GkQ+tr25Kp5C3Th2j1s2J+EewVl6OpugVB/FxjrS1QSS2vNFaLGxlwhUo46FuW8NSM1Om9zTzgYtMX6hK0IT96JK/fiMan9WG6d2ML4OJuhfVtjRJ26hahTt3ExJQcjerdD/0620BDxTRYREVFj4kr5Q1wpb3xyuRzH0k8hMvkvaAg1MM5tNDpadFB1WNQAd/PuY8OBZFxKyYGNuS4mBkrham/cbNdv7blC1FiYK0TKUceVchblD7EobzpZ97Px59VNuFV0B10tO2KsdDi0NdT7ZjVUm1wux4Xke9hwIBk5hWXo4WGJsX5OMNRr+paWlyVXiF4Uc4VIOepYlHP3lYe4+0rT0dPURXerzgAE+CftJM5kXYCdng1MtZtvpZVenEAggJWpLvr6WAMAjlxIx+ELaZBoitDWUh9CQdPt0vKy5ArRi2KuECmHu6+oMa6UN48bBbewOm4TckpzEWDfF4Mdg6Ap5FcbWqLM3PtYvy8RV2/mwd5CDxODXOFs2zTfG3gZc4WoIZgrRMpRx5VyFuUPsShvPmVV5Yi8tgvH02Ngo2eFV9uPg7WeparDogaQy+U4m5iNjdHJyCsqR28vK4T4OcHgKasADfWy5gpRfTFXiJSjjkU521ceYvtK89EQasDLrD3s9KxxJus8jqSdgEQkRlsDOwiasAWCGp9AIIC12YOWFplMjiMX0nHkfDq0JSK0baPfaP+eL2uuENUXc4VIOWxfUWNcKVeNwooirI8Px5WceLgaO2OS+1gYaxmpOixqoPR7JVi3LxEJt/PR1lIfk4Jc4Wht8MLzMleIlMNcIVKOOq6Usyh/iEW56sjlchxPP42I5F0QCTUwznUkOrXxUXVY1EByuRyn47Ow+eA1FBZXoI+3NUL6OUFPW7PBczJXiJTDXCFSDotyNcaiXPXu3s/Gn3GbcbPwNrq08cVY6QjoaHLrxJaqtLwKO47dwIHYVOhoaSCknxN6d7Bq0C4tzBUi5TBXiJTDolyNsShXD9Wyauy5dRB7bkbDUGyAye1DITV2UnVY9AJS7xZj3b5EJKUWwNHaAJOCXNHWUr9eczBXiJTDXCFSDotyNcaiXL3cKLiNNXGbkF2aA3/7PhjqGMytE1swuVyOk1czseXgNRTdr0S/jjYY9YojdLWUa2lhrhAph7lCpBx1LMq5+8pD3H1FvRhrGaKHdReUVJbgSOoJXL4XB0dDBxiI67fCSupBIBDAzkIfr3hbo7xShkPn0/DPpQzoa4tha6H33F1amCtEymGuECmHu6+oMa6Uq6/L9+KwPj4cpVWlGOY0EH52vSEUCFUdFr2A21lFWLsvESlphXC2NcTEQCns2zz9DRdzhUg5zBUi5ajjSjmL8odYlKu3oopirE8Ix+V7cZAaO2Myt05s8WRyOY5fzsDWQykoKatE/062GNHbETpatduUmCtEymGuECmHRbkaY1Gu/uRyOU5kxCA8eRdEAhHCXEeiM7dObPGKSyux7eh1HD6fBn1dMUL9nNHdo41CSwtzhUg5zBUi5bAoV2MsyluOu/fvYU3cJtwovI3ObXwQKh0BHU0dVYdFL+hGRiHW7UvEjYwiSO2MMDFIijt3ixF5JAW5heUwMZBgVF8n9PCwVHWoRGqLrytEymFRrsZYlLcs1bJq7Lt1CFE3D8BArI/J7qFwNXFWdVj0gmRyOY5eTEfE4RSUlFVBKBQo5KVYQ4gpA91YmBM9BV9XiJSjjkU5vy1HLZJIKMLAdgF4t9NsiEWaWH7hF0Qk70JldaWqQ6MXIBQI0M/HBl/M6A6JprDWG+WKKhkij6SoKDoiIqKmw6KcWrS2Bnb4oMu/0MemBw7e+QdLYlcgrThD1WHRC9LXEaO8UlbnYzmF5c0cDRERUdNjUU4tnkQkRpjrSLzRYSqKKoux5MxyHLh9BDJ53UUdtQymBpI6x7U0RSgu5SciRETUurAop1bD08wdH3V9Bx6mbth27W8sP/8LcsvyVB0WNdCovk4Qayj+J0ooEKCsshof/nIKhy+kNfv3QIiIiJoK7+j5EO/o2TpIRGJ0tPCGsZYxTmacwbH00zCWGMFGz0rVoVE92VnowdRQC7cyC1FWXg1TAwkmBEkxrJcDbmYU4uC5NFy8lgNbcz2YGGipOlwitcDXFSLl8I6eaoy7r7Q+90pz8GfcJlwvuIVOFt4Icx3JrRNbqCdzRS6XIyb+LrYcuoa8onL08LDEGD8nGOnV3fJC9LLg6wqRctRx9xWulD/ElfLWR0dTB90sO0FDqIGjaScRk3kONnpWMNM2VXVoVE9P5opAIICtuR76+lgDAI5eTMfB82kQCQVoZ2UAoVDwtKmIWjW+rhAphyvlaowr5a3brcI7+DNuE7LuZ8Pfrg+GOQZDU6Sp6rBISc/Llay8+9h0IBkXU3LQxkQH4wNc4OXIN1/08uHrCpFy1HGlnEX5QyzKW7+K6gpsuxaFo2knYKXbBq+2HwdbfWtVh0VKUDZXLqXcw8YDycjKK4WPsxnC+jvDwpgtS/Ty4OsKkXLUsShn+8pDbF9p/URCETzN3OBgYI+zWRdwOPU4NIQaaGdoD4GA7Q7qTNlcaWOig36+NtASi3D8Siaiz6aiokoGJ2tDaIi42RS1fnxdIVIO21fUGFfKXy7FFSXYmBiBC9lX4GzUDpPdw2CqbazqsOgpGpIreUXlCD98DSevZsFYX4Kxfs7o6m7BN2DUqvF1hUg56rhS3ihFeVVVFaKjo1FQUAA/Pz+Ym5u/6JTNjkX5y0cul+NU5lmEJ+0AIMBY6XB0tezIok0NvUiuJKfmY/3+JNzOKobUzggTAqWws6j7P4hELR1fV4iU0yqK8iVLluD06dOIiIgA8KCwmTx5MmJjYyGXy2FkZIQtW7bA3t7+xSNvRizKX173SnMfbp14E74WHTDOdRR0uXWiWnnRXJHJ5Dh6MR2RR6+jpKwSfr42GNHHEXra/LIvtS58XSFSjjoW5fVusvznn3/QuXPnmp8PHjyIM2fOYPr06fjm6SV0xwAAIABJREFUm28AAL/88ksDQyVqfmbaJni74ywMcwzGxewrWHx6KRJyk1UdFjUioVCAfr42+GJGd/j52uDQ+bQHdwU9z7uCEhGRetCo7wmZmZlo27Ztzc+HDh2Cra0t3n33XQBAcnIydu3a9dx5Ll++jJ9++glxcXHIycmBvr4+3NzcMHv2bHTs2PG552dlZeGLL77A8ePHIZPJ0L17dyxYsAB2dnb1/ZWIIBQIMcDBH+6mUvx5dRNWXFgFP9veGOY0EGJundhq6GlrYmKQK/r62GD9/iSs2ZuIwxfSMCFQChdbI1WHR0REL7F6r5RXVlZCQ+N/tfzp06fRs2fPmp/t7OyQnZ393Hnu3LmD6upqjBkzBh9//DGmT5+O3NxcTJw4EcePH3/muSUlJZg8eTLOnj2LWbNmYd68eYiLi8PkyZNRUFBQ31+JqIa9vi3md3kLfW174VDqMfw3djnuFKWrOixqZHYWepg/3hczh3mg6H4l/rPuHFbtuoq8onJVh0ZERC+peq+UW1pa4vz58xg7diySk5Nx584dzJs3r+bxnJwc6Og8vx930KBBGDRokMLYuHHjEBAQgDVr1qBXr15PPXfDhg24desWIiMj0b59ewBAnz59MHToUKxevRpvvfVWfX8tohpikSbGSofD09QN6+K34KvYFRjiGIQA+74QCritXmshEAjQrX0b+Dib4e9TN7Hn9G2cS7qHob0cENjZDpoa/LcmIqLmU+9XncGDB2P79u2YOXMmZs6cCT09PfTt27fm8fj4+AZ/yVNbWxsmJiYoLCx85nF79+6Fj49PTUEOAE5OTujRowd2797doGsTPam9qSs+7PYOOpi1x46U3fj23M/IKc1VdVjUyCRiEUa94oT/91o3uLc1RvjhFHzy22lcSslRdWhERPQSqXdRPnPmTIwcORIXLlyAQCDAf//7XxgYGAAAioqKcPDgQfTo0UPp+YqLi5Gbm4vr169j6dL/396dx0dVn/sD/8xMZiaTZZYkkxCy74GABFQQqQgCvQjihoqA1KVSrbZeahdv9S6/Llpua716be214quAokgDsrghm7iggEa2hGyTDCFkT2ayzUxmO78/ZhgIAZyEJOck+bxfr/uyzJxJHnr7mA+H53yfF1BWVnbZz3u9XpSWlmLChAm93ps4cSLMZjPsdntff1tEFxWhDMcPJ9yHH4xbgjOdtXju0P/gqzrfSUM0ssQawvDEXVdh1d2TAJkML/7zKF7651E0WGxil0ZERKPAgC4P8nq96OrqQmhoKJTK4B6Oe+KJJ7Bz504AgFKpxOLFi/HMM89Apbr4tqPW1lZMnz4dTz75JB555JEe723YsAG//e1vsWvXrmF3JCNJX2NXC/56cC1ONlVgWuJk/OiaZYhU87zrkcjl9mLHZyZs3FUKl1vAHbMycPecbGjUfZ74IyIiCsqA/oRxu92IjIzs02cef/xxLFmyBPX19di2bRucTidcLtclQ3l3t+9BrIu9r1arAQAOh6OPlfOccvpuMqjw2ISHsbt6P96r/BgljRW4b9w9GB+dI3ZpI54YvXLDhDGY4B9n+eeecuw+VI27Z2dg2rg4LpgiyeLPFaLgjIhzyvfv34+XX365x2sbNmzAlClTkJ+fj5///OdwuVxBf72cnBzMmDEDixcvxuuvv46ioiL8+te/vuT1Z4O30+ns9d7ZwB4aGhr09yfqC7lMju+nzMYvr/kJNMow/PXo69hUtg1OT/D/m6fhwxCpxspF4/Hr+6ZAG6bC37cX4783FKK6gaGHiIgGVp9D+euvv47KysrAr00mE5577jnExsbi+uuvxwcffIANGzb0qxilUok5c+bg448/vuTdbr1eD5VKddFjF5uamiCTyWA0Gvv1/YmClRSZgKeueQKzE7+H/TVf4L8Pv4Tqjhqxy6JBkpWox3/cfw1+MD8HtS02/GbtYbzxcSk67fzDGBERDYw+h/LKysoeD1l+8MEHUKvVKCgowJo1a7BgwQJs3bq13wU5HA4IgoCurq6LFyyXIzs7GydOnOj13rFjx5CSkgKNRtPv708ULJVCibuyb8VP8h+G3e3An77+C3aa98IreMUujQaBXC7DrPwE/OGR63DTlER88u0Z/PrVL7GPW0GJiGgA9DmUt7W1wWAwBH594MABXHfddYiI8M3HTJ06FTU1333HsLW199FynZ2d2LlzJ+Lj4xEdHQ0AqK2thclk6nHdv/zLv+DIkSMoLi4OvFZZWYmvvvoK8+fP7+tvieiKjIvKxjPTnkS+cQK2V36EFwv/D808OnHECg9VYvm8bPzmwalINEbgjZ2l+O3awyg7bRW7NCIiGsb6/KCnwWBAba1vw2FnZyeOHz+OJ598MvC+2+2Gx+P5zq+zatUqqNVqTJ48GUajEXV1ddiyZQvq6+vxwgsvBK576qmncOjQIZSWlgZeW7ZsGf75z3/iRz/6ER588EEoFAqsXbsWRqMRDzzwQF9/S0RXLFwZhofylmNizHi8U7oVfzj0P7gr+zZcN+ZqPhQ4QiXGRuBXyybjcEkj3tlbgdUbCnFdXhzunpUJQ6Ra7PKIiGiY6XMoz8/Px8aNG5GZmYlPP/0UHo8HM2fODLx/6tQpxMbGfufXufXWW7Ft2za88cYbaG9vR2RkJPLz8/HHP/4RU6dOvexnIyIi8MYbb+C5557DK6+8Aq/Xi2nTpuGZZ57pcRefaCjJZDJMHTMFGbo0vHHyHbx5chOONxdjWc5iRKjCxS6PBoFMJsPUcXGYlBGD9786hY8OVuNbbgUlIqJ+6PM55RUVFfjBD34QGD+544478Ic//AEAIAgC5syZg2nTpgVeGy54JCINJK/gxZ7qT7GjcifClWG4b9zdyIvOFbusYWu49Eqj1Y539pTj2/JmxBk0WDo3C1dlxIhdFo0iw6VXiMQmxSMR+7U8yGq1orCwEJGRkbj22msDr7e1tWHr1q2YNm0acnOHVwBhKKfBUNNRi7XFb6OuqwEzE6bjjsyFUCkufgY/Xdpw65XjlS14e3c56lttuCojGkvnZiHOECZ2WTQKDLdeIRLLiAnlIxFDOQ0Wl8eF7ZUfYe/pzxAXZsQD45ciWZsodlnDynDsFbfHi91f12DbF1XweLz4/rXJuOX6FISquBWUBs9w7BUiMYyoUF5dXY09e/bg9OnTAICkpCTMmTNn2K63ZyinwVbSWo43Tm5Cu7MDC9PmYV7yLCjkCrHLGhaGc69YO7tR8IkJB07UQx+hwj2zMzFtPLeC0uAYzr1CNJRGTCh/8cUX8dprr/U6ZUUul+ORRx7Bv/7rv/avUhExlNNQsLls2Fj6Lr5pPIp0XQruH38vYjTRYpcleSOhVyrOtGHDrjKcqu9AVqIOy+dlIzkuUuyyaIQZCb1CNBRGRCgvKCjAv//7v2Py5Ml4+OGHkZWVBQAoLy/H66+/jm+//RbPPvss7rzzziuvfAgxlNNQOlz/Ld4pexdewYu7sm7D9PhreOf0MkZKr3i9Aj4/XoeCT0zocrgwKz8Bd8xMR4RGKXZpNEKMlF4hGmwjIpTfeeedUCqV2LBhA0JCes5Gut1uLF++HC6XC1u2bOl/xSJgKKeh1uqwYH3xOyi3VmJSTB6W5i5GpOrijTrajbRe6XK4sO2zKuwtPAONWoE7Z6bjxvwEyOX8gxldmZHWK0SDRYqhvM+H6JpMJixYsKBXIAeAkJAQLFiwoNcGTiLqLSrUgCcm/wh3ZC5EUUsJnj30Ak40nxS7LBoC4aFKLJuXjf/34LVIio3AGx+X4TfcCkpENKr1OZQrlUrYbLZLvt/V1QWlkn8VSxQMuUyOuck34lfXPoFIZQT+duwfeLt0C7o9TrFLoyGQGBuBXy6djB/fPgFdDhdWbyjEq9uLYOnoFrs0IiIaYn0O5RMnTsQ777yD5ubmXu+1tLRg06ZNmDRp0oAURzRaJETE41fX/BRzkmbiizMHsfrwizjVflrssmgIyGQyXJsbi2cfvg6Lrk/FN6VNePrvX+H9L81wub1il0dEREOkzzPlhw8fxgMPPIDw8HAsXrwYmZmZAHybPrds2YKuri6sXbsW11xzzaAUPFg4U05SUWapwPriTWhztmNB6lx8P2X2qD86cTT1yvlbQWP1Gtw7Nwv5mdwKSsEZTb1CdCWkOFPeryMR9+7di9/97neoq6vr8frYsWPxn//5n5g1a1a/ChUTQzlJic1lxztl7+LrhiNI0ybj/vFLYQwbvUcnjsZeOVHVgrd2nbcVdE4W4qK4FZQubzT2ClF/jJhQDgBerxcnTpxATU0NAN/yoLy8PGzatAnr16/HBx980P+KRcBQTlL0dcMRbCx9Fx7Bg7uyFuH6+Kmj8ujE0dorZ7eCbv+iCi63F9+fmoRF16dyKyhd0mjtFaK+kmIo7/e/2eVyOa666ipcddVVPV63WCyoqqrq75clovNcE5ePDF0q1p/chLdKNuN480ksz72LRyeOEiEKOeZPS8b0vDgUfGLCh19V48sT9bh7diau41ZQIqIRpc8PehLR0DKE6vHT/IexOPMWnGwtw7MHeXTiaKOLUOOHt4zHMyuuhj5Cjdd2FOMPGwpxqp53RImIRgqGcqJhQC6T46bkmXjqmiegVUf6jk4s2cyjE0eZjAQd/v3+a/DAzbloaLXht2sPY/3OUnTaXWKXRkREV4iDiUTDyNiIMfjlNT/Fe5U7saf6U5RZTPjB+HuRpksWuzQaInKZDDMnjcU1OUZs9W8FPXyyAXfMTMeN+WOhkPNeCxHRcMR/exMNM0p5CO7IXIgnJv8ILq8bLxS+gvcrP4bH6xG7NBpCYWe3gj7k2wr65sdl+O3ar1FabRG7NCIi6oegTl/5xz/+EfQXPHDgAD7//HOcPDm8Zl55+goNRzaXHZvKtuFwQyFStEl4YPy9iA0zil3WgGOvXJ4gCPimtAnv7C1HS3s3po6LxT2zMxGlDRW7NBpi7BWi4Ejx9JWgQnlubm6fvqFMJmMoDwL/5UkD5ZuGo9hYugVurxuLsxZhxthpI+pkDvZKcLpdHnz41Sl88FU15HJg0fWp+P61yVCG8C9FRwv2ClFwhm0oP3ToUJ+/6dSpU/v8GTExlNNwZ+1uwxvFm1BiKceE6HFYPu4uaFWRYpc1INgrfdNkteOdvRUoLGsKbAWdlBE9ov6gRhfHXiEKzrAN5aMBQzmNBF7Bi/01B7DV9AFCFWrcN+5uTIwZL3ZZV4y90j9FVa14a3cZ6lpsmJgejaVzszCGW0FHNPYKUXAYyiWMoZxGktrOeqwr3oiazlrMGDsVd2YuQmiIWuyy+o290n9ujxd7vqnBts/9W0GvTcIt16dCo+bhWyMRe4UoOAzlEsZQTiONy+vG+5UfY3f1fsRoonD/+HuRpksRu6x+Ya9cubbObhTsN+GL4/XQRahwz6xMXJfHraAjDXuFKDgM5RLGUE4jVbmlEutPvgNrdxv+JeUm3Jw6Bwq5Quyy+oS9MnBMtW14a1cZquo6kJmgw/J52UgZMzKePSD2ClGwGMoljKGcRjK723d04qH6QqREJuH+vHsRN4yOTmSvDCyvIOCLY3Uo2G9Cp82FG/PH4o6Z6YgMU4ldGl0h9gpRcBjKJYyhnEaDwsZjeLtkM1xeNxZn3YLvjb1uWIwvsFcGh83hwrbPzdjzTQ00agVuvyEdsyZzK+hwxl4hCg5DuYQxlNNoYe1uw5sn/4mTrWWYEJ2LZbl3Q6eW9vgCe2VwnWnqxFu7y3HylAWJxggsn5eFnGSD2GVRP7BXiILDUC5hDOU0mngFLz6t+RJbTe9DrVBjWe5dmGTME7usS2KvDD5uBR0Z2CtEwWEolzCGchqN6roasK7obZzurMX18ddicdYihIZIL4SxV4bO2a2gHx6shkwGLJyeivlTk6AMGV4PB49W7BWi4DCUSxhDOY1Wbq8b71ftwq5TnyA61ID785YiXWJHJ7JXhl6zfyvoN2VNMOpDce+cLORnxgyLZxBGM/YKUXAYyiWMoZxGuwprFdYXb0Srw4p/Sb0JC1LnSuboRPaKeIrMrXhrl28r6IT0KCydk4X46HCxy6JLYK8QBYehXMIYyokAu9uBgrLt+Kr+ayRHJuKB8fciLjxW7LLYKyJze7zYW3gG2z6vhNPlxbxrk7CIW0Elib1CFByGcgljKCc659vG43i7ZDOcXhfuzFyIGxKmizq2wF6RhrYuJzZ/YsLnx+ugC1fh7tkZuC5vDOQcaZEM9gpRcBjKJYyhnKin849OHB+dg/ty7xHt6ET2irRU1rZjw64yVNW1IyNBi/vm5XArqESwV4iCw1AuYQzlRL0JgoD9Zw5ga8X7UClUWJZ7F/KNE4a8DvaK9HgFAV8cr8PmT0zosLkw078VVMutoKJirxAFh6FcwhjKiS6tvqsR64rfRnXHGVwXfw3uzrp1SI9OZK9Il83hxvYvqrD76xqEqhS4/YY0zJ6SwK2gImGvEAWHoVzCGMqJLs/tdePDqt3YeWofokINuH/8vcjQpw7J92avSN+Z5i68tavMvxU0HMvmZiM3hVtBhxp7hSg4DOUSxlBOFByT1Yx1xRvR6rDg+ymzsSBtLkLkg3sKB3tleBAEAYVlTdi4pwIt7Q5cmxuLJTdxK+hQYq8QBYehXMIYyomC53A7UFC+A1/WHUZSZAIeGH8vxoTHDdr3Y68ML06XBx8erMYHX52CDMDC6SmYPy2ZW0GHAHuFKDgM5RLGUE7Ud0eaTuCtkgI4PU7cnrkQNyZcPyhHJ7JXhqfmNv9W0NImxOhCsXROFvKzuBV0MLFXiILDUC5hDOVE/dPW3Y43S/6J4pZSjIvKxn3j7oZerRvQ78FeGd6Kza14a3c5apu7MCEtCkvncivoYGGvEAWHoVzCGMqJ+k8QBHx25ktsqXgfKrkSS3MXY3LsxAH7+uyV4c/t8WJf4Rls/bwKTpcH865JwqIZ3Ao60NgrRMFhKJcwhnKiK9fQ1Yi1xRtR3VGDaWOuxt3Zt0EzAEcnsldGjvYuJzbvN+HzY3XQhqtw16wMTJ/AraADhb1CFByGcgljKCcaGB6vBx+Yd2OneS+iQvX4wfh7kalPu6KvyV4Zearq2vHmx+e2gi6fl43UMVqxyxr22CtEwWEolzCGcqKBVdl2CuuKN6LF3op5KbOwMG1ev49OZK+MTF5BwIHj9Sj4pAIdNhdumDQWd97IraBXgr1CFByGcgljKCcaeA53NzaX78CBukNIihiL+/OWIr4fRyeyV0a2s1tB93xTA7WSW0GvBHuFKDgM5ec5duwY3n33XRw8eBC1tbXQ6/WYPHkyVq1ahZSUlO/8/IEDB/C3v/0NZWVl8Hq9SE9Px/33348FCxb0qx6GcqLBc7SpCG+VFKDb043bMhbgxsTrIZcFH7jYK6NDbXMX3t5dhiKzBQn+raDjuBW0T9grRMFhKD/PE088gcLCQsyfPx85OTloamrChg0bYLPZUFBQgIyMjEt+dt++ffjxj3+MyZMnY+HChQCA999/H4WFhfj973+Pu+++u8/1MJQTDa52Zwc2nPwnTrSUINeQhRXj7wn66ET2yughCAK+LW/Gxj3laG5z4JrcWCyZnYloHbeCBoO9QhQchvLzFBYWYsKECVCpzs0Oms1mLFq0CAsXLsTq1asv+dmHH34YpaWl2LNnT+DzTqcTc+bMQUpKCt58880+18NQTjT4BEHA57UHsaV8B0LkIViauxhTYq/6zs+xV0Yfp8uDjw5W433/VtAF01NwM7eCfif2ClFwpBjKRRvYmzJlSo9ADgCpqanIysqCyWS67Gc7Ozuh0+l6fF6lUkGn00GtVg9KvUR05WQyGW5IuA7/NnUVjJoYvH7iTawr3gi72y52aSQxKqUCt34vDc+unIarMqKx9bMqPPPaQRSWNYGPQhHRSCSpp2gEQUBzczMMhsvPEE6dOhXl5eV48cUXUV1djerqarz44oswm8146KGHhqhaIuqvuDAjfn71Y7g5dS6+bjiC5w69iHJLpdhlkQTF6DR47I6J+OW9+VArFfjLluN4YdNR1LV0iV0aEdGAktTpK9u2bcOvfvUrrF69Gnfcccclr7PZbHj66afx0UcfBe6YhIWF4aWXXsLMmTOHqlwiGgBlzZV4+eBaNHY2Y1HuPCyZcAuUCqXYZZEEuT1efHCgCm99VAKH04NFN6Rj6fdzEBbK/70Q0fAnmVBuMplwzz33ICcnB2+++SbklzkKy+124y9/+QvMZjPmzZsHj8eDTZs2obi4GGvXrsVVV333jOqFOFNOJB6HuxtbKnbgi9pDSIiIxwPjl2JsxJjA++wVOl97lxNbPjXhs6N1iAxX4W5uBQ1grxAFR4oz5ZII5U1NTVi6dCm8Xi/eeecdGI3Gy17/X//1Xzh+/DgKCgoC4d3lcuGWW26BwWDAxo0b+1wDQzmR+I41FWFDSQEcnm7clnEzwkPCsKNyJ6zdVujVetyaMR9Tx0wRu0ySiKq6dmzYVYbK2nZkjNVi2bxspMWP7q2g/LlCFBwphnLRZ8o7OjqwcuVKdHR0YM2aNd8ZyJ1OJwoKCjBr1qwed9OVSiVuuOEGHD9+HG63e7DLJqJBcJUxD/8+7ecYF5WFzeU78MbJTbB0WyEAsHRb8VbJZhyqLxS7TJKItHgtnl5xNX64cBya2hz4/bqv8Y8PTqK9yyl2aUREfSZqKO/u7sajjz4Ks9mMV199Fenp6d/5GavVCrfbDY/H0+s9t9sNt9vNJ/OJhrFIVQQemfgAwkI0ENCzl11eF7abPhKpMpIiuUyGGRPj8YcfXYfvT03CgRP1+PXfv8Kur0/D7fGKXR4RUdBEC+UejwerVq3CkSNH8NJLLyE/P/+i19XW1vY4IjE6OhparRa7du2Cy+UKvN7V1YV9+/YhOzsbSiUf+iEazmQyGWyXOCbR0m1FUUspHG7HEFdFUqZRh2DJTVn47Q+nIn2sFm/vLsdv/nEYJ82tYpdGRBSUELG+8erVq7F3717Mnj0bVqsV27ZtC7wXHh6OuXPnAgCeeuopHDp0CKWlpQAAhUKBhx56CC+++CKWLFmCW2+9FV6vFwUFBaivr8dTTz0lyu+HiAaWQa2Hpdt60fdeOfo65DI5kiMTkW3IQJY+Hem6VISGcE/BaBcfHY4n75mEI+XNeHtPOf608QiuyTFiyU1Z3ApKRJIm2oOeK1aswKFDhy76XkJCAvbu3dvjurOh/KwdO3Zg/fr1MJvNcDqdyMnJwcqVKzFv3rx+1cMHPYmk5VB9Id4q2QyX99zfiCnlStyTfRuiQg0os5hQbjXB3H4aXsELuUyOlMgkZBnSkW3IQLouFWqF6jLfgUY6p8uDjw5V44MvTwEAFlyXgvnTkqFSjtytoPy5QhQcKT7oKYnTV6SAoZxIeg7VF2K76aPLnr7S7XGi0mpGmdWEcosJpzpq4BW8UMgUSNEmIVufjixDBtJ1KVAxpI9KLW0ObNpXgcMljYjRhWLJTVmYkh0D2Qg8QpE/V4iCw1AuYQzlRNLVl15xuLtR2Wb230mvRPV5IT1Vm+Qfd8lAmi4FKi4pGlVOnrLgrd1lONPUhbxUA5bOzcbYmHCxyxpQ/LlCFByGcgljKCeSrivpFYfbAVObGeWWSpRZTKjuqIEAASEyBVJ1ycjSZyDbkI40bQo3iY4CHq8X+wrPYOtnVeh2eTDn6kTcOiMNYaGiPWI1oPhzhSg4DOUSxlBOJF0D2St2twMma5V/3KUSpzvO+EK6PARp2mRkGTKQrU9HqjaZIX0Ea7c5sWV/JT47WovIcBXuujED108c/ltB+XOFKDgM5RLGUE4kXYPZKzaXHaa2qsC4S01HLQQIUMpDkKZN8T84mokUbRKU8pFxN5XOqaprx1u7ymCqbUf6WC2WD/OtoPy5QhQchnIJYygnkq6h7BWby44KayXKrZUot5hQ01l3LqTrUpGtz0CWIR2p2iSEMKSPCF5BwJcn6lHwiQltXU5876p43HVjBrThw+/BYP5cIQoOQ7mEMZQTSZeYvdLlsqHCWoVyiwllVhPOdNYB8B3PmKFLDRzBmByZyJA+zNm73djxhRm7vj4NlVKB27+XhtlTEhCiEHX5dZ/w5wpRcBjKJYyhnEi6pNQrna6uQEgvt1YGQrpKrkS6LtV3uoshAymRiVDIR+552CNZXUsX3t5djhNVrUiICcfSuVkYnxoldllBkVKvEEkZQ7mEMZQTSZeUe6XT2YUKa2XgwdHarnoAgEqhQkZg3CUDyZEJDOnDiCAIOFLRjI17ytFkdeDqHCOW3JSJGJ1G7NIuS8q9QiQlDOUSxlBOJF3DqVc6nJ3+eXRfUK/vagAAqBUqZOjS/HfS05EUwZA+HLjcHnx06DTeP2CGAN9W0JslvBV0OPUKkZgYyiWMoZxIuoZzr7Q7O1Bh9Z/uYjGh3tYIAAhVqJGhT0OW3jeTnhSZALls+Mwujzat7b6toIdONiJaG4p752RiSrZRcltBh3OvEA0lhnIJYygnkq6R1Ctt3R2osJpQ5j/dpcHWBAAIVYQiU58WeHA0MWIsQ7oElfi3gtY0dWG8fytogoS2go6kXiEaTAzlEsZQTiRdI7lX2rrbUW6t9J+TbkKjrRkAoAnxhfSzM+kJEfEM6RLh8Xrxybe1ePfTSsltBR3JvUI0kBjKJYyhnEi6RlOvWLvbUG6pRLnVhDKLCU32FgBAWIgGmfp03510fQbGRoxhSBdZu82Jdz+txKdHahEZpsTiWRmYMTFe1K2go6lXiK4EQ7mEMZQTSddo7hWLwxpYZFRmrUTzeSE9S5+OLEMGsg0ZiA+PY0gXyan6DmzYVYaKM21Ii/dtBU0fK85W0NHcK0R9wVAuYQzlRNLFXjnH4rCizL/IqNxSiRZHKwAgXBnmC+l6X0gfEx7LkD6EBEHAV0UN2PRJBdo6fVtBF9+YAd0QbwVlrxAFh6FcwhjKiaSLvXJpLXakJoWZAAAgAElEQVQLyv0BvcxqQqvDAgCIUIYj03+yS5Y+HfHhcZI7KWQksne7seOAGbsOn4ZKKcdtM9Jw09WJQ7YVlL1CFByGcgljKCeSLvZK8FrsrYGTXcosJli6rQB8IT3LkIFsf1CPC4tlSB9EdS1deHtPOU5UtiI+OgzL5mUjbwi2grJXiILDUC5hDOVE0sVe6R9BENDisAROdimzmGDtbgMARKoi/Ce7+EZe4sKkd+b2cCcIAo5WtODtPWW+raDZ/q2g+sHbCspeIQoOQ7mEMZQTSRd7ZWAIgoBme2sgoJdZTGhztgMAtKrIHg+OxmpiGNIHiMvtwc5Dp/Hel2YIAnDztGQsuC5lULaCsleIgsNQLmEM5UTSxV4ZHIIgoMne4j/ZxbdxtM3p++9Zp9IGjl/MMqTDyJB+xS7cCrrkpkxcnTOwf0PBXiEKDkO5hDGUE0kXe2VoCIKARntzYB693FqJdn9I16t1yAo8OJqBGE0UQ3o/lVZbsGGXbyvouBQDls3NQoLx4j+k+4q9QhQchnIJYygnki72ijgEQUCDrenc6S4WEzpcnQAAg1ofmEfPNmQgOtTAkN4HZ7eCbv2sEvZu31bQ276XirBQ5RV9XfYKUXAYyiWMoZxIutgr0uAL6Y3+c9J9J7x0uroA+EJ6tiEjcMJLtGbwTxoZCTr8W0H3n90KemMGZlzV/62g7BWi4DCUSxhDOZF0sVekSRAE1HU1BDaOllsrAyE9OtSALP88erYhA1GhBpGrlbZT9R3YsLsMFTVtSIuPxLJ52cgYq+vz12GvEAWHoVzCGMqJpIu9Mjx4BS/quxoDRzCWWyvR5bIBAKJDowIPjmYbMmAI1YtcrfRcuBV0xsQxuGtWZp+2grJXiILDUC5hDOVE0sVeGZ68ghd1XQ2Bh0YrLJXocvtCeowmGtnnHcGoV/f9rvBIZe92470vzfj4kG8r6K0z0jAnyK2g7BWi4DCUSxhDOZF0sVdGBq/gRW1nvf/4xUqUWythd9sBAEZNdOCh0SxDOkM6gPpWG97eXY7jlS2+raBzs5GXdvlZffYKUXAYyiWMoZxIutgrI5NX8OJMZ13gnPQKaxXsbgcAIDYs5lxI16dDp9aKXK04BEHAUVMLNu4uR6PVjin+raDGS2wFZa8QBYehXMIYyomki70yOngFL2o6a33jLpZKVFir4PD4QnpcmDFwskumPgM6daTI1Q4tl9uLjw9XY8eBc1tBb74uBeoLtoKyV4iCw1AuYQzlRNLFXhmdPF4PajprUW71nZFuslbB4ekGAIwJiw3Mo2fp0xGpGpjlO1LX2u7APz8x4WBxA6K1aiy5KavHVlD2ClFwGMoljKGcSLrYKwT4QvrpzjO+RUZWX0jv9jgBAGPC45DtP4JxNIR031bQctQ0dWJcigF5qQbs+/YMWtu7EaVV484bMzA9b4zYZRJJFkO5hDGUE0kXe4UuxuP1oLrjTGAm3dRmhtMf0seGjwkcwZipT0eEKlzkageex+vF/iO12LS3HE53z59fqhA57r85l8Gc6BIYyiWMoZxIutgrFAyP14NTHTW+kG4xobLNDKfXBcAX0s9uHM3SpyNcGSZytQPn53/9ApaO7l6v68JVeOEnMwKjLUR0DkO5hDGUE0kXe4X6w+11o7qjJvDgqKnNDJfXBRlkGBsxxj/ukoEsfRrChnFIf2j13ku+p49QYXxqFPJSozA+1QBdhHoIKyOSLimG8pAhroWIiGhIhMhDkK5LRbouFfNT58DtdcPcfjowk/557VfYV/M5ZJAhMSI+cBc9U5+OMOXFjxyUomitGi3tve+Uh4eGIDtJj2OmFhw4UQ8ASDSG+0J6WhSyk/S9Tm8hIvHwTrkf75QTSRd7hQaDy+vGqfbTKLNUoNxSicr2U3B73b6QHjkWWfp0ZBsykKlPgyZEuiH9y6J6rPuwBE63N/Da+TPlXkFAdUMHiqpaUWy2oLzGCrdHQIhChswEHfLSfCE9OS4Sco660CghxTvlDOV+DOVE0sVeoaHg8rhgbq9GmbUS5RYTqtpOwS14IIMMSZEJgQdHM/Rp0ISEil1uD18W1WPLflNQp690uzwoP21FkbkVRVUW1DR1AgAiNErfSS5pvlGXGJ10/yBCdKUYyiWMoZxIutgrJAbn2ZBuMaHcaoK5rToQ0pMjE/0PjqYjQ5eKUImE9P70SluXE8XmVhRXtaLI3Aprp+8EmziDBuPTfPPouckGhIVy4pVGDoZyCWMoJ5Iu9gpJgdPjRFVbNcqtvtNdzO2n4RE8kMvkSI5MDIy7pOtSERoizgOVV9orgiCgtsUWCOil1VZ0uzyQy2RIGxvpf2A0CuljtQhRyAewcqKhxVAuYQzlRNLFXiEpcnqcqGw75T8nvRLm9mp4BS/kMjlSIhN9G0f1GUjXp0KtUA1JTQPdK26PF6YzbYFRF3N9OwQBCFUpkJt8btRlTFQYj16kYYWhXMIYyomki71Cw0G3x4nKNrPvdBeLCac6TgdCeqo2CVn6DP+d9BSoBimkD3avdDlcOGm2oNjsu5PeZHUAAKK06sDRi+NSDdCGDc0fQoj6i6FcwhjKiaSLvULDkcPd7Qvp/gdHT3XUwCt4oZApkKJN8s2k69ORrkuFSqEckO851L3SaLUHRl1Omi2wdbsBAMlxEb5Rl7QoZCfqoAzh0YskLQzlEsZQTiRd7BUaCRxuB0yBcRcTTnecgVfwIkSmQIo2GdkG30x6qjal3yFdzF7xegVU1bf7Q7oFpjNt8HgFKEPkyE7SBxYYJcZG8OhFEh1DuYQxlBNJF3uFRiK72wGTtQrlVt+4y+mOMxAgIESmQKouObBxNE2bDGWQIV1KveJwulFa7Tt6sdhsQW1zFwBAG6bEeP8Do3lpUTBEcssoDT2GcgljKCeSLvYKjQZ2tx0mqzlwBOPpjlpfSJeHIE2b7B93yUCqLhlKec/jCQ/VF2K76SNYu63Qq/W4NWM+po6ZItLv5OIsHd2BWfTiqla021wAgPjosMCoS06SHho1j16kwcdQfoFjx47h3XffxcGDB1FbWwu9Xo/Jkydj1apVSElJCepr7NixA+vWrUNFRQVUKhWys7Pxq1/9CldddVWfamEoJ5Iu9gqNRjaXHaa2Kl9It5hQ01kHAQKU8hCk6VKRrU9HliEDTfZmvFO6FS6vK/BZpVyJZbmLJRfMz/IKAmoaO1FstqDI3Iqy01a43F4o5DJkjNX6zkdPi0LaGC3kco660MBjKL/AE088gcLCQsyfPx85OTloamrChg0bYLPZUFBQgIyMjMt+/n/+53+wZs0a3HrrrZgyZQpsNhtKSkowd+5czJkzp0+1MJQTSRd7hQiwuWyosFahzGpCuaUSZ/wh/VIMaj1+P+PpIayw/1xuD8pr2vx30S2obuiAACBMHYJxKQb/EiMDYg1hYpdKIwRD+QUKCwsxYcIEqFTnjk4ym81YtGgRFi5ciNWrV1/2s8uWLcPLL7+MefPmXXEtDOVE0sVeIeqty2VDhbUSfz++/pLXXB07CbFhRsSFGREbFoPYsBhoQjRDWGX/dNicOHnKgqKqVhSbW9HS3g0AiNGFIu/sltEUAyI0A3NqDY0+Ugzlog5uTZnS+6/VUlNTkZWVBZPJdNnPrl+/HhMnTsS8efPg9Xpht9sRHh4+WKUSERFJSrgyDJOME2BQ62HptvZ6P0QeAnP7aRQ2HutxRz1SFYFYjRFxYTGIDYR1I2I00b1m1cUSGabC1HFxmDouDoIgoMFiR1FVK4qqWnGwuAH7j9RCBiA1PjJwPnpmoo5bRmlYk9yDnoIg4MYbb0Rubi7+/ve/X/K6adOmYeHChYiIiMAbb7wBm82GhIQErFq1Crfeemufvy/vlBNJF3uF6NIO1RfirZLNl5wpd3lcaHa0otHWhEZbMxpsTYH/3OHqDHxGBhmiQg2BkB4bFoM4je+fhlA95DJpBF63x4uqunb/XXQLKmvb4RUEqJUK5CTr/SHdgLEx4dwySpfEO+VB2L59OxoaGvCzn/3skte0tbXBarXi/fffh0KhwC9+8Qvo9Xps2LABv/zlL6HRaAZkpIWIiEjqzj7MeanTV5QKJeLD4xAfHtfrszaXHU32s0G92RfW7c2orDOj2+MMXBciD4FRE+0L65oY/ziML7BHKIc2/IYo5MhK1CMrUY/bbwBsDjdKq30PjBaZLThmKgcA6CNUgbvo41MN0EXw6EWSNkndKTeZTLjnnnuQk5ODN998E3L5xf9UXldXh1mzZgEANm3ahEmTJgEAnE4n5s2bB4PBgK1btw5V2URERCOKIAiwOtpR19GA2o5G1HU0oK6jEXUdjajvaoLH6wlcG67UID4yDvGRsYiPjMNY/z/jI4wIVYYOee2NFhuOlDUF/q/D5vvDRWq8FvnZRuRnG5GXHo1QleTuS9IoJ5lQ3tTUhKVLl8Lr9eKdd96B0Wi85LWtra2YPn06EhMTsWfPnh7vPffcc1i/fj2++eabPs2Yc3yFSLrYK0TBGYpe8Xg9aHVY0WjvPQ5z4Wy7TqXtOQ7jv9Meo4mGQq4Y1DoB39GL1Q0dgVGX8hor3B4BIQoZMhN0vodG06KQHBfJLaOjDMdXLqGjowMrV65ER0cH3n777csGcgDQ6/VQqVSIiYnp9V5MTAwEQUBnZycf/CQiIhpgCrkCxrBoGMOikRfd8z2nx4kme0vPcRhbM440HUeXyxa4Ti6TIyY0qkdgj/XPr+vU2gGbX5fLZEgdo0XqGC0WTk9Ft8uD8tO+LaNFVRZs3l+JzfsrEaFRYlyKAXlpvlGXGJ30T6ihkUf0UN7d3Y1HH30UZrMZa9euRXp6+nd+Ri6XY9y4cWhoaOj1Xn19PRQKBXQ63WCUS0RERJegUqiQEBGPhIj4Xu91urrQZGsOhPUGu++fpRZTj4dUVXIljP6wHqfpeUJMuPLKzilXKxWYkB6NCem+P020dTlR7N8wWmRuxeGSRgBAnEHjPxs9CrnJBoSFih6XaBQQ9X9lHo8Hq1atwpEjR/DKK68gPz//otfV1tbCbrf3WCY0f/58/Pd//ze++OILzJgxAwDQ2dmJDz/8EJMnT0Zo6NDPsREREdHFRSjDEaELR5qu58Zur+BFW3e7L6zbmwJ32Ws6zuBo0wl4BW/g2nBlWOCO+vkjMUZNNFQK1YXf8jvpwlWYnjcG0/PGQBAE1DZ3ochsQbG5FQeO12Nf4RnIZTKkjY30PzAahfSxWh69SINC1JnyZ599FuvXr8fs2bNx880393gvPDwcc+fOBQCsWLEChw4dQmlpaeB9u92OO++8Ew0NDXjggQeg1WqxefNmVFVVYd26dbj66qv7VAtnyomki71CFJyR1iturxst9lY0XnhCjK0Zbc72Htca1PrzliSdG4mJCtX3a37d7fHCdKYtMOpirm+HIAChKgVyk8+NuoyJCuPRi8MQZ8ovUFJSAgDYt28f9u3b1+O9hISEQCi/GI1Gg/Xr1+OPf/wj3nzzTTgcDuTl5eEf//hHnwM5ERERSU+IPARx4bGIC4/FxAvec7gdaLK3nHf+uu9O++GGb2F3OwLXKWQKxGiiAxtN4867065VRV4yUIco5MhJNiAn2YA7ZwJdDhdO+u+iF5lbcaSiGQAQpVUHjl4cl2qANqzvd+yJAAmdviI23iknki72ClFw2Cu+4xw7XV3nZtf9Z6832prQZG+B2+sOXKtWqAJnr8eGGc+70x4DTcjlH/ZstNoDs+gnzRbYun1fNzkuwjfqkhaF7EQdlCGDf8oM9Z0U75QzlPsxlBNJF3uFKDjslcvzCl5YHFbfnXV7z3GYVocFAs7lgEhVBGI1RsSF9XzYNEYTDaW856CB1yugqr7dH9ItMJ1pg8crQBkiR3aiLvDQaGJsBI9elAiGcgljKCeSLvYKUXDYK/3n8rjQ7GgNhHTfXXbfSEyHszNwnQwyRIUaej5s6h+JMYTqIZfJ4XC6UVrtO3qx2GxBbXMXAEAbpsR4/wOjeWlRMERyy6hYpBjKecYPERERjXpKhRLx4XGID4/r9Z7NZUfThQ+b2ptRWWdGt8cZuC5EHgKjJjowEpM23ohp18RB7dXCXONA8SkLiqta8VWx70jn+OiwwKhLTpIeGjVj2WjG/+8TERERXUaYUoMUZRJStEk9XhcEAe3OjnMPm/pHYuq7GnGi+SQ8gidwrSZEg9j4GOSnx0Dt1cLWHoqGehv2H+/E7m9qoJDLkDFW6xt1SYtC2hgt5HKOuowmDOVERERE/SCTyaBTa6FTa5FlyOjxnsfrQavDisYLZtcrrFWwdFt9FxmAEAOgU0RA6YlEU4cG75Ursf14ONReLcaNSUBeWgzyUg2INVzZ4iSSPoZyIiIiogGmkCtgDIuGMSwaedE933N6nGiyt/Q6e71RWQ9lpC1w3UlBhqI6DYQqX0hP1MYhJy4RU1JSMVYXBbmMS4xGEoZyIiIioiGkUqiQEBGPhIj4Xu91urrQZGv2n73ehGprPWo7GtHuMcMsM8HcAuxsAWReBcLleowJNyI9Oh5jwmMDD5+GK3lXfThiKCciIiKSiAhlOCJ04UjTpfR43St40Wpvw7GaUyiqPY3TnQ1oc1vQ7qhGeWcpZLJzJ8iFK8MQq+m53TQuzAijJhoqBZcbSRVDOREREZHEyWVyxIQZcFO2ATdl5wMAbA43SqstOG5uwokzZ9Da3QJ5aBfskQ40abtR31kCu/ebHl/HoNaftyTJ/0+NEVGheijkXHQkJoZyIiIiomEoLDQEk7ONmJxtBDAeLW0O/9norSgutqDT7gLkboyJFxA/VkCkwQmvqhPNjhYcbvgWdrcj8LUUMgViNNGBjaZx591p16oiIePSo0HHUE5EREQ0AkTrQjFz0ljMnDQWXkFAdUMHiqp8C4yOF1rh9qgQoohEZkIuZqUakJ4eClW4A032ZjTazy5MasLJ1jK4ve7A11UrVIGz12PDjOfdaY+BJkQj4u94ZOFGTz9u9CSSLvYKUXDYK3Qp3S4Pyk/7towWVVlQ0+TbUhqhUWJcigF5aVEYn2pAjE4Dr+CFxWHtcfb62RNiWh0WCDiXlyJVEYjVGBF3/jhMmBExmmgo5dK998uNnkREREQ05NRKBSakR2NCuu98xrYup2/MpaoVReZWHC5pBADEGTS+BUapUchNTse46OweX8flcaHZ0XruGEdbExpszTjechIddYcD18kgQ1SooefDpv6RGEOonsc5XgTvlPvxTjmRdLFXiILDXqH+EAQBtc1dKDJbUGxuRWm1Fd0uD+QyGdLGRiIvNQrjU6OQPlaLEMWlw7TdbfcHdf+ddXuz/yz2JnR7nIHrQuQhMGqiAyMxvnEYX2CPUIYPyfy6FO+UM5T7MZQTSRd7hSg47BUaCG6PF6YzbYFRF3N9OwQBCFUpkJt8btRlTFRYUAFaEAS0OzsCd9fPjcQ0o9neAo/gCVyrCdEEToSJ88+tx4YZYdTEIDREPWC/R4ZyCWMoJ5Iu9gpRcNgrNBi6HC6c9N9FLzK3osnqO7UlSqvG+FTfqMu4VAO0YX0/A93j9aDVYUXjBbPrDbYmWLqtPa7VqbS9zl6P1cQgRhPd5+McGcoljKGcSLrYK0TBYa/QUGi02HyjLlWtOHnKAlu376SW5LgI36hLWhSyE3VQhlzZuedOjxNN9pZeYb3R3oQuly1wnVwmR0xoVK+z12PDYqBTa3vMrx+qL8R200ewdluhV+txa8Z8TB0z5Yrq7AuG8iAwlBNJF3uFKDjsFRpqXq+Aqvp2/wOjFpjOtMHjFaAMkSM7URd4aDQxNgLyAZwV73R1oem8+fUG+7ng7vK6Atep5EoY/WHd43GjqLW0x7iMUq7EstzFQxbMGcqDwFBOJF3sFaLgsFdIbA6nG6XVVv8SIwtqm7sAANowJcb7HxjNS4uCIXLg5sPP5xW8aOtu94X1C0ZiGu3NF/2MQa3H72c8PSj1XIhHIhIRERHRoAtVhWBSZgwmZcYAACwd3YFZ9OKqVnxV3AAAiI8OC4y65CTpoVEPTCSVy+QwhOphCNUjB5k93nt8768u+pkLZ9fFwlBORERERIPCEKnGjInxmDExHl5BQE1jJ4rNFhSZW7H/aC12f1MDhVyGjLHawKhLanwkFPKBP8fcoNZfNIAb1PoB/179wfEVP46vEEkXe4UoOOwVGk5cbg/Ka9r8d9EtqG7ogABAow7B+BSDP6QbEGsIG5Dvd6i+EG+VbO4xcy6lmXLeKSciIiKiIacMUQTmzDEL6LA5cfKUBUVVrSg2t+KbsiYAQIwuFHlnt4ymGBChUfbr+50N3mKevnI5vFPuxzvlRNLFXiEKDnuFRgpBEFDfavONulS1oqTaAofTAxmA1PjIwPnomYm6y24ZvRQpnlPOO+VEREREJCkymQzx0eGIjw7HnKsT4fZ4UVXX7r+LbsGHX1Xj/S9PQaWUIyfJgLxU37hLQkx4UFtGpYihnIiIiIgkLUQhR1aiHlmJetx+A2BzuFFa7XtgtMhswfG9LQAAXYQKef676ONTDdBF9Dx68cuiemzZb0JrezeitGrceWMGpueNEeO31AtDORERERENK2GhIZicbcTkbCMAoLnNHhh1OWZqwYET9QCARGN44Gx0a2c3NnxcBqfbCwBoae/Gug9LAEASwZwz5X6cKSeSLvYKUXDYK0SAVxBQ3dARGHUpr7HC7bl0xovWqvGnx2YMSW2cKSciIiKiUUEukyF1jBapY7RYOD0V3S4Pyk9b8cKmoxe9vqW9e4grvLiBP5mdiIiIiEgi1EoFJqRHI1qrvuj7l3p9qDGUExEREdGId+eNGVCF9Iy+qhA57rwxQ6SKeuL4ChERERGNeGcf5uTpK0REREREIpqeNwbT88ZI8qFojq8QEREREYmMoZyIiIiISGQM5UREREREImMoJyIiIiISGUM5EREREZHIGMqJiIiIiETGUE5EREREJDKGciIiIiIikTGUExERERGJjBs9/eRy2aj6vkTDDXuFKDjsFaLgiNErl/ueMkEQhCGshYiIiIiILsDxFSIiIiIikTGUExERERGJjKGciIiIiEhkDOVERERERCJjKCciIiIiEhlDORERERGRyBjKiYiIiIhExlBORERERCQyhnIiIiIiIpExlBMRERERiSxE7AJGm8bGRqxfvx5Hjx7FiRMnYLPZsH79ekybNk3s0ogk49ixY3j33Xdx8OBB1NbWQq/XY/LkyVi1ahVSUlLELo9IMo4fP47/+7//Q3FxMVpaWhAZGYnc3Fw8/vjjmDJlitjlEUnaa6+9hueffx65ubnYtm2b2OUwlA+1qqoqvPbaa0hJSUFOTg6+/fZbsUsikpw1a9agsLAQ8+fPR05ODpqamrBhwwbcfvvtKCgoQEZGhtglEknC6dOn4fF4cPfdd8NoNKKjowM7duzAfffdh9deew0zZswQu0QiSWpqasLf/vY3hIWFiV1KgEwQBEHsIkaTzs5OuFwuGAwG7N69G48//jjvlBNdoLCwEBMmTIBKpQq8ZjabsWjRIixcuBCrV68WsToiabPb7Zg7dy4mTJiAV199VexyiCTp3/7t31BbWwtBENDe3i6JO+WcKR9iERERMBgMYpdBJGlTpkzpEcgBIDU1FVlZWTCZTCJVRTQ8aDQaREVFob29XexSiCTp2LFj2L59O37961+LXUoPDOVENCwIgoDm5mb+oZboIjo7O9Ha2orKykq88MILKCsrw/Tp08Uui0hyBEHA7373O9x+++0YN26c2OX0wJlyIhoWtm/fjoaGBvzsZz8TuxQiyXn66aexc+dOAIBSqcS9996LRx99VOSqiKRn69atqKiowF//+lexS+mFoZyIJM9kMuG3v/0trr76atx2221il0MkOY8//jiWLFmC+vp6bNu2DU6nEy6Xq9cYGNFo1tnZiT//+c/40Y9+hNjYWLHL6YXjK0QkaU1NTXjkkUeg0+nw0ksvQS7nv7aILpSTk4MZM2Zg8eLFeP3111FUVCS5eVkisf3tb3+DUqnEgw8+KHYpF8WfbkQkWR0dHVi5ciU6OjqwZs0aGI1GsUsikjylUok5c+bg448/hsPhELscIklobGzEunXrsGzZMjQ3N6OmpgY1NTXo7u6Gy+VCTU0N2traRK2R4ytEJEnd3d149NFHYTabsXbtWqSnp4tdEtGw4XA4IAgCurq6EBoaKnY5RKJraWmBy+XC888/j+eff77X+3PmzMHKlSvxi1/8QoTqfBjKiUhyPB4PVq1ahSNHjuCVV15Bfn6+2CURSVJrayuioqJ6vNbZ2YmdO3ciPj4e0dHRIlVGJC2JiYkXfbjzxRdfhM1mw9NPP43U1NShL+w8DOUieOWVVwAgcN7ytm3b8M0330Cr1eK+++4TszQiSVi9ejX27t2L2bNnw2q19ljqEB4ejrlz54pYHZF0rFq1Cmq1GpMnT4bRaERdXR22bNmC+vp6vPDCC2KXRyQZkZGRF/3ZsW7dOigUCkn8XOFGTxHk5ORc9PWEhATs3bt3iKshkp4VK1bg0KFDF32PfUJ0TkFBAbZt24aKigq0t7cjMjIS+fn5eOihhzB16lSxyyOSvBUrVkhmoydDORERERGRyHj6ChERERGRyBjKiYiIiIhExlBORERERCQyhnIiIiIiIpExlBMRERERiYyhnIiIiIhIZAzlREREREQiYygnIiLRrFixAjfddJPYZRARiS5E7AKIiGhgHTx4ED/4wQ8u+b5CoUBxcfEQVkRERN+FoZyIaIS65ZZbMHPmzF6vy+X8S1IiIqlhKCciGqHGjx+P2267TewyiIgoCLxdQkQ0StXU1CAnJwcvv/wy3nvvPSxatAgTJ07ErFmz8PLLL8Ptdvf6TElJCR5//HFMmzYNEydOxIIFC/Daa6/B4/H0urapqQm///3vMWfOHEyYMAHTp0/Hgw8+iC+++KLXtQ0NDXjyySdx7bXXYgPXyWUAAAQMSURBVNKkSfjhD3+IqqqqQfl9ExFJEe+UExGNUHa7Ha2trb1eV6lUiIiICPx67969OH36NJYvX46YmBjs3bsXf/nLX1BbW4s//OEPgeuOHz+OFStWICQkJHDtvn378Pzzz6OkpAR//vOfA9fW1NRg6dKlaGlpwW233YYJEybAbrfj6NGjOHDgAGbMmBG41maz4b777sOkSZPws5/9DDU1NVi/fj0ee+wxvPfee1AoFIP03xARkXQwlBMRjVAvv/wyXn755V6vz5o1C6+++mrg1yUlJSgoKEBeXh4A4L777sNPfvITbNmyBUuWLEF+fj4A4Nlnn4XT6cTGjRuRm5sbuHbVqlV47733cNddd2H69OkAgN/85jdobGzEmjVrcMMNN/T4/l6vt8evLRYLfvjDH2LlypWB16KiovCnP/0JBw4c6PV5IqKRiKGciGiEWrJkCebPn9/r9aioqB6/vv766wOBHABkMhkefvhh7N69G7t27UJ+fj5aWlrw7bffYt68eYFAfvbaH//4x/joo4+wa9cuTJ8+HVarFZ999hluuOGGiwbqCx80lcvlvU6Lue666wAAp06dYignolGBoZyIaIRKSUnB9ddf/53XZWRk9HotMzMTAHD69GkAvnGU818/X3p6OuRyeeDa6upqCIKA8ePHB1VnbGws1Gp1j9f0ej0AwGq1BvU1iIiGOz7oSUREorrczLggCENYCRGReBjKiYhGOZPJ1Ou1iooKAEBSUhIAIDExscfr56usrITX6w1cm5ycDJlMhpMnTw5WyUREIw5DORHRKHfgwAEUFRUFfi0IAtasWQMAmDt3LgAgOjoakydPxr59+1BWVtbj2r///e8AgHnz5gHwjZ7MnDkTn376KQ4cONDr+/HuNxFRb5wpJyIaoYqLi7Ft27aLvnc2bANAbm4u7r//fixfvhxGoxF79uzBgQMHcNttt2Hy5MmB65555hmsWLECy5cvx7Jly2A0GrFv3z58/vnnuOWWWwInrwDAf/zHf6C4uBgrV67E7bffjry8PHR3d+Po0aNISEjAL3/5y8H7jRMRDUMM5UREI9R7772H995776Lvffzxx4FZ7ptuuglpaWl49dVXUVVVhejoaDz22GN47LHHenxm4sSJ2LhxI/73f/8Xb7/9Nmw2G5KSkvCLX/wCDz30UI9rk5KSsHnzZvz1r3/Fp59+im3btkGr1SI3NxdLliwZnN8wEdEwJhP494hERKNSTU0N5syZg5/85Cf46U9/KnY5RESjGmfKiYiIiIhExlBORERERCQyhnIiIiIiIpFxppyIiIiISGS8U05EREREJDKGciIiIiIikTGUExERERGJjKGciIiIiEhkDOVERERERCJjKCciIiIiEtn/B6Bwaobtf5RzAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Tuto Camembert"
      ],
      "metadata": {
        "id": "1OmVjPgLniQO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "https://colab.research.google.com/drive/1htht86fSGmznsUGSb1Wbj7kwsgowTe1M#scrollTo=-qSoutA_OZHF"
      ],
      "metadata": {
        "id": "b16yfEgfnqRX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install transformers plotly==5.8.0 pyyaml==5.4.1 datasets pytorch-lightning > /dev/null 2>&1"
      ],
      "metadata": {
        "id": "oAKKLmd-pa0Y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import pytorch_lightning as pl\n",
        "\n",
        "from transformers import AutoModelForSequenceClassification, CamembertForMaskedLM, AutoTokenizer, AutoConfig\n"
      ],
      "metadata": {
        "id": "QHz2K-S4pTQA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class LightningModel(pl.LightningModule):\n",
        "    def __init__(self, model_name, num_labels, lr, weight_decay, from_scratch=False):\n",
        "        super().__init__()\n",
        "        self.save_hyperparameters()\n",
        "        if from_scratch:\n",
        "            # Si `from_scratch` est vrai, on charge uniquement la config (nombre de couches, hidden size, etc.) et pas les poids du modèle \n",
        "            config = AutoConfig.from_pretrained(\n",
        "                model_name, num_labels=num_labels\n",
        "            )\n",
        "            self.model = AutoModelForSequenceClassification.from_config(config)\n",
        "        else:\n",
        "            # Cette méthode permet de télécharger le bon modèle pré-entraîné directement depuis le Hub de HuggingFace sur lequel sont stockés de nombreux modèles\n",
        "            self.model = AutoModelForSequenceClassification.from_pretrained(\n",
        "                model_name, num_labels=num_labels\n",
        "            )\n",
        "        self.lr = lr\n",
        "        self.weight_decay = weight_decay\n",
        "        self.num_labels = self.model.num_labels\n",
        "\n",
        "    def forward(self, batch):\n",
        "        return self.model(\n",
        "            input_ids=batch[\"input_ids\"],\n",
        "            attention_mask=batch[\"attention_mask\"]\n",
        "        )\n",
        "\n",
        "    def training_step(self, batch):\n",
        "        out = self.forward(batch)\n",
        "\n",
        "        logits = out.logits\n",
        "        # -------- MASKED --------\n",
        "        loss_fn = torch.nn.CrossEntropyLoss()\n",
        "        loss = loss_fn(logits.view(-1, self.num_labels), batch[\"labels\"].view(-1))\n",
        "\n",
        "        # ------ END MASKED ------\n",
        "\n",
        "        self.log(\"train/loss\", loss)\n",
        "\n",
        "        return loss\n",
        "\n",
        "    def validation_step(self, batch, batch_index):\n",
        "        labels = batch[\"labels\"]\n",
        "        out = self.forward(batch)\n",
        "\n",
        "        preds = torch.max(out.logits, -1).indices\n",
        "        # -------- MASKED --------\n",
        "        acc = (batch[\"labels\"] == preds).float().mean()\n",
        "        # ------ END MASKED ------\n",
        "        self.log(\"valid/acc\", acc)\n",
        "\n",
        "        f1 = f1_score(batch[\"labels\"].cpu().tolist(), preds.cpu().tolist(), average=\"macro\")\n",
        "        self.log(\"valid/f1\", f1)\n",
        "\n",
        "    def predict_step(self, batch, batch_idx):\n",
        "        \"\"\"La fonction predict step facilite la prédiction de données. Elle est \n",
        "        similaire à `validation_step`, sans le calcul des métriques.\n",
        "        \"\"\"\n",
        "        out = self.forward(batch)\n",
        "\n",
        "        return torch.max(out.logits, -1).indices\n",
        "\n",
        "    def configure_optimizers(self):\n",
        "        return torch.optim.AdamW(\n",
        "            self.model.parameters(), lr=self.lr, weight_decay=self.weight_decay\n",
        "        )"
      ],
      "metadata": {
        "id": "zXUr1mTSpEV-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "num_labels = len(set(all_intents))\n",
        "lr = 1e-5\n",
        "weight_decay=0.\n",
        "\n",
        "lightning_model = LightningModel(\"camembert-base\", num_labels, lr, weight_decay)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lbwSnR51pyMW",
        "outputId": "50779f07-4fc4-4484-a6b9-f5379394760b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at camembert-base were not used when initializing CamembertForSequenceClassification: ['lm_head.layer_norm.weight', 'roberta.pooler.dense.bias', 'roberta.pooler.dense.weight', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.dense.bias']\n",
            "- This IS expected if you are initializing CamembertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing CamembertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of CamembertForSequenceClassification were not initialized from the model checkpoint at camembert-base and are newly initialized: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%load_ext tensorboard"
      ],
      "metadata": {
        "id": "tAqVgMCXq4A3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "%tensorboard --logdir /content/lightning_logs"
      ],
      "metadata": {
        "id": "clZXti65Y3Zb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_checkpoint = pl.callbacks.ModelCheckpoint(monitor=\"valid/acc\", mode=\"max\")\n",
        "\n",
        "camembert_trainer = pl.Trainer(\n",
        "    max_epochs=20,\n",
        "    gpus=1,\n",
        "    callbacks=[\n",
        "        pl.callbacks.EarlyStopping(monitor=\"valid/acc\", patience=4, mode=\"max\"),\n",
        "        model_checkpoint,\n",
        "    ]\n",
        ")"
      ],
      "metadata": {
        "id": "eOBwBQ4RY7GH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "camembert_trainer.fit(lightning_model, train_dataloaders=train_dataloader, val_dataloaders=val_dataloader)"
      ],
      "metadata": {
        "id": "8_wuR90aY-76"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "lightning_model = LightningModel.load_from_checkpoint(checkpoint_path=model_checkpoint.best_model_path)"
      ],
      "metadata": {
        "id": "uuYbODJdZBS4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def get_preds(model, tokenizer, sentence):\n",
        "    tokenized_sentence = tokenizer(sentence, return_tensors=\"pt\")\n",
        "    input_ids, attention_mask = tokenized_sentence.input_ids, tokenized_sentence.attention_mask\n",
        "\n",
        "    out = model(\n",
        "        input_ids=tokenized_sentence.input_ids,\n",
        "        attention_mask=tokenized_sentence.attention_mask\n",
        "    )\n",
        "\n",
        "    logits = out.logits\n",
        "\n",
        "    probas = torch.softmax(logits, -1).squeeze()\n",
        "\n",
        "    pred = torch.argmax(probas)\n",
        "\n",
        "    return ID_TO_LABEL[pred], probas[pred].item()"
      ],
      "metadata": {
        "id": "138__eOWZOWJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "KIpLZJf8ns_w"
      }
    }
  ]
}