HopeSpeechEDIBERT_English.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copy of HopeSpeechEDIBERT_English.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "e981309df3474d39b1e2c728951a25e7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_482828427b7d4747a4269cb580f69fb2",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_8a650f7e34b046d985f310a1c8009b12",
              "IPY_MODEL_5e3c77b0aa6e47719ff3f3e28a24d782"
            ]
          }
        },
        "482828427b7d4747a4269cb580f69fb2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "8a650f7e34b046d985f310a1c8009b12": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_f8f87a8e5d2f43dca10e881ade0a649c",
            "_dom_classes": [],
            "description": "Downloading: 100%",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 231508,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 231508,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_48c0d21b62a34811b9baca0619f313ff"
          }
        },
        "5e3c77b0aa6e47719ff3f3e28a24d782": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_c122d5060e2b45909331975ba436d7f1",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 232k/232k [00:00&lt;00:00, 3.18MB/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_a1237c1c63624dacb6f3f5f461725d78"
          }
        },
        "f8f87a8e5d2f43dca10e881ade0a649c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "initial",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "48c0d21b62a34811b9baca0619f313ff": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "c122d5060e2b45909331975ba436d7f1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "a1237c1c63624dacb6f3f5f461725d78": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "838d72a9cc3b453392405ce72048f461": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_23f0f52c8dcc4984882261d9ee749fa3",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_9ef4cf142e3b4f4a95f182a3bccf7123",
              "IPY_MODEL_3c53de0bfddb4891bc8446adf7608731"
            ]
          }
        },
        "23f0f52c8dcc4984882261d9ee749fa3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "9ef4cf142e3b4f4a95f182a3bccf7123": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_7b8901afe9e2498b9467c87f08cf81eb",
            "_dom_classes": [],
            "description": "Downloading: 100%",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 433,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 433,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_3ca33d0a7b9e475e880112897ae2edca"
          }
        },
        "3c53de0bfddb4891bc8446adf7608731": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_04b8464f597d43e88866adc81c1249bd",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 433/433 [00:00&lt;00:00, 2.17kB/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_69da6f2fbdd24af099379cc7534f945d"
          }
        },
        "7b8901afe9e2498b9467c87f08cf81eb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "initial",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "3ca33d0a7b9e475e880112897ae2edca": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "04b8464f597d43e88866adc81c1249bd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "69da6f2fbdd24af099379cc7534f945d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "18060a5a10bc4ee5bb23fdb0297a7b2a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "state": {
            "_view_name": "HBoxView",
            "_dom_classes": [],
            "_model_name": "HBoxModel",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "box_style": "",
            "layout": "IPY_MODEL_42c93e704e9f4b4ca7931ab8cbe12e15",
            "_model_module": "@jupyter-widgets/controls",
            "children": [
              "IPY_MODEL_2d408db371064eefbdc50755ab709a85",
              "IPY_MODEL_23da42ad57ac4982a378acfa93e7e926"
            ]
          }
        },
        "42c93e704e9f4b4ca7931ab8cbe12e15": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "2d408db371064eefbdc50755ab709a85": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "state": {
            "_view_name": "ProgressView",
            "style": "IPY_MODEL_06411a62e17b4cbba04dac587dd12f19",
            "_dom_classes": [],
            "description": "Downloading: 100%",
            "_model_name": "FloatProgressModel",
            "bar_style": "success",
            "max": 440473133,
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": 440473133,
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "orientation": "horizontal",
            "min": 0,
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_4602db38e1ff4cdf86ca5dffdde78f45"
          }
        },
        "23da42ad57ac4982a378acfa93e7e926": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "state": {
            "_view_name": "HTMLView",
            "style": "IPY_MODEL_edeb18092ab840c2bf1bfef66d78cd46",
            "_dom_classes": [],
            "description": "",
            "_model_name": "HTMLModel",
            "placeholder": "​",
            "_view_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "value": " 440M/440M [00:20&lt;00:00, 22.0MB/s]",
            "_view_count": null,
            "_view_module_version": "1.5.0",
            "description_tooltip": null,
            "_model_module": "@jupyter-widgets/controls",
            "layout": "IPY_MODEL_63980dd638314cf283f0bae6e9bad739"
          }
        },
        "06411a62e17b4cbba04dac587dd12f19": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "ProgressStyleModel",
            "description_width": "initial",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "bar_color": null,
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "4602db38e1ff4cdf86ca5dffdde78f45": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        },
        "edeb18092ab840c2bf1bfef66d78cd46": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_view_name": "StyleView",
            "_model_name": "DescriptionStyleModel",
            "description_width": "",
            "_view_module": "@jupyter-widgets/base",
            "_model_module_version": "1.5.0",
            "_view_count": null,
            "_view_module_version": "1.2.0",
            "_model_module": "@jupyter-widgets/controls"
          }
        },
        "63980dd638314cf283f0bae6e9bad739": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "state": {
            "_view_name": "LayoutView",
            "grid_template_rows": null,
            "right": null,
            "justify_content": null,
            "_view_module": "@jupyter-widgets/base",
            "overflow": null,
            "_model_module_version": "1.2.0",
            "_view_count": null,
            "flex_flow": null,
            "width": null,
            "min_width": null,
            "border": null,
            "align_items": null,
            "bottom": null,
            "_model_module": "@jupyter-widgets/base",
            "top": null,
            "grid_column": null,
            "overflow_y": null,
            "overflow_x": null,
            "grid_auto_flow": null,
            "grid_area": null,
            "grid_template_columns": null,
            "flex": null,
            "_model_name": "LayoutModel",
            "justify_items": null,
            "grid_row": null,
            "max_height": null,
            "align_content": null,
            "visibility": null,
            "align_self": null,
            "height": null,
            "min_height": null,
            "padding": null,
            "grid_auto_rows": null,
            "grid_gap": null,
            "max_width": null,
            "order": null,
            "_view_module_version": "1.2.0",
            "grid_template_areas": null,
            "object_position": null,
            "object_fit": null,
            "grid_auto_columns": null,
            "margin": null,
            "display": null,
            "left": null
          }
        }
      }
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "DEfSbAA4QHas"
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "# Get the GPU device name.\n",
        "device_name = tf.test.gpu_device_name()\n",
        "\n",
        "# The device name should look like the following:\n",
        "if device_name == '/device:GPU:0':\n",
        "    print('Found GPU at: {}'.format(device_name))\n",
        "else:\n",
        "    raise SystemError('GPU device not found')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cqG7FzRVFEIv"
      },
      "source": [
        "In order for torch to use the GPU, we need to identify and specify the GPU as the device. Later, in our training loop, we will load data onto the device. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oYsV4H8fCpZ-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ba837ae6-19bd-47a3-ffad-ed36919bd22d"
      },
      "source": [
        "import torch\n",
        "\n",
        "# If there's a GPU available...\n",
        "if torch.cuda.is_available():    \n",
        "\n",
        "    # Tell PyTorch to use the GPU.    \n",
        "    device = torch.device(\"cuda\")\n",
        "\n",
        "    print('There are %d GPU(s) available.' % torch.cuda.device_count())\n",
        "\n",
        "    print('We will use the GPU:', torch.cuda.get_device_name(0))\n",
        "\n",
        "# If not...\n",
        "else:\n",
        "    print('No GPU available, using the CPU instead.')\n",
        "    device = torch.device(\"cpu\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "There are 1 GPU(s) available.\n",
            "We will use the GPU: Tesla T4\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ElsnSNUridI"
      },
      "source": [
        "## 1.2. Installing the Hugging Face Library\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G_N2UDLevYWn"
      },
      "source": [
        "\n",
        "Next, let's install the [transformers](https://github.com/huggingface/transformers) package from Hugging Face which will give us a pytorch interface for working with BERT. (This library contains interfaces for other pretrained language models like OpenAI's GPT and GPT-2.) We've selected the pytorch interface because it strikes a nice balance between the high-level APIs (which are easy to use but don't provide insight into how things work) and tensorflow code (which contains lots of details but often sidetracks us into lessons about tensorflow, when the purpose here is BERT!).\n",
        "\n",
        "At the moment, the Hugging Face library seems to be the most widely accepted and powerful pytorch interface for working with BERT. In addition to supporting a variety of different pre-trained transformer models, the library also includes pre-built modifications of these models suited to your specific task. For example, in this tutorial we will use `BertForSequenceClassification`.\n",
        "\n",
        "The library also includes task-specific classes for token classification, question answering, next sentence prediciton, etc. Using these pre-built classes simplifies the process of modifying BERT for your purposes.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0NmMdkZO8R6q",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1c159773-97c1-4ae1-cf54-4e48a0726e01"
      },
      "source": [
        "!pip install transformers"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting transformers\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/50/0c/7d5950fcd80b029be0a8891727ba21e0cd27692c407c51261c3c921f6da3/transformers-4.1.1-py3-none-any.whl (1.5MB)\n",
            "\u001b[K     |████████████████████████████████| 1.5MB 11.6MB/s \n",
            "\u001b[?25hRequirement already satisfied: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from transformers) (0.8)\n",
            "Collecting sacremoses\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz (883kB)\n",
            "\u001b[K     |████████████████████████████████| 890kB 54.4MB/s \n",
            "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers) (2019.12.20)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.6/dist-packages (from transformers) (3.0.12)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.6/dist-packages (from transformers) (20.8)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from transformers) (1.19.4)\n",
            "Collecting tokenizers==0.9.4\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/0f/1c/e789a8b12e28be5bc1ce2156cf87cb522b379be9cadc7ad8091a4cc107c4/tokenizers-0.9.4-cp36-cp36m-manylinux2010_x86_64.whl (2.9MB)\n",
            "\u001b[K     |████████████████████████████████| 2.9MB 52.2MB/s \n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.6/dist-packages (from transformers) (4.41.1)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (1.15.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers) (1.0.0)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2020.12.5)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers) (2.4.7)\n",
            "Building wheels for collected packages: sacremoses\n",
            "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893261 sha256=a47e4cd6e374a3a089011f724c38859bee8f322ac5fef0ec1a9253e463276d0b\n",
            "  Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45\n",
            "Successfully built sacremoses\n",
            "Installing collected packages: sacremoses, tokenizers, transformers\n",
            "Successfully installed sacremoses-0.0.43 tokenizers-0.9.4 transformers-4.1.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lxddqmruamSj"
      },
      "source": [
        "The code in this notebook is actually a simplified version of the [run_glue.py](https://github.com/huggingface/transformers/blob/master/examples/run_glue.py) example script from huggingface.\n",
        "\n",
        "`run_glue.py` is a helpful utility which allows you to pick which GLUE benchmark task you want to run on, and which pre-trained model you want to use (you can see the list of possible models [here](https://github.com/huggingface/transformers/blob/e6cff60b4cbc1158fbd6e4a1c3afda8dc224f566/examples/run_glue.py#L69)). It also supports using either the CPU, a single GPU, or multiple GPUs. It even supports using 16-bit precision if you want further speed up.\n",
        "\n",
        "Unfortunately, all of this configurability comes at the cost of *readability*. In this Notebook, we've simplified the code greatly and added plenty of comments to make it clear what's going on. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pMPmX1mwHoyw",
        "outputId": "2a67dca3-9353-45a1-c32f-26e1c7afd426"
      },
      "source": [
        "from google.colab import drive \r\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t58wmyn7H7QN"
      },
      "source": [
        "from os import getcwd , listdir\r\n",
        "from os.path import join \r\n",
        "curr_dir = getcwd()\r\n",
        "drive_dir = join(curr_dir,'drive','MyDrive')\r\n",
        "dataset_dir = join(drive_dir,'Codalab','HopeSpeech Detection For Equality,DiversityAnd Inclusion','Dataset')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NpOgPNg8I7qL"
      },
      "source": [
        "import pandas as pd\r\n",
        "train_df_path = join(dataset_dir,'english_hope_train.csv')\r\n",
        "train_df = pd.read_csv(train_df_path)\r\n",
        "dev_df_path = join(dataset_dir,'english_hope_dev.csv')\r\n",
        "dev_df = pd.read_csv(dev_df_path)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_UkeC7SG2krJ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 377
        },
        "outputId": "225a8d2c-397a-4787-e7c2-6ef62e965306"
      },
      "source": [
        "# Report the number of sentences.\n",
        "print('Number of training sentences: {:,}\\n'.format(train_df.shape[0]))\n",
        "\n",
        "# Display 10 random rows from the data.\n",
        "train_df.sample(10)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of training sentences: 22,762\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>14495</th>\n",
              "      <td>Everyone is entitled to their opinion</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13167</th>\n",
              "      <td>@Wario &amp; Waluigi productions black lives matter</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2513</th>\n",
              "      <td>@Toxic Sheeple yes you are</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22470</th>\n",
              "      <td>“That would just uh be pure mean”  This girl</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1868</th>\n",
              "      <td>It wasn’t erected as a symbol of slavery... it...</td>\n",
              "      <td>Hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19199</th>\n",
              "      <td>How does he tell them the same thing so many t...</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17825</th>\n",
              "      <td>What an inspiring lady. G-day from Australia. ...</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9850</th>\n",
              "      <td>@mawairu nit doesnt mean the entire world is d...</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22292</th>\n",
              "      <td>@oomz1975 Don't remember them hurling bikes at...</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20858</th>\n",
              "      <td>is this buzzfeed?</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                    text            label\n",
              "14495              Everyone is entitled to their opinion  Non_hope_speech\n",
              "13167    @Wario & Waluigi productions black lives matter  Non_hope_speech\n",
              "2513                          @Toxic Sheeple yes you are  Non_hope_speech\n",
              "22470       “That would just uh be pure mean”  This girl  Non_hope_speech\n",
              "1868   It wasn’t erected as a symbol of slavery... it...      Hope_speech\n",
              "19199  How does he tell them the same thing so many t...  Non_hope_speech\n",
              "17825  What an inspiring lady. G-day from Australia. ...  Non_hope_speech\n",
              "9850   @mawairu nit doesnt mean the entire world is d...  Non_hope_speech\n",
              "22292  @oomz1975 Don't remember them hurling bikes at...  Non_hope_speech\n",
              "20858                                  is this buzzfeed?  Non_hope_speech"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4SMZ5T5Imhlx"
      },
      "source": [
        "\n",
        "\n",
        "Let's extract the sentences and labels of our training set as numpy ndarrays."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XT-tNewxVXtK"
      },
      "source": [
        "from sklearn.preprocessing import LabelEncoder\r\n",
        "le = LabelEncoder()\r\n",
        "train_df['label'] = le.fit_transform(train_df['label'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 402
        },
        "id": "OODHfIngVmS9",
        "outputId": "00f91554-77ea-4652-b46a-d5b7b1fe9cdd"
      },
      "source": [
        "train_df"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>these tiktoks radiate gay chaotic energy and i...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>@Champions Again He got killed for using false...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>It's not that all lives don't matter</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Is it really that difficult to understand? Bla...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Whenever we say black isn't that racists?  Why...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22757</th>\n",
              "      <td>It's a load of bollocks every life matters sim...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22758</th>\n",
              "      <td>no say it because all lives matter! deku would...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22759</th>\n",
              "      <td>God says her life matters</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22760</th>\n",
              "      <td>This video is just shit. A bunch of whiny ass ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22761</th>\n",
              "      <td>Mc Fortnut2821 she did 4 months ago in west ch...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>22762 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                    text  label\n",
              "0      these tiktoks radiate gay chaotic energy and i...      1\n",
              "1      @Champions Again He got killed for using false...      1\n",
              "2                   It's not that all lives don't matter      1\n",
              "3      Is it really that difficult to understand? Bla...      1\n",
              "4      Whenever we say black isn't that racists?  Why...      1\n",
              "...                                                  ...    ...\n",
              "22757  It's a load of bollocks every life matters sim...      1\n",
              "22758  no say it because all lives matter! deku would...      1\n",
              "22759                          God says her life matters      1\n",
              "22760  This video is just shit. A bunch of whiny ass ...      1\n",
              "22761  Mc Fortnut2821 she did 4 months ago in west ch...      1\n",
              "\n",
              "[22762 rows x 2 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GuE5BqICAne2"
      },
      "source": [
        "# Get the lists of sentences and their labels.\n",
        "sentences = train_df.text.values\n",
        "labels = train_df.label.values"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ex5O1eV-Pfct"
      },
      "source": [
        "# 3. Tokenization & Input Formatting\n",
        "\n",
        "In this section, we'll transform our dataset into the format that BERT can be trained on."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-8kEDRvShcU5"
      },
      "source": [
        "## 3.1. BERT Tokenizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bWOPOyWghJp2"
      },
      "source": [
        "\n",
        "To feed our text to BERT, it must be split into tokens, and then these tokens must be mapped to their index in the tokenizer vocabulary.\n",
        "\n",
        "The tokenization must be performed by the tokenizer included with BERT--the below cell will download this for us. We'll be using the \"uncased\" version here.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z474sSC6oe7A",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 82,
          "referenced_widgets": [
            "e981309df3474d39b1e2c728951a25e7",
            "482828427b7d4747a4269cb580f69fb2",
            "8a650f7e34b046d985f310a1c8009b12",
            "5e3c77b0aa6e47719ff3f3e28a24d782",
            "f8f87a8e5d2f43dca10e881ade0a649c",
            "48c0d21b62a34811b9baca0619f313ff",
            "c122d5060e2b45909331975ba436d7f1",
            "a1237c1c63624dacb6f3f5f461725d78"
          ]
        },
        "outputId": "81012ca9-a48e-47dd-a376-497621476a67"
      },
      "source": [
        "from transformers import BertTokenizer\n",
        "\n",
        "# Load the BERT tokenizer.\n",
        "print('Loading BERT tokenizer...')\n",
        "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loading BERT tokenizer...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e981309df3474d39b1e2c728951a25e7",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=231508.0, style=ProgressStyle(descripti…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dFzmtleW6KmJ"
      },
      "source": [
        "Let's apply the tokenizer to one sentence just to see the output.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dLIbudgfh6F0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1cdd7304-6d95-41f8-f371-2b9766d5ebce"
      },
      "source": [
        "# Print the original sentence.\n",
        "print(' Original: ', sentences[0])\n",
        "\n",
        "# Print the sentence split into tokens.\n",
        "print('Tokenized: ', tokenizer.tokenize(sentences[0]))\n",
        "\n",
        "# Print the sentence mapped to token ids.\n",
        "print('Token IDs: ', tokenizer.convert_tokens_to_ids(tokenizer.tokenize(sentences[0])))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            " Original:  these tiktoks radiate gay chaotic energy and i love it\n",
            "Tokenized:  ['these', 'ti', '##kt', '##ok', '##s', 'ra', '##dia', '##te', 'gay', 'chaotic', 'energy', 'and', 'i', 'love', 'it']\n",
            "Token IDs:  [2122, 14841, 25509, 6559, 2015, 10958, 9032, 2618, 5637, 19633, 2943, 1998, 1045, 2293, 2009]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WeNIc4auFUdF"
      },
      "source": [
        "When we actually convert all of our sentences, we'll use the `tokenize.encode` function to handle both steps, rather than calling `tokenize` and `convert_tokens_to_ids` separately. \n",
        "\n",
        "Before we can do that, though, we need to talk about some of BERT's formatting requirements."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "viKGCCh8izww"
      },
      "source": [
        "## 3.2. Required Formatting"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yDcqNlvVhL5W"
      },
      "source": [
        "The above code left out a few required formatting steps that we'll look at here.\n",
        "\n",
        "*Side Note: The input format to BERT seems \"over-specified\" to me... We are required to give it a number of pieces of information which seem redundant, or like they could easily be inferred from the data without us explicity providing it. But it is what it is, and I suspect it will make more sense once I have a deeper understanding of the BERT internals.*\n",
        "\n",
        "We are required to:\n",
        "1. Add special tokens to the start and end of each sentence.\n",
        "2. Pad & truncate all sentences to a single constant length.\n",
        "3. Explicitly differentiate real tokens from padding tokens with the \"attention mask\".\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V6mceWWOjZnw"
      },
      "source": [
        "### Special Tokens\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ykk0P9JiKtVe"
      },
      "source": [
        "\n",
        "**`[SEP]`**\n",
        "\n",
        "At the end of every sentence, we need to append the special `[SEP]` token. \n",
        "\n",
        "This token is an artifact of two-sentence tasks, where BERT is given two separate sentences and asked to determine something (e.g., can the answer to the question in sentence A be found in sentence B?). \n",
        "\n",
        "I am not certain yet why the token is still required when we have only single-sentence input, but it is!\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "86C9objaKu8f"
      },
      "source": [
        "**`[CLS]`**\n",
        "\n",
        "For classification tasks, we must prepend the special `[CLS]` token to the beginning of every sentence.\n",
        "\n",
        "This token has special significance. BERT consists of 12 Transformer layers. Each transformer takes in a list of token embeddings, and produces the same number of embeddings on the output (but with the feature values changed, of course!).\n",
        "\n",
        "![Illustration of CLS token purpose](http://www.mccormickml.com/assets/BERT/CLS_token_500x606.png)\n",
        "\n",
        "On the output of the final (12th) transformer, *only the first embedding (corresponding to the [CLS] token) is used by the classifier*.\n",
        "\n",
        ">  \"The first token of every sequence is always a special classification token (`[CLS]`). The final hidden state\n",
        "corresponding to this token is used as the aggregate sequence representation for classification\n",
        "tasks.\" (from the [BERT paper](https://arxiv.org/pdf/1810.04805.pdf))\n",
        "\n",
        "You might think to try some pooling strategy over the final embeddings, but this isn't necessary. Because BERT is trained to only use this [CLS] token for classification, we know that the model has been motivated to encode everything it needs for the classification step into that single 768-value embedding vector. It's already done the pooling for us!\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u51v0kFxeteu"
      },
      "source": [
        "### Sentence Length & Attention Mask\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qPNuwqZVK3T6"
      },
      "source": [
        "The sentences in our dataset obviously have varying lengths, so how does BERT handle this?\n",
        "\n",
        "BERT has two constraints:\n",
        "1. All sentences must be padded or truncated to a single, fixed length.\n",
        "2. The maximum sentence length is 512 tokens.\n",
        "\n",
        "Padding is done with a special `[PAD]` token, which is at index 0 in the BERT vocabulary. The below illustration demonstrates padding out to a \"MAX_LEN\" of 8 tokens.\n",
        "\n",
        "<img src=\"http://www.mccormickml.com/assets/BERT/padding_and_mask.png\" width=\"600\">\n",
        "\n",
        "The \"Attention Mask\" is simply an array of 1s and 0s indicating which tokens are padding and which aren't (seems kind of redundant, doesn't it?!). This mask tells the \"Self-Attention\" mechanism in BERT not to incorporate these PAD tokens into its interpretation of the sentence.\n",
        "\n",
        "The maximum length does impact training and evaluation speed, however. \n",
        "For example, with a Tesla K80:\n",
        "\n",
        "`MAX_LEN = 128  -->  Training epochs take ~5:28 each`\n",
        "\n",
        "`MAX_LEN = 64   -->  Training epochs take ~2:57 each`\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l6w8elb-58GJ"
      },
      "source": [
        "## 3.3. Tokenize Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U28qy4P-NwQ9"
      },
      "source": [
        "The transformers library provides a helpful `encode` function which will handle most of the parsing and data prep steps for us.\n",
        "\n",
        "Before we are ready to encode our text, though, we need to decide on a **maximum sentence length** for padding / truncating to.\n",
        "\n",
        "The below cell will perform one tokenization pass of the dataset in order to measure the maximum sentence length."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296
        },
        "id": "ktpCXDzeSHZk",
        "outputId": "a6b34d5a-c9e9-47ee-e615-4e48593e9b32"
      },
      "source": [
        "import seaborn as sns\r\n",
        "sns.countplot(x = train_df['label'])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7f930d479c18>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEGCAYAAACkQqisAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAV70lEQVR4nO3df7BfdX3n8efLINZWWUO5zcYEG+pGZ5HtBrmDbF0dlQqB2TXosC7sVqJljI6wq9PO7mJ3ZnFx2bFb0SmupRPXlGQXobToknWwNM04Ou6I5gazQEDKBWFJJpJb4orWljb2vX98P7cc401yOcn3+/Vyn4+Z73zPeX8+55zPmQt5zfnxPSdVhSRJfTxv3AOQJC1chogkqTdDRJLUmyEiSerNEJEk9XbCuAcwaqecckqtWrVq3MOQpAVl586df1ZVE4fWF12IrFq1iqmpqXEPQ5IWlCSPzVX3dJYkqTdDRJLUmyEiSerNEJEk9WaISJJ6M0QkSb0ZIpKk3gwRSVJvhogkqbdF94t1LR7/95p/MO4hPOe97D/cO+4haMw8EpEk9WaISJJ6M0QkSb0NLUSSnJrki0nuT7I7yftb/eQk25I81L6XtnqSXJ9kOsk9SV7dWdf61v+hJOs79bOS3NuWuT5JhrU/kqQfN8wjkYPAr1fV6cA5wBVJTgeuArZX1Wpge5sHuABY3T4bgBtgEDrA1cBrgLOBq2eDp/V5d2e5tUPcH0nSIYYWIlW1r6rubtPfAx4AVgDrgM2t22bgoja9DthSA3cBL0myHDgf2FZVB6rqO8A2YG1rO6mq7qqqArZ01iVJGoGRXBNJsgo4E/gasKyq9rWmbwPL2vQK4PHOYnta7Uj1PXPU59r+hiRTSaZmZmaOaV8kSc8YeogkeRFwG/CBqnqq29aOIGrYY6iqjVU1WVWTExM/9nZHSVJPQw2RJM9nECA3VdVnW/mJdiqK9r2/1fcCp3YWX9lqR6qvnKMuSRqRYd6dFeDTwANV9bFO01Zg9g6r9cDtnfpl7S6tc4DvttNedwLnJVnaLqifB9zZ2p5Kck7b1mWddUmSRmCYjz15LfAO4N4ku1rtN4CPALcmuRx4DHh7a7sDuBCYBn4AvAugqg4k+TCwo/W7pqoOtOn3ATcCLwS+0D6SpBEZWohU1VeAw/1u49w5+hdwxWHWtQnYNEd9CjjjGIYpSToG/mJdktSbISJJ6s0QkST1ZohIknozRCRJvRkikqTeDBFJUm+GiCSpN0NEktSbISJJ6s0QkST1ZohIknozRCRJvRkikqTeDBFJUm+GiCSpt2G+HndTkv1J7uvUfj/JrvZ5dPaNh0lWJfmLTtvvdpY5K8m9SaaTXN9ehUuSk5NsS/JQ+146rH2RJM1tmEciNwJru4Wq+udVtaaq1gC3AZ/tND8821ZV7+3UbwDeDaxun9l1XgVsr6rVwPY2L0kaoaGFSFV9GTgwV1s7mng7cPOR1pFkOXBSVd3VXp+7BbioNa8DNrfpzZ26JGlExnVN5HXAE1X1UKd2WpJvJPlSkte12gpgT6fPnlYDWFZV+9r0t4Flh9tYkg1JppJMzczMHKddkCSNK0Qu5UePQvYBL6uqM4FfAz6T5KT5rqwdpdQR2jdW1WRVTU5MTPQdsyTpECeMeoNJTgDeBpw1W6uqp4Gn2/TOJA8DrwD2Ais7i69sNYAnkiyvqn3ttNf+UYxfkvSMcRyJ/DLwzar629NUSSaSLGnTv8DgAvoj7XTVU0nOaddRLgNub4ttBda36fWduiRpRIZ5i+/NwFeBVybZk+Ty1nQJP35B/fXAPe2W3z8E3ltVsxfl3wf8N2AaeBj4Qqt/BHhzkocYBNNHhrUvkqS5De10VlVdepj6O+eo3cbglt+5+k8BZ8xRfxI499hGKUk6Fv5iXZLUmyEiSerNEJEk9WaISJJ6M0QkSb0ZIpKk3gwRSVJvhogkqTdDRJLUmyEiSerNEJEk9WaISJJ6M0QkSb0ZIpKk3gwRSVJvhogkqbdhvtlwU5L9Se7r1D6UZG+SXe1zYaftg0mmkzyY5PxOfW2rTSe5qlM/LcnXWv33k5w4rH2RJM1tmEciNwJr56h/vKrWtM8dAElOZ/Da3Fe1ZX4nyZL23vVPAhcApwOXtr4Av9nW9feA7wCXH7ohSdJwDS1EqurLwIGjdhxYB9xSVU9X1bcYvE/97PaZrqpHquqvgFuAdUkCvInB+9gBNgMXHdcdkCQd1TiuiVyZ5J52umtpq60AHu/02dNqh6v/LPD/qurgIfU5JdmQZCrJ1MzMzPHaD0la9EYdIjcALwfWAPuA60ax0araWFWTVTU5MTExik1K0qJwwig3VlVPzE4n+RTw+Ta7Fzi103Vlq3GY+pPAS5Kc0I5Guv0lSSMy0iORJMs7s28FZu/c2gpckuQFSU4DVgNfB3YAq9udWCcyuPi+taoK+CJwcVt+PXD7KPZBkvSMoR2JJLkZeANwSpI9wNXAG5KsAQp4FHgPQFXtTnIrcD9wELiiqn7Y1nMlcCewBNhUVbvbJv4dcEuS/wR8A/j0sPZFkjS3oYVIVV06R/mw/9BX1bXAtXPU7wDumKP+CIO7tyRJY+Iv1iVJvRkikqTeDBFJUm+GiCSpN0NEktSbISJJ6s0QkST1ZohIknozRCRJvRkikqTeDBFJUm+GiCSpN0NEktSbISJJ6s0QkST1ZohIknobWogk2ZRkf5L7OrXfSvLNJPck+VySl7T6qiR/kWRX+/xuZ5mzktybZDrJ9UnS6icn2Zbkofa9dFj7Ikma2zCPRG4E1h5S2wacUVW/CPwp8MFO28NVtaZ93tup3wC8m8F711d31nkVsL2qVgPb27wkaYSGFiJV9WXgwCG1P66qg232LmDlkdaRZDlwUlXdVVUFbAEuas3rgM1tenOnLkkakXFeE/lV4Aud+dOSfCPJl5K8rtVWAHs6ffa0GsCyqtrXpr8NLDvchpJsSDKVZGpmZuY4DV+SNJYQSfLvgYPATa20D3hZVZ0J/BrwmSQnzXd97SiljtC+saomq2pyYmLiGEYuSeo6YdQbTPJO4J8A57Z//Kmqp4Gn2/TOJA8DrwD28qOnvFa2GsATSZZX1b522mv/iHZBktSM9EgkyVrg3wJvqaofdOoTSZa06V9gcAH9kXa66qkk57S7si4Dbm+LbQXWt+n1nbokaUSGdiSS5GbgDcApSfYAVzO4G+sFwLZ2p+5d7U6s1wPXJPlr4G+A91bV7EX59zG40+uFDK6hzF5H+Qhwa5LLgceAtw9rXyRJc5tXiCTZXlXnHq3WVVWXzlH+9GH63gbcdpi2KeCMOepPAofdviRp+I4YIkl+CvhpBkcTS4G0ppN45i4pSdIidbQjkfcAHwBeCuzkmRB5CvivQxyXJGkBOGKIVNVvA7+d5F9V1SdGNCZJ0gIxr2siVfWJJL8ErOouU1VbhjQuSdICMN8L6/8deDmwC/hhK88+hkSStEjN9xbfSeD02R8HSpIE8/+x4X3A3x3mQCRJC898j0ROAe5P8nXa40kAquotQxmVJGlBmG+IfGiYg5AkLUzzvTvrS8MeiCRp4Znv3Vnf45lHrZ8IPB/486qa9+PaJUnPPfM9Ennx7HR7mu464JxhDUqStDA860fB18D/BM4fwngkSQvIfE9nva0z+zwGvxv5y6GMSJK0YMz37qx/2pk+CDzK4JSWJGkRm+81kXcNeyCSpIVnXtdEkqxM8rkk+9vntiQr57Hcptb/vk7t5CTbkjzUvpe2epJcn2Q6yT1JXt1ZZn3r/1CS9Z36WUnubctc3y76S5JGZL4X1n+PwTvNX9o+/6vVjuZGYO0htauA7VW1Gtje5gEuYPBu9dXABuAGGIQOg1frvgY4G7h6Nnhan3d3ljt0W5KkIZpviExU1e9V1cH2uRGYONpCVfVl4MAh5XXA5ja9GbioU9/S7v66C3hJkuUM7gLbVlUHquo7wDZgbWs7qaruag+G3NJZlyRpBOYbIk8m+ZUkS9rnV4Ane25zWVXta9PfBpa16RXA451+e1rtSPU9c9R/TJINSaaSTM3MzPQctiTpUPMNkV8F3s7gH/19wMXAO4914+0IYuiPl6+qjVU1WVWTExNHPYCSJM3TfEPkGmB9VU1U1c8xCJX/2HObT7RTUbTv/a2+Fzi1029lqx2pvnKOuiRpROYbIr/YrkcAUFUHgDN7bnMrMHuH1Xrg9k79snaX1jnAd9tprzuB85IsbRfUzwPubG1PJTmn3ZV1WWddkqQRmO+PDZ+XZOlskLQ7po66bJKbgTcApyTZw+Auq48Atya5HHiMwWkygDuAC4Fp4AfAu2AQWEk+DOxo/a5pIQbwPgZ3gL0Q+EL7SJJGZL4hch3w1SR/0Ob/GXDt0RaqqksP03TuHH0LuOIw69kEbJqjPgWccbRxSJKGY76/WN+SZAp4Uyu9raruH96wJEkLwXyPRGihYXBIkv7Ws34UvCRJswwRSVJvhogkqTdDRJLUmyEiSerNEJEk9WaISJJ6M0QkSb0ZIpKk3gwRSVJvhogkqTdDRJLUmyEiSerNEJEk9TbyEEnyyiS7Op+nknwgyYeS7O3UL+ws88Ek00keTHJ+p7621aaTXDXqfZGkxW7e7xM5XqrqQWANQJIlwF7gcwxeh/vxqvpot3+S04FLgFcBLwX+JMkrWvMngTcDe4AdSbb6sixJGp2Rh8ghzgUerqrHkhyuzzrglqp6GvhWkmng7NY2XVWPACS5pfU1RCRpRMZ9TeQS4ObO/JVJ7kmyKcnSVlsBPN7ps6fVDlf/MUk2JJlKMjUzM3P8Ri9Ji9zYQiTJicBbgD9opRuAlzM41bUPuO54bauqNlbVZFVNTkxMHK/VStKiN87TWRcAd1fVEwCz3wBJPgV8vs3uBU7tLLey1ThCXZI0AuM8nXUpnVNZSZZ32t4K3NemtwKXJHlBktOA1cDXgR3A6iSntaOaS1pfSdKIjOVIJMnPMLir6j2d8n9JsgYo4NHZtqraneRWBhfMDwJXVNUP23quBO4ElgCbqmr3yHZCkjSeEKmqPwd+9pDaO47Q/1rg2jnqdwB3HPcBSpLmZdx3Z0mSFjBDRJLUmyEiSerNEJEk9WaISJJ6M0QkSb0ZIpKk3gwRSVJvhogkqTdDRJLUmyEiSerNEJEk9WaISJJ6M0QkSb0ZIpKk3gwRSVJvYwuRJI8muTfJriRTrXZykm1JHmrfS1s9Sa5PMp3kniSv7qxnfev/UJL149ofSVqMxn0k8saqWlNVk23+KmB7Va0Gtrd5gAsYvFt9NbABuAEGoQNcDbwGOBu4ejZ4JEnDN+4QOdQ6YHOb3gxc1KlvqYG7gJckWQ6cD2yrqgNV9R1gG7B21IOWpMVqnCFSwB8n2ZlkQ6stq6p9bfrbwLI2vQJ4vLPsnlY7XP1HJNmQZCrJ1MzMzPHcB0la1E4Y47b/cVXtTfJzwLYk3+w2VlUlqeOxoaraCGwEmJycPC7rlCSN8Uikqva27/3A5xhc03iinaaife9v3fcCp3YWX9lqh6tLkkZgLCGS5GeSvHh2GjgPuA/YCszeYbUeuL1NbwUua3dpnQN8t532uhM4L8nSdkH9vFaTJI3AuE5nLQM+l2R2DJ+pqj9KsgO4NcnlwGPA21v/O4ALgWngB8C7AKrqQJIPAztav2uq6sDodkOSFrexhEhVPQL8wznqTwLnzlEv4IrDrGsTsOl4j1GSdHQ/abf4SpIWEENEktSbISJJ6s0QkST1ZohIknozRCRJvRkikqTeDBFJUm+GiCSpN0NEktSbISJJ6s0QkST1ZohIknozRCRJvRkikqTeDBFJUm8jD5Ekpyb5YpL7k+xO8v5W/1CSvUl2tc+FnWU+mGQ6yYNJzu/U17badJKrRr0vkrTYjePNhgeBX6+qu9t71ncm2dbaPl5VH+12TnI6cAnwKuClwJ8keUVr/iTwZmAPsCPJ1qq6fyR7IUkafYhU1T5gX5v+XpIHgBVHWGQdcEtVPQ18K8k0cHZrm26v2iXJLa2vISJJIzLWayJJVgFnAl9rpSuT3JNkU5KlrbYCeLyz2J5WO1x9ru1sSDKVZGpmZuY47oEkLW5jC5EkLwJuAz5QVU8BNwAvB9YwOFK57nhtq6o2VtVkVU1OTEwcr9VK0qI3jmsiJHk+gwC5qao+C1BVT3TaPwV8vs3uBU7tLL6y1ThCXZI0AuO4OyvAp4EHqupjnfryTre3Ave16a3AJUlekOQ0YDXwdWAHsDrJaUlOZHDxfeso9kGSNDCOI5HXAu8A7k2yq9V+A7g0yRqggEeB9wBU1e4ktzK4YH4QuKKqfgiQ5ErgTmAJsKmqdo9yRyRpsRvH3VlfATJH0x1HWOZa4No56nccabljdda/2TKsVavZ+VuXjXsIko6Bv1iXJPVmiEiSejNEJEm9GSKSpN4MEUlSb4aIJKk3Q0SS1JshIknqzRCRJPVmiEiSejNEJEm9GSKSpN4MEUlSb4aIJKk3Q0SS1JshIknqbcGHSJK1SR5MMp3kqnGPR5IWkwUdIkmWAJ8ELgBOZ/CK3dPHOypJWjwWdIgAZwPTVfVIVf0VcAuwbsxjkqRFY+TvWD/OVgCPd+b3AK85tFOSDcCGNvv9JA+OYGzjcgrwZ+MexHzlo+vHPYSfJAvqbwfA1Rn3CH6SLLy/37Pz83MVF3qIzEtVbQQ2jnsco5Bkqqomxz0OPXv+7Ra2xfr3W+ins/YCp3bmV7aaJGkEFnqI7ABWJzktyYnAJcDWMY9JkhaNBX06q6oOJrkSuBNYAmyqqt1jHta4LYrTds9R/u0WtkX590tVjXsMkqQFaqGfzpIkjZEhIknqzRB5jvDxLwtXkk1J9ie5b9xj0bOT5NQkX0xyf5LdSd4/7jGNmtdEngPa41/+FHgzgx9c7gAurar7xzowzUuS1wPfB7ZU1RnjHo/mL8lyYHlV3Z3kxcBO4KLF9P+eRyLPDT7+ZQGrqi8DB8Y9Dj17VbWvqu5u098DHmDwJI1FwxB5bpjr8S+L6j9kadySrALOBL423pGMliEiSccoyYuA24APVNVT4x7PKBkizw0+/kUakyTPZxAgN1XVZ8c9nlEzRJ4bfPyLNAZJAnwaeKCqPjbu8YyDIfIcUFUHgdnHvzwA3OrjXxaOJDcDXwVemWRPksvHPSbN22uBdwBvSrKrfS4c96BGyVt8JUm9eSQiSerNEJEk9WaISJJ6M0QkSb0ZIpKk3gwRaYiSfP8o7aue7dN7k9yY5OJjG5l0fBgikqTeDBFpBJK8KMn2JHcnuTdJ9ynLJyS5KckDSf4wyU+3Zc5K8qUkO5Pc2R47Lv1EMUSk0fhL4K1V9WrgjcB17ZEZAK8Efqeq/j7wFPC+9jymTwAXV9VZwCbg2jGMWzqiE8Y9AGmRCPCf2wuo/obBo/qXtbbHq+p/t+n/Afxr4I+AM4BtLWuWAPtGOmJpHgwRaTT+JTABnFVVf53kUeCnWtuhzx4qBqGzu6r+0eiGKD17ns6SRuPvAPtbgLwR+PlO28uSzIbFvwC+AjwITMzWkzw/yatGOmJpHgwRaTRuAiaT3AtcBnyz0/YgcEWSB4ClwA3tNccXA7+Z5P8Au4BfGvGYpaPyKb6SpN48EpEk9WaISJJ6M0QkSb0ZIpKk3gwRSVJvhogkqTdDRJLU2/8HqUZkHXLF4CYAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 698
        },
        "id": "4aJgLPH9SQLc",
        "outputId": "3db254d1-389e-4674-a4f0-2fe5a0e9f38c"
      },
      "source": [
        "train_df[train_df['label'] == 'not-English']"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1123</th>\n",
              "      <td>BLM stands for big losers matter all these pro...</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1486</th>\n",
              "      <td>CASA.   LA.  FEMMEnWesT.    ViLLAGEn2008nNyc</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2337</th>\n",
              "      <td>What seems SATANIC here is you...</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3064</th>\n",
              "      <td>thats true hate to the white community</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3196</th>\n",
              "      <td>Nycn   lovesnCARLOS    LEONn       &amp;     MADON...</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3824</th>\n",
              "      <td>@Nhi Tran aah it's never too late uh can if u ...</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4169</th>\n",
              "      <td>@Dus Bus  This is racist movement.</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5514</th>\n",
              "      <td>You can make all the covid-19 jokes you want. ...</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5867</th>\n",
              "      <td>CASA.   LA.  FEMMEnWesT.    ViLLAGEn2008nNyc</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7046</th>\n",
              "      <td>Fox News is pure Garbage!</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7318</th>\n",
              "      <td>She stans MJ ... Even Mariah Carey whose life ...</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9940</th>\n",
              "      <td>As I commented of the video of her speech</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10397</th>\n",
              "      <td>@Miguel Baham Prove it is what lazy and deeply...</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10664</th>\n",
              "      <td>Stephanie South You may live paycheck to paycheck</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10831</th>\n",
              "      <td>ESTE PSICÓPATA MASÓN LUCIFERIANO ES HOMBRE TRANS.</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10913</th>\n",
              "      <td>Hive-Mind. Emotional Contagion. Mass Hysteria....</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11235</th>\n",
              "      <td>@baily my dog hahahahaha GEYHN vakhsnzolstu ey...</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11532</th>\n",
              "      <td>Nycn   lovesnCARLOS    LEONn       &amp;     MADON...</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11545</th>\n",
              "      <td>Satanic Kabbalah witch.  Trying to bring in th...</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13991</th>\n",
              "      <td>It's controversial because you don't actually ...</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14329</th>\n",
              "      <td>ada Figueroa Too right!</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17775</th>\n",
              "      <td>Madonna is so genuine in this interview</td>\n",
              "      <td>not-English</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                    text        label\n",
              "1123   BLM stands for big losers matter all these pro...  not-English\n",
              "1486        CASA.   LA.  FEMMEnWesT.    ViLLAGEn2008nNyc  not-English\n",
              "2337                   What seems SATANIC here is you...  not-English\n",
              "3064              thats true hate to the white community  not-English\n",
              "3196   Nycn   lovesnCARLOS    LEONn       &     MADON...  not-English\n",
              "3824   @Nhi Tran aah it's never too late uh can if u ...  not-English\n",
              "4169                  @Dus Bus  This is racist movement.  not-English\n",
              "5514   You can make all the covid-19 jokes you want. ...  not-English\n",
              "5867        CASA.   LA.  FEMMEnWesT.    ViLLAGEn2008nNyc  not-English\n",
              "7046                           Fox News is pure Garbage!  not-English\n",
              "7318   She stans MJ ... Even Mariah Carey whose life ...  not-English\n",
              "9940           As I commented of the video of her speech  not-English\n",
              "10397  @Miguel Baham Prove it is what lazy and deeply...  not-English\n",
              "10664  Stephanie South You may live paycheck to paycheck  not-English\n",
              "10831  ESTE PSICÓPATA MASÓN LUCIFERIANO ES HOMBRE TRANS.  not-English\n",
              "10913  Hive-Mind. Emotional Contagion. Mass Hysteria....  not-English\n",
              "11235  @baily my dog hahahahaha GEYHN vakhsnzolstu ey...  not-English\n",
              "11532  Nycn   lovesnCARLOS    LEONn       &     MADON...  not-English\n",
              "11545  Satanic Kabbalah witch.  Trying to bring in th...  not-English\n",
              "13991  It's controversial because you don't actually ...  not-English\n",
              "14329                            ada Figueroa Too right!  not-English\n",
              "17775            Madonna is so genuine in this interview  not-English"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cKsH2sU0OCQA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "42dae365-1fb4-41cc-b9c2-7e30202e8933"
      },
      "source": [
        "max_len = 0\n",
        "\n",
        "# For every sentence...\n",
        "for sent in sentences:\n",
        "\n",
        "    # Tokenize the text and add `[CLS]` and `[SEP]` tokens.\n",
        "    input_ids = tokenizer.encode(sent, add_special_tokens=True)\n",
        "\n",
        "    # Update the maximum sentence length.\n",
        "    max_len = max(max_len, len(input_ids))\n",
        "\n",
        "print('Max sentence length: ', max_len)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Max sentence length:  249\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1M296yz577fV"
      },
      "source": [
        "Just in case there are some longer test sentences, I'll set the maximum length to 64.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tIWAoWL2RK1p"
      },
      "source": [
        "Now we're ready to perform the real tokenization.\n",
        "\n",
        "The `tokenizer.encode_plus` function combines multiple steps for us:\n",
        "\n",
        "1. Split the sentence into tokens.\n",
        "2. Add the special `[CLS]` and `[SEP]` tokens.\n",
        "3. Map the tokens to their IDs.\n",
        "4. Pad or truncate all sentences to the same length.\n",
        "5. Create the attention masks which explicitly differentiate real tokens from `[PAD]` tokens.\n",
        "\n",
        "The first four features are in `tokenizer.encode`, but I'm using `tokenizer.encode_plus` to get the fifth item (attention masks). Documentation is [here](https://huggingface.co/transformers/main_classes/tokenizer.html?highlight=encode_plus#transformers.PreTrainedTokenizer.encode_plus).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DxIccYpyU_Ra"
      },
      "source": [
        "max_input_length = 320"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2bBdb3pt8LuQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8c05316b-5315-4e66-d3af-7be18e47fa57"
      },
      "source": [
        "# Tokenize all of the sentences and map the tokens to thier word IDs.\n",
        "input_ids = []\n",
        "attention_masks = []\n",
        "\n",
        "# For every sentence...\n",
        "for sent in sentences:\n",
        "    # `encode_plus` will:\n",
        "    #   (1) Tokenize the sentence.\n",
        "    #   (2) Prepend the `[CLS]` token to the start.\n",
        "    #   (3) Append the `[SEP]` token to the end.\n",
        "    #   (4) Map tokens to their IDs.\n",
        "    #   (5) Pad or truncate the sentence to `max_length`\n",
        "    #   (6) Create attention masks for [PAD] tokens.\n",
        "    encoded_dict = tokenizer.encode_plus(\n",
        "                        sent,                      # Sentence to encode.\n",
        "                        add_special_tokens = True, # Add '[CLS]' and '[SEP]'\n",
        "                        max_length = max_input_length,           # Pad & truncate all sentences.\n",
        "                        pad_to_max_length = True,\n",
        "                        return_attention_mask = True,   # Construct attn. masks.\n",
        "                        return_tensors = 'pt',     # Return pytorch tensors.\n",
        "                   )\n",
        "    \n",
        "    # Add the encoded sentence to the list.    \n",
        "    input_ids.append(encoded_dict['input_ids'])\n",
        "    \n",
        "    # And its attention mask (simply differentiates padding from non-padding).\n",
        "    attention_masks.append(encoded_dict['attention_mask'])\n",
        "\n",
        "# Convert the lists into tensors.\n",
        "input_ids = torch.cat(input_ids, dim=0)\n",
        "attention_masks = torch.cat(attention_masks, dim=0)\n",
        "labels = torch.tensor(labels)\n",
        "\n",
        "# Print sentence 0, now as a list of IDs.\n",
        "print('Original: ', sentences[0])\n",
        "print('Token IDs:', input_ids[0])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.\n",
            "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py:2179: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).\n",
            "  FutureWarning,\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Original:  these tiktoks radiate gay chaotic energy and i love it\n",
            "Token IDs: tensor([  101,  2122, 14841, 25509,  6559,  2015, 10958,  9032,  2618,  5637,\n",
            "        19633,  2943,  1998,  1045,  2293,  2009,   102,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0,\n",
            "            0,     0,     0,     0,     0,     0,     0,     0,     0,     0])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aRp4O7D295d_"
      },
      "source": [
        "## 3.4. Training & Validation Split\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qu0ao7p8rb06"
      },
      "source": [
        "Divide up our training set to use 90% for training and 10% for validation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GEgLpFVlo1Z-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cdb63a51-c562-441c-f5e9-5fcd3d273f2b"
      },
      "source": [
        "from torch.utils.data import TensorDataset, random_split\n",
        "\n",
        "# Combine the training inputs into a TensorDataset.\n",
        "dataset = TensorDataset(input_ids, attention_masks, labels)\n",
        "\n",
        "# Create a 90-10 train-validation split.\n",
        "\n",
        "# Calculate the number of samples to include in each set.\n",
        "train_size = int(0.9 * len(dataset))\n",
        "val_size = len(dataset) - train_size\n",
        "\n",
        "# Divide the dataset by randomly selecting samples.\n",
        "train_dataset, val_dataset = random_split(dataset, [train_size, val_size])\n",
        "\n",
        "print('{:>5,} training samples'.format(train_size))\n",
        "print('{:>5,} validation samples'.format(val_size))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "20,485 training samples\n",
            "2,277 validation samples\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dD9i6Z2pG-sN"
      },
      "source": [
        "We'll also create an iterator for our dataset using the torch DataLoader class. This helps save on memory during training because, unlike a for loop, with an iterator the entire dataset does not need to be loaded into memory."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XGUqOCtgqGhP"
      },
      "source": [
        "from torch.utils.data import DataLoader, RandomSampler, SequentialSampler\n",
        "\n",
        "# The DataLoader needs to know our batch size for training, so we specify it \n",
        "# here. For fine-tuning BERT on a specific task, the authors recommend a batch \n",
        "# size of 16 or 32.\n",
        "batch_size = 32\n",
        "\n",
        "# Create the DataLoaders for our training and validation sets.\n",
        "# We'll take training samples in random order. \n",
        "train_dataloader = DataLoader(\n",
        "            train_dataset,  # The training samples.\n",
        "            sampler = RandomSampler(train_dataset), # Select batches randomly\n",
        "            batch_size = batch_size # Trains with this batch size.\n",
        "        )\n",
        "\n",
        "# For validation the order doesn't matter, so we'll just read them sequentially.\n",
        "validation_dataloader = DataLoader(\n",
        "            val_dataset, # The validation samples.\n",
        "            sampler = SequentialSampler(val_dataset), # Pull out batches sequentially.\n",
        "            batch_size = batch_size # Evaluate with this batch size.\n",
        "        )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8bwa6Rts-02-"
      },
      "source": [
        "# 4. Train Our Classification Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3xYQ3iLO08SX"
      },
      "source": [
        "Now that our input data is properly formatted, it's time to fine tune the BERT model. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D6TKgyUzPIQc"
      },
      "source": [
        "## 4.1. BertForSequenceClassification"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1sjzRT1V0zwm"
      },
      "source": [
        "For this task, we first want to modify the pre-trained BERT model to give outputs for classification, and then we want to continue training the model on our dataset until that the entire model, end-to-end, is well-suited for our task. \n",
        "\n",
        "Thankfully, the huggingface pytorch implementation includes a set of interfaces designed for a variety of NLP tasks. Though these interfaces are all built on top of a trained BERT model, each has different top layers and output types designed to accomodate their specific NLP task.  \n",
        "\n",
        "Here is the current list of classes provided for fine-tuning:\n",
        "* BertModel\n",
        "* BertForPreTraining\n",
        "* BertForMaskedLM\n",
        "* BertForNextSentencePrediction\n",
        "* **BertForSequenceClassification** - The one we'll use.\n",
        "* BertForTokenClassification\n",
        "* BertForQuestionAnswering\n",
        "\n",
        "The documentation for these can be found under [here](https://huggingface.co/transformers/v2.2.0/model_doc/bert.html)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BXYitPoE-cjH"
      },
      "source": [
        "\n",
        "\n",
        "We'll be using [BertForSequenceClassification](https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#bertforsequenceclassification). This is the normal BERT model with an added single linear layer on top for classification that we will use as a sentence classifier. As we feed input data, the entire pre-trained BERT model and the additional untrained classification layer is trained on our specific task. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WnQW9E-bBCRt"
      },
      "source": [
        "OK, let's load BERT! There are a few different pre-trained BERT models available. \"bert-base-uncased\" means the version that has only lowercase letters (\"uncased\") and is the smaller version of the two (\"base\" vs \"large\").\n",
        "\n",
        "The documentation for `from_pretrained` can be found [here](https://huggingface.co/transformers/v2.2.0/main_classes/model.html#transformers.PreTrainedModel.from_pretrained), with the additional parameters defined [here](https://huggingface.co/transformers/v2.2.0/main_classes/configuration.html#transformers.PretrainedConfig)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gFsCTp_mporB",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000,
          "referenced_widgets": [
            "838d72a9cc3b453392405ce72048f461",
            "23f0f52c8dcc4984882261d9ee749fa3",
            "9ef4cf142e3b4f4a95f182a3bccf7123",
            "3c53de0bfddb4891bc8446adf7608731",
            "7b8901afe9e2498b9467c87f08cf81eb",
            "3ca33d0a7b9e475e880112897ae2edca",
            "04b8464f597d43e88866adc81c1249bd",
            "69da6f2fbdd24af099379cc7534f945d",
            "18060a5a10bc4ee5bb23fdb0297a7b2a",
            "42c93e704e9f4b4ca7931ab8cbe12e15",
            "2d408db371064eefbdc50755ab709a85",
            "23da42ad57ac4982a378acfa93e7e926",
            "06411a62e17b4cbba04dac587dd12f19",
            "4602db38e1ff4cdf86ca5dffdde78f45",
            "edeb18092ab840c2bf1bfef66d78cd46",
            "63980dd638314cf283f0bae6e9bad739"
          ]
        },
        "outputId": "f953fc26-3878-439f-b564-f9f61d276fef"
      },
      "source": [
        "from transformers import BertForSequenceClassification, AdamW, BertConfig\n",
        "\n",
        "# Load BertForSequenceClassification, the pretrained BERT model with a single \n",
        "# linear classification layer on top. \n",
        "model = BertForSequenceClassification.from_pretrained(\n",
        "    \"bert-base-uncased\", # Use the 12-layer BERT model, with an uncased vocab.\n",
        "    num_labels = 3, # The number of output labels--2 for binary classification.\n",
        "                    # You can increase this for multi-class tasks.   \n",
        "    output_attentions = False, # Whether the model returns attentions weights.\n",
        "    output_hidden_states = False, # Whether the model returns all hidden-states.\n",
        ")\n",
        "\n",
        "# Tell pytorch to run this model on the GPU.\n",
        "model.cuda()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "838d72a9cc3b453392405ce72048f461",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=433.0, style=ProgressStyle(description_…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "18060a5a10bc4ee5bb23fdb0297a7b2a",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=440473133.0, style=ProgressStyle(descri…"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
            "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BertForSequenceClassification(\n",
              "  (bert): BertModel(\n",
              "    (embeddings): BertEmbeddings(\n",
              "      (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
              "      (position_embeddings): Embedding(512, 768)\n",
              "      (token_type_embeddings): Embedding(2, 768)\n",
              "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "      (dropout): Dropout(p=0.1, inplace=False)\n",
              "    )\n",
              "    (encoder): BertEncoder(\n",
              "      (layer): ModuleList(\n",
              "        (0): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (1): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (2): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (3): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (4): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (5): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (6): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (7): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (8): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (9): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (10): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "        (11): BertLayer(\n",
              "          (attention): BertAttention(\n",
              "            (self): BertSelfAttention(\n",
              "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "            (output): BertSelfOutput(\n",
              "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "              (dropout): Dropout(p=0.1, inplace=False)\n",
              "            )\n",
              "          )\n",
              "          (intermediate): BertIntermediate(\n",
              "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
              "          )\n",
              "          (output): BertOutput(\n",
              "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
              "            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            (dropout): Dropout(p=0.1, inplace=False)\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "    )\n",
              "    (pooler): BertPooler(\n",
              "      (dense): Linear(in_features=768, out_features=768, bias=True)\n",
              "      (activation): Tanh()\n",
              "    )\n",
              "  )\n",
              "  (dropout): Dropout(p=0.1, inplace=False)\n",
              "  (classifier): Linear(in_features=768, out_features=3, bias=True)\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e0Jv6c7-HHDW"
      },
      "source": [
        "Just for curiosity's sake, we can browse all of the model's parameters by name here.\n",
        "\n",
        "In the below cell, I've printed out the names and dimensions of the weights for:\n",
        "\n",
        "1. The embedding layer.\n",
        "2. The first of the twelve transformers.\n",
        "3. The output layer.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8PIiVlDYCtSq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c9034555-023a-46e8-c1a2-fb1755d4d4a7"
      },
      "source": [
        "# Get all of the model's parameters as a list of tuples.\n",
        "params = list(model.named_parameters())\n",
        "\n",
        "print('The BERT model has {:} different named parameters.\\n'.format(len(params)))\n",
        "\n",
        "print('==== Embedding Layer ====\\n')\n",
        "\n",
        "for p in params[0:5]:\n",
        "    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))\n",
        "\n",
        "print('\\n==== First Transformer ====\\n')\n",
        "\n",
        "for p in params[5:21]:\n",
        "    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))\n",
        "\n",
        "print('\\n==== Output Layer ====\\n')\n",
        "\n",
        "for p in params[-4:]:\n",
        "    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The BERT model has 201 different named parameters.\n",
            "\n",
            "==== Embedding Layer ====\n",
            "\n",
            "bert.embeddings.word_embeddings.weight                  (30522, 768)\n",
            "bert.embeddings.position_embeddings.weight                (512, 768)\n",
            "bert.embeddings.token_type_embeddings.weight                (2, 768)\n",
            "bert.embeddings.LayerNorm.weight                              (768,)\n",
            "bert.embeddings.LayerNorm.bias                                (768,)\n",
            "\n",
            "==== First Transformer ====\n",
            "\n",
            "bert.encoder.layer.0.attention.self.query.weight          (768, 768)\n",
            "bert.encoder.layer.0.attention.self.query.bias                (768,)\n",
            "bert.encoder.layer.0.attention.self.key.weight            (768, 768)\n",
            "bert.encoder.layer.0.attention.self.key.bias                  (768,)\n",
            "bert.encoder.layer.0.attention.self.value.weight          (768, 768)\n",
            "bert.encoder.layer.0.attention.self.value.bias                (768,)\n",
            "bert.encoder.layer.0.attention.output.dense.weight        (768, 768)\n",
            "bert.encoder.layer.0.attention.output.dense.bias              (768,)\n",
            "bert.encoder.layer.0.attention.output.LayerNorm.weight        (768,)\n",
            "bert.encoder.layer.0.attention.output.LayerNorm.bias          (768,)\n",
            "bert.encoder.layer.0.intermediate.dense.weight           (3072, 768)\n",
            "bert.encoder.layer.0.intermediate.dense.bias                 (3072,)\n",
            "bert.encoder.layer.0.output.dense.weight                 (768, 3072)\n",
            "bert.encoder.layer.0.output.dense.bias                        (768,)\n",
            "bert.encoder.layer.0.output.LayerNorm.weight                  (768,)\n",
            "bert.encoder.layer.0.output.LayerNorm.bias                    (768,)\n",
            "\n",
            "==== Output Layer ====\n",
            "\n",
            "bert.pooler.dense.weight                                  (768, 768)\n",
            "bert.pooler.dense.bias                                        (768,)\n",
            "classifier.weight                                           (3, 768)\n",
            "classifier.bias                                                 (3,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qRWT-D4U_Pvx"
      },
      "source": [
        "## 4.2. Optimizer & Learning Rate Scheduler"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8o-VEBobKwHk"
      },
      "source": [
        "Now that we have our model loaded we need to grab the training hyperparameters from within the stored model.\n",
        "\n",
        "For the purposes of fine-tuning, the authors recommend choosing from the following values (from Appendix A.3 of the [BERT paper](https://arxiv.org/pdf/1810.04805.pdf)):\n",
        "\n",
        ">- **Batch size:** 16, 32  \n",
        "- **Learning rate (Adam):** 5e-5, 3e-5, 2e-5  \n",
        "- **Number of epochs:** 2, 3, 4 \n",
        "\n",
        "We chose:\n",
        "* Batch size: 32 (set when creating our DataLoaders)\n",
        "* Learning rate: 2e-5\n",
        "* Epochs: 4 (we'll see that this is probably too many...)\n",
        "\n",
        "The epsilon parameter `eps = 1e-8` is \"a very small number to prevent any division by zero in the implementation\" (from [here](https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/)).\n",
        "\n",
        "You can find the creation of the AdamW optimizer in `run_glue.py` [here](https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/examples/run_glue.py#L109)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GLs72DuMODJO"
      },
      "source": [
        "# Note: AdamW is a class from the huggingface library (as opposed to pytorch) \n",
        "# I believe the 'W' stands for 'Weight Decay fix\"\n",
        "optimizer = AdamW(model.parameters(),\n",
        "                  lr = 2e-5, # args.learning_rate - default is 5e-5, our notebook had 2e-5\n",
        "                  eps = 1e-8 # args.adam_epsilon  - default is 1e-8.\n",
        "                )\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-p0upAhhRiIx"
      },
      "source": [
        "from transformers import get_linear_schedule_with_warmup\n",
        "\n",
        "# Number of training epochs. The BERT authors recommend between 2 and 4. \n",
        "# We chose to run for 4, but we'll see later that this may be over-fitting the\n",
        "# training data.\n",
        "epochs = 4\n",
        "\n",
        "# Total number of training steps is [number of batches] x [number of epochs]. \n",
        "# (Note that this is not the same as the number of training samples).\n",
        "total_steps = len(train_dataloader) * epochs\n",
        "\n",
        "# Create the learning rate scheduler.\n",
        "scheduler = get_linear_schedule_with_warmup(optimizer, \n",
        "                                            num_warmup_steps = 0, # Default value in run_glue.py\n",
        "                                            num_training_steps = total_steps)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RqfmWwUR_Sox"
      },
      "source": [
        "## 4.3. Training Loop"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_QXZhFb4LnV5"
      },
      "source": [
        "Below is our training loop. There's a lot going on, but fundamentally for each pass in our loop we have a trianing phase and a validation phase. \n",
        "\n",
        "> *Thank you to [Stas Bekman](https://ca.linkedin.com/in/stasbekman) for contributing the insights and code for using validation loss to detect over-fitting!*\n",
        "\n",
        "**Training:**\n",
        "- Unpack our data inputs and labels\n",
        "- Load data onto the GPU for acceleration\n",
        "- Clear out the gradients calculated in the previous pass. \n",
        "    - In pytorch the gradients accumulate by default (useful for things like RNNs) unless you explicitly clear them out.\n",
        "- Forward pass (feed input data through the network)\n",
        "- Backward pass (backpropagation)\n",
        "- Tell the network to update parameters with optimizer.step()\n",
        "- Track variables for monitoring progress\n",
        "\n",
        "**Evalution:**\n",
        "- Unpack our data inputs and labels\n",
        "- Load data onto the GPU for acceleration\n",
        "- Forward pass (feed input data through the network)\n",
        "- Compute loss on our validation data and track variables for monitoring progress\n",
        "\n",
        "Pytorch hides all of the detailed calculations from us, but we've commented the code to point out which of the above steps are happening on each line. \n",
        "\n",
        "> *PyTorch also has some [beginner tutorials](https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py) which you may also find helpful.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pE5B99H5H2-W"
      },
      "source": [
        "Define a helper function for calculating accuracy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9cQNvaZ9bnyy"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "# Function to calculate the accuracy of our predictions vs labels\n",
        "def flat_accuracy(preds, labels):\n",
        "    pred_flat = np.argmax(preds, axis=1).flatten()\n",
        "    labels_flat = labels.flatten()\n",
        "    return np.sum(pred_flat == labels_flat) / len(labels_flat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KNhRtWPXH9C3"
      },
      "source": [
        "Helper function for formatting elapsed times as `hh:mm:ss`\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gpt6tR83keZD"
      },
      "source": [
        "import time\n",
        "import datetime\n",
        "\n",
        "def format_time(elapsed):\n",
        "    '''\n",
        "    Takes a time in seconds and returns a string hh:mm:ss\n",
        "    '''\n",
        "    # Round to the nearest second.\n",
        "    elapsed_rounded = int(round((elapsed)))\n",
        "    \n",
        "    # Format as hh:mm:ss\n",
        "    return str(datetime.timedelta(seconds=elapsed_rounded))\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cfNIhN19te3N"
      },
      "source": [
        "We're ready to kick off the training!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6J-FYdx6nFE_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "30d30c79-bfeb-4644-9295-94d0fb802716"
      },
      "source": [
        "import random\n",
        "import numpy as np\n",
        "\n",
        "# This training code is based on the `run_glue.py` script here:\n",
        "# https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/examples/run_glue.py#L128\n",
        "\n",
        "# Set the seed value all over the place to make this reproducible.\n",
        "seed_val = 42\n",
        "\n",
        "random.seed(seed_val)\n",
        "np.random.seed(seed_val)\n",
        "torch.manual_seed(seed_val)\n",
        "torch.cuda.manual_seed_all(seed_val)\n",
        "\n",
        "# We'll store a number of quantities such as training and validation loss, \n",
        "# validation accuracy, and timings.\n",
        "training_stats = []\n",
        "\n",
        "# Measure the total training time for the whole run.\n",
        "total_t0 = time.time()\n",
        "\n",
        "# For each epoch...\n",
        "for epoch_i in range(0, epochs):\n",
        "    \n",
        "    # ========================================\n",
        "    #               Training\n",
        "    # ========================================\n",
        "    \n",
        "    # Perform one full pass over the training set.\n",
        "\n",
        "    print(\"\")\n",
        "    print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, epochs))\n",
        "    print('Training...')\n",
        "\n",
        "    # Measure how long the training epoch takes.\n",
        "    t0 = time.time()\n",
        "\n",
        "    # Reset the total loss for this epoch.\n",
        "    total_train_loss = 0\n",
        "\n",
        "    # Put the model into training mode. Don't be mislead--the call to \n",
        "    # `train` just changes the *mode*, it doesn't *perform* the training.\n",
        "    # `dropout` and `batchnorm` layers behave differently during training\n",
        "    # vs. test (source: https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch)\n",
        "    model.train()\n",
        "\n",
        "    # For each batch of training data...\n",
        "    for step, batch in enumerate(train_dataloader):\n",
        "\n",
        "        # Progress update every 40 batches.\n",
        "        if step % 40 == 0 and not step == 0:\n",
        "            # Calculate elapsed time in minutes.\n",
        "            elapsed = format_time(time.time() - t0)\n",
        "            \n",
        "            # Report progress.\n",
        "            print('  Batch {:>5,}  of  {:>5,}.    Elapsed: {:}.'.format(step, len(train_dataloader), elapsed))\n",
        "\n",
        "        # Unpack this training batch from our dataloader. \n",
        "        #\n",
        "        # As we unpack the batch, we'll also copy each tensor to the GPU using the \n",
        "        # `to` method.\n",
        "        #\n",
        "        # `batch` contains three pytorch tensors:\n",
        "        #   [0]: input ids \n",
        "        #   [1]: attention masks\n",
        "        #   [2]: labels \n",
        "        b_input_ids = batch[0].to(device)\n",
        "        b_input_mask = batch[1].to(device)\n",
        "        b_labels = batch[2].to(device)\n",
        "\n",
        "        # Always clear any previously calculated gradients before performing a\n",
        "        # backward pass. PyTorch doesn't do this automatically because \n",
        "        # accumulating the gradients is \"convenient while training RNNs\". \n",
        "        # (source: https://stackoverflow.com/questions/48001598/why-do-we-need-to-call-zero-grad-in-pytorch)\n",
        "        model.zero_grad()        \n",
        "\n",
        "        # Perform a forward pass (evaluate the model on this training batch).\n",
        "        # The documentation for this `model` function is here: \n",
        "        # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification\n",
        "        # It returns different numbers of parameters depending on what arguments\n",
        "        # arge given and what flags are set. For our useage here, it returns\n",
        "        # the loss (because we provided labels) and the \"logits\"--the model\n",
        "        # outputs prior to activation.\n",
        "        output = model(b_input_ids, \n",
        "                             token_type_ids=None, \n",
        "                             attention_mask=b_input_mask, \n",
        "                             labels=b_labels)\n",
        "\n",
        "        # Accumulate the training loss over all of the batches so that we can\n",
        "        # calculate the average loss at the end. `loss` is a Tensor containing a\n",
        "        # single value; the `.item()` function just returns the Python value \n",
        "        # from the tensor.\n",
        "        loss = output.loss \n",
        "        logits = output.logits\n",
        "        total_train_loss += loss.item()\n",
        "\n",
        "        # Perform a backward pass to calculate the gradients.\n",
        "        loss.backward()\n",
        "\n",
        "        # Clip the norm of the gradients to 1.0.\n",
        "        # This is to help prevent the \"exploding gradients\" problem.\n",
        "        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n",
        "\n",
        "        # Update parameters and take a step using the computed gradient.\n",
        "        # The optimizer dictates the \"update rule\"--how the parameters are\n",
        "        # modified based on their gradients, the learning rate, etc.\n",
        "        optimizer.step()\n",
        "\n",
        "        # Update the learning rate.\n",
        "        scheduler.step()\n",
        "\n",
        "    # Calculate the average loss over all of the batches.\n",
        "    avg_train_loss = total_train_loss / len(train_dataloader)            \n",
        "    \n",
        "    # Measure how long this epoch took.\n",
        "    training_time = format_time(time.time() - t0)\n",
        "\n",
        "    print(\"\")\n",
        "    print(\"  Average training loss: {0:.2f}\".format(avg_train_loss))\n",
        "    print(\"  Training epcoh took: {:}\".format(training_time))\n",
        "        \n",
        "    # ========================================\n",
        "    #               Validation\n",
        "    # ========================================\n",
        "    # After the completion of each training epoch, measure our performance on\n",
        "    # our validation set.\n",
        "\n",
        "    print(\"\")\n",
        "    print(\"Running Validation...\")\n",
        "\n",
        "    t0 = time.time()\n",
        "\n",
        "    # Put the model in evaluation mode--the dropout layers behave differently\n",
        "    # during evaluation.\n",
        "    model.eval()\n",
        "\n",
        "    # Tracking variables \n",
        "    total_eval_accuracy = 0\n",
        "    total_eval_loss = 0\n",
        "    nb_eval_steps = 0\n",
        "\n",
        "    # Evaluate data for one epoch\n",
        "    for batch in validation_dataloader:\n",
        "        \n",
        "        # Unpack this training batch from our dataloader. \n",
        "        #\n",
        "        # As we unpack the batch, we'll also copy each tensor to the GPU using \n",
        "        # the `to` method.\n",
        "        #\n",
        "        # `batch` contains three pytorch tensors:\n",
        "        #   [0]: input ids \n",
        "        #   [1]: attention masks\n",
        "        #   [2]: labels \n",
        "        b_input_ids = batch[0].to(device)\n",
        "        b_input_mask = batch[1].to(device)\n",
        "        b_labels = batch[2].to(device)\n",
        "        \n",
        "        # Tell pytorch not to bother with constructing the compute graph during\n",
        "        # the forward pass, since this is only needed for backprop (training).\n",
        "        with torch.no_grad():        \n",
        "\n",
        "            # Forward pass, calculate logit predictions.\n",
        "            # token_type_ids is the same as the \"segment ids\", which \n",
        "            # differentiates sentence 1 and 2 in 2-sentence tasks.\n",
        "            # The documentation for this `model` function is here: \n",
        "            # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification\n",
        "            # Get the \"logits\" output by the model. The \"logits\" are the output\n",
        "            # values prior to applying an activation function like the softmax.\n",
        "            output = model(b_input_ids, \n",
        "                                   token_type_ids=None, \n",
        "                                   attention_mask=b_input_mask,\n",
        "                                   labels=b_labels)\n",
        "        loss = output.loss\n",
        "        logits = output.logits\n",
        "        # Accumulate the validation loss.\n",
        "        total_eval_loss += loss.item()\n",
        "\n",
        "        # Move logits and labels to CPU\n",
        "        logits = logits.detach().cpu().numpy()\n",
        "        label_ids = b_labels.to('cpu').numpy()\n",
        "\n",
        "        # Calculate the accuracy for this batch of test sentences, and\n",
        "        # accumulate it over all batches.\n",
        "        total_eval_accuracy += flat_accuracy(logits, label_ids)\n",
        "        \n",
        "\n",
        "    # Report the final accuracy for this validation run.\n",
        "    avg_val_accuracy = total_eval_accuracy / len(validation_dataloader)\n",
        "    print(\"  Accuracy: {0:.2f}\".format(avg_val_accuracy))\n",
        "\n",
        "    # Calculate the average loss over all of the batches.\n",
        "    avg_val_loss = total_eval_loss / len(validation_dataloader)\n",
        "    \n",
        "    # Measure how long the validation run took.\n",
        "    validation_time = format_time(time.time() - t0)\n",
        "    \n",
        "    print(\"  Validation Loss: {0:.2f}\".format(avg_val_loss))\n",
        "    print(\"  Validation took: {:}\".format(validation_time))\n",
        "\n",
        "    # Record all statistics from this epoch.\n",
        "    training_stats.append(\n",
        "        {\n",
        "            'epoch': epoch_i + 1,\n",
        "            'Training Loss': avg_train_loss,\n",
        "            'Valid. Loss': avg_val_loss,\n",
        "            'Valid. Accur.': avg_val_accuracy,\n",
        "            'Training Time': training_time,\n",
        "            'Validation Time': validation_time\n",
        "        }\n",
        "    )\n",
        "\n",
        "print(\"\")\n",
        "print(\"Training complete!\")\n",
        "\n",
        "print(\"Total training took {:} (h:mm:ss)\".format(format_time(time.time()-total_t0)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "======== Epoch 1 / 4 ========\n",
            "Training...\n",
            "  Batch    40  of    641.    Elapsed: 0:01:09.\n",
            "  Batch    80  of    641.    Elapsed: 0:02:19.\n",
            "  Batch   120  of    641.    Elapsed: 0:03:30.\n",
            "  Batch   160  of    641.    Elapsed: 0:04:40.\n",
            "  Batch   200  of    641.    Elapsed: 0:05:51.\n",
            "  Batch   240  of    641.    Elapsed: 0:07:01.\n",
            "  Batch   280  of    641.    Elapsed: 0:08:12.\n",
            "  Batch   320  of    641.    Elapsed: 0:09:23.\n",
            "  Batch   360  of    641.    Elapsed: 0:10:33.\n",
            "  Batch   400  of    641.    Elapsed: 0:11:44.\n",
            "  Batch   440  of    641.    Elapsed: 0:12:54.\n",
            "  Batch   480  of    641.    Elapsed: 0:14:05.\n",
            "  Batch   520  of    641.    Elapsed: 0:15:15.\n",
            "  Batch   560  of    641.    Elapsed: 0:16:26.\n",
            "  Batch   600  of    641.    Elapsed: 0:17:36.\n",
            "  Batch   640  of    641.    Elapsed: 0:18:47.\n",
            "\n",
            "  Average training loss: 0.22\n",
            "  Training epcoh took: 0:18:47\n",
            "\n",
            "Running Validation...\n",
            "  Accuracy: 0.93\n",
            "  Validation Loss: 0.19\n",
            "  Validation took: 0:00:47\n",
            "\n",
            "======== Epoch 2 / 4 ========\n",
            "Training...\n",
            "  Batch    40  of    641.    Elapsed: 0:01:11.\n",
            "  Batch    80  of    641.    Elapsed: 0:02:21.\n",
            "  Batch   120  of    641.    Elapsed: 0:03:32.\n",
            "  Batch   160  of    641.    Elapsed: 0:04:42.\n",
            "  Batch   200  of    641.    Elapsed: 0:05:53.\n",
            "  Batch   240  of    641.    Elapsed: 0:07:03.\n",
            "  Batch   280  of    641.    Elapsed: 0:08:14.\n",
            "  Batch   320  of    641.    Elapsed: 0:09:24.\n",
            "  Batch   360  of    641.    Elapsed: 0:10:35.\n",
            "  Batch   400  of    641.    Elapsed: 0:11:45.\n",
            "  Batch   440  of    641.    Elapsed: 0:12:56.\n",
            "  Batch   480  of    641.    Elapsed: 0:14:07.\n",
            "  Batch   520  of    641.    Elapsed: 0:15:17.\n",
            "  Batch   560  of    641.    Elapsed: 0:16:28.\n",
            "  Batch   600  of    641.    Elapsed: 0:17:38.\n",
            "  Batch   640  of    641.    Elapsed: 0:18:49.\n",
            "\n",
            "  Average training loss: 0.14\n",
            "  Training epcoh took: 0:18:49\n",
            "\n",
            "Running Validation...\n",
            "  Accuracy: 0.92\n",
            "  Validation Loss: 0.21\n",
            "  Validation took: 0:00:47\n",
            "\n",
            "======== Epoch 3 / 4 ========\n",
            "Training...\n",
            "  Batch    40  of    641.    Elapsed: 0:01:10.\n",
            "  Batch    80  of    641.    Elapsed: 0:02:21.\n",
            "  Batch   120  of    641.    Elapsed: 0:03:31.\n",
            "  Batch   160  of    641.    Elapsed: 0:04:42.\n",
            "  Batch   200  of    641.    Elapsed: 0:05:53.\n",
            "  Batch   240  of    641.    Elapsed: 0:07:03.\n",
            "  Batch   280  of    641.    Elapsed: 0:08:13.\n",
            "  Batch   320  of    641.    Elapsed: 0:09:24.\n",
            "  Batch   360  of    641.    Elapsed: 0:10:35.\n",
            "  Batch   400  of    641.    Elapsed: 0:11:45.\n",
            "  Batch   440  of    641.    Elapsed: 0:12:56.\n",
            "  Batch   480  of    641.    Elapsed: 0:14:06.\n",
            "  Batch   520  of    641.    Elapsed: 0:15:17.\n",
            "  Batch   560  of    641.    Elapsed: 0:16:27.\n",
            "  Batch   600  of    641.    Elapsed: 0:17:38.\n",
            "  Batch   640  of    641.    Elapsed: 0:18:49.\n",
            "\n",
            "  Average training loss: 0.09\n",
            "  Training epcoh took: 0:18:49\n",
            "\n",
            "Running Validation...\n",
            "  Accuracy: 0.93\n",
            "  Validation Loss: 0.27\n",
            "  Validation took: 0:00:47\n",
            "\n",
            "======== Epoch 4 / 4 ========\n",
            "Training...\n",
            "  Batch    40  of    641.    Elapsed: 0:01:11.\n",
            "  Batch    80  of    641.    Elapsed: 0:02:21.\n",
            "  Batch   120  of    641.    Elapsed: 0:03:32.\n",
            "  Batch   160  of    641.    Elapsed: 0:04:42.\n",
            "  Batch   200  of    641.    Elapsed: 0:05:53.\n",
            "  Batch   240  of    641.    Elapsed: 0:07:03.\n",
            "  Batch   280  of    641.    Elapsed: 0:08:14.\n",
            "  Batch   320  of    641.    Elapsed: 0:09:24.\n",
            "  Batch   360  of    641.    Elapsed: 0:10:35.\n",
            "  Batch   400  of    641.    Elapsed: 0:11:45.\n",
            "  Batch   440  of    641.    Elapsed: 0:12:56.\n",
            "  Batch   480  of    641.    Elapsed: 0:14:06.\n",
            "  Batch   520  of    641.    Elapsed: 0:15:17.\n",
            "  Batch   560  of    641.    Elapsed: 0:16:28.\n",
            "  Batch   600  of    641.    Elapsed: 0:17:38.\n",
            "  Batch   640  of    641.    Elapsed: 0:18:48.\n",
            "\n",
            "  Average training loss: 0.06\n",
            "  Training epcoh took: 0:18:49\n",
            "\n",
            "Running Validation...\n",
            "  Accuracy: 0.92\n",
            "  Validation Loss: 0.31\n",
            "  Validation took: 0:00:47\n",
            "\n",
            "Training complete!\n",
            "Total training took 1:18:21 (h:mm:ss)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VQTvJ1vRP7u4"
      },
      "source": [
        "Let's view the summary of the training process."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6O_NbXFGMukX",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 195
        },
        "outputId": "7b8b5552-7af5-4ed6-9d03-efd111f57d28"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Display floats with two decimal places.\n",
        "pd.set_option('precision', 2)\n",
        "\n",
        "# Create a DataFrame from our training statistics.\n",
        "df_stats = pd.DataFrame(data=training_stats)\n",
        "\n",
        "# Use the 'epoch' as the row index.\n",
        "df_stats = df_stats.set_index('epoch')\n",
        "\n",
        "# A hack to force the column headers to wrap.\n",
        "#df = df.style.set_table_styles([dict(selector=\"th\",props=[('max-width', '70px')])])\n",
        "\n",
        "# Display the table.\n",
        "df_stats"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Valid. Loss</th>\n",
              "      <th>Valid. Accur.</th>\n",
              "      <th>Training Time</th>\n",
              "      <th>Validation Time</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>epoch</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.22</td>\n",
              "      <td>0.19</td>\n",
              "      <td>0.93</td>\n",
              "      <td>0:18:47</td>\n",
              "      <td>0:00:47</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.14</td>\n",
              "      <td>0.21</td>\n",
              "      <td>0.92</td>\n",
              "      <td>0:18:49</td>\n",
              "      <td>0:00:47</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.09</td>\n",
              "      <td>0.27</td>\n",
              "      <td>0.93</td>\n",
              "      <td>0:18:49</td>\n",
              "      <td>0:00:47</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.06</td>\n",
              "      <td>0.31</td>\n",
              "      <td>0.92</td>\n",
              "      <td>0:18:49</td>\n",
              "      <td>0:00:47</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       Training Loss  Valid. Loss  Valid. Accur. Training Time Validation Time\n",
              "epoch                                                                         \n",
              "1               0.22         0.19           0.93       0:18:47         0:00:47\n",
              "2               0.14         0.21           0.92       0:18:49         0:00:47\n",
              "3               0.09         0.27           0.93       0:18:49         0:00:47\n",
              "4               0.06         0.31           0.92       0:18:49         0:00:47"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1-G03mmwH3aI"
      },
      "source": [
        "Notice that, while the the training loss is going down with each epoch, the validation loss is increasing! This suggests that we are training our model too long, and it's over-fitting on the training data. \n",
        "\n",
        "(For reference, we are using 7,695 training samples and 856 validation samples).\n",
        "\n",
        "Validation Loss is a more precise measure than accuracy, because with accuracy we don't care about the exact output value, but just which side of a threshold it falls on. \n",
        "\n",
        "If we are predicting the correct answer, but with less confidence, then validation loss will catch this, while accuracy will not."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "68xreA9JAmG5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 427
        },
        "outputId": "fb2bd8bb-fbc6-43d6-979d-cc118316948c"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "% matplotlib inline\n",
        "\n",
        "import seaborn as sns\n",
        "\n",
        "# Use plot styling from seaborn.\n",
        "sns.set(style='darkgrid')\n",
        "\n",
        "# Increase the plot size and font size.\n",
        "sns.set(font_scale=1.5)\n",
        "plt.rcParams[\"figure.figsize\"] = (12,6)\n",
        "\n",
        "# Plot the learning curve.\n",
        "plt.plot(df_stats['Training Loss'], 'b-o', label=\"Training\")\n",
        "plt.plot(df_stats['Valid. Loss'], 'g-o', label=\"Validation\")\n",
        "\n",
        "# Label the plot.\n",
        "plt.title(\"Training & Validation Loss\")\n",
        "plt.xlabel(\"Epoch\")\n",
        "plt.ylabel(\"Loss\")\n",
        "plt.legend()\n",
        "plt.xticks([1, 2, 3, 4])\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvAAAAGaCAYAAABpIXfbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeVyU5f4//tcM+74OyI5gLAIC7iZlbohLWYrL0aOlWVmZ/urjST22ek6n8zU7mFmetKyOuaTigmluqJWnjjsICpqQCoIwbMMOM8z9+2OYkXEABwVmwNfz8ehRXPd9X/d1j9z25uJ9XW+RIAgCiIiIiIioSxAbegBERERERKQ/BvBERERERF0IA3giIiIioi6EATwRERERURfCAJ6IiIiIqAthAE9ERERE1IUwgCeih15ubi6Cg4Px6aef3ncfS5cuRXBwcDuOqvtq6fMODg7G0qVL9erj008/RXBwMHJzc9t9fLt27UJwcDBOnTrV7n0TEbUHU0MPgIjobm0JhJOTk+Ht7d2Bo+l6qqur8e9//xsHDhxAYWEhnJ2d0a9fP7zyyisIDAzUq4+FCxfi0KFD2LNnD0JDQ5s9RxAEjBw5EuXl5Th58iQsLS3b8zE61KlTp3D69Gk8++yzsLe3N/RwdOTm5mLkyJGYOXMm3nnnHUMPh4iMDAN4IjI6K1eu1Pr63Llz+P777zFt2jT069dP65izs/MD38/LywsXL16EiYnJfffxt7/9De+///4Dj6U9vPXWW9i/fz8mTJiAgQMHQiqV4tixY0hNTdU7gI+Pj8ehQ4eQmJiIt956q9lz/ve//+HWrVuYNm1auwTvFy9ehFjcOb8YPn36NNauXYtnnnlGJ4CfOHEixo8fDzMzs04ZCxFRWzGAJyKjM3HiRK2vGxoa8P333yMqKkrn2N0qKytha2vbpvuJRCJYWFi0eZxNGUuwV1NTg4MHDyImJgYff/yxpn3BggWor6/Xu5+YmBh4eHhg3759ePPNN2Fubq5zzq5duwCogv328KB/Bu3FxMTkgX6YIyLqaMyBJ6Iua8SIEZg1axYuX76M559/Hv369cNTTz0FQBXIJyQkYMqUKRg0aBDCw8MxevRorFq1CjU1NVr9NJeT3bTt+PHjmDx5MiIiIhATE4P/9//+HxQKhVYfzeXAq9sqKirw7rvvYsiQIYiIiMD06dORmpqq8zylpaVYtmwZBg0ahOjoaMyePRuXL1/GrFmzMGLECL0+E5FIBJFI1OwPFM0F4S0Ri8V45plnUFZWhmPHjukcr6ysxOHDhxEUFIQ+ffq06fNuSXM58EqlEl988QVGjBiBiIgITJgwAUlJSc1en5WVhffeew/jx49HdHQ0IiMjMWnSJOzYsUPrvKVLl2Lt2rUAgJEjRyI4OFjrz7+lHPiSkhK8//77GDZsGMLDwzFs2DC8//77KC0t1TpPff1vv/2Gr776CqNGjUJ4eDjGjBmD3bt36/VZtEVmZiZeffVVDBo0CBERERg3bhw2bNiAhoYGrfPy8/OxbNkyDB8+HOHh4RgyZAimT5+uNSalUolvvvkGTz75JKKjo9G3b1+MGTMGf/3rXyGXy9t97ER0fzgDT0RdWl5eHp599lnExcUhNjYW1dXVAICCggLs3LkTsbGxmDBhAkxNTXH69Gl8+eWXyMjIwFdffaVX/z/99BO2bNmC6dOnY/LkyUhOTsbGjRvh4OCA+fPn69XH888/D2dnZ7z66qsoKyvD119/jRdffBHJycma3xbU19djzpw5yMjIwKRJkxAREYErV65gzpw5cHBw0PvzsLS0xNNPP43ExET88MMPmDBhgt7X3m3SpElYt24ddu3ahbi4OK1j+/fvR21tLSZPngyg/T7vu3344Yf4z3/+gwEDBuC5555DcXExVqxYAR8fH51zT58+jbNnz+KJJ56At7e35rcRb731FkpKSvDSSy8BAKZNm4bKykocOXIEy5Ytg5OTE4DW115UVFTgT3/6E27cuIHJkyejd+/eyMjIwNatW/G///0PO3bs0PnNT0JCAmprazFt2jSYm5tj69atWLp0KXx9fXVSwe5XWloaZs2aBVNTU8ycOROurq44fvw4Vq1ahczMTM1vYRQKBebMmYOCggLMmDED/v7+qKysxJUrV3D27Fk888wzAIB169ZhzZo1GD58OKZPnw4TExPk5ubi2LFjqK+vN5rfNBE99AQiIiOXmJgoBAUFCYmJiVrtw4cPF4KCgoTt27frXFNXVyfU19frtCckJAhBQUFCamqqpi0nJ0cICgoS1qxZo9MWGRkp5OTkaNqVSqUwfvx4YejQoVr9LlmyRAgKCmq27d1339VqP3DggBAUFCRs3bpV0/bdd98JQUFBwueff651rrp9+PDhOs/SnIqKCuGFF14QwsPDhd69ewv79+/X67qWzJ49WwgNDRUKCgq02qdOnSqEhYUJxcXFgiA8+OctCIIQFBQkLFmyRPN1VlaWEBwcLMyePVtQKBSa9vT0dCE4OFgICgrS+rOpqqrSuX9DQ4Pw5z//Wejbt6/W+NasWaNzvZr6++1///ufpu1f//qXEBQUJHz33Xda56r/fBISEnSunzhxolBXV6dpv337thAWFia8/vrrOve8m/ozev/991s9b9q0aUJoaKiQkZGhaVMqlcLChQuFoKAg4ddffxUEQRAyMjKEoKAgYf369a329/TTTwtjx4695/iIyLCYQkNEXZqjoyMmTZqk025ubq6ZLVQoFJDJZCgpKcGjjz4KAM2msDRn5MiRWrvciEQiDBo0CFKpFFVVVXr18dxzz2l9PXjwYADAjRs3NG3Hjx+HiYkJZs+erXXulClTYGdnp9d9lEolFi1ahMzMTPz44494/PHHsXjxYuzbt0/rvLfffhthYWF65cTHx8ejoaEBe/bs0bRlZWUhJSUFI0aM0Cwibq/Pu6nk5GQIgoA5c+Zo5aSHhYVh6NChOudbW1tr/ruurg6lpaUoKyvD0KFDUVlZiezs7DaPQe3IkSNwdnbGtGnTtNqnTZsGZ2dnHD16VOeaGTNmaKUtubu7o2fPnrh+/fp9j6Op4uJiXLhwASNGjEBISIimXSQS4eWXX9aMG4Dme+jUqVMoLi5usU9bW1sUFBTg7Nmz7TJGIuoYTKEhoi7Nx8enxQWHmzdvxrZt23Dt2jUolUqtYzKZTO/+7+bo6AgAKCsrg42NTZv7UKdslJWVadpyc3Ph5uam05+5uTm8vb1RXl5+z/skJyfj5MmT+Oijj+Dt7Y1PPvkECxYswJtvvgmFQqFJk7hy5QoiIiL0yomPjY2Fvb09du3ahRdffBEAkJiYCACa9Bm19vi8m8rJyQEABAQE6BwLDAzEyZMntdqqqqqwdu1a/Pjjj8jPz9e5Rp/PsCW5ubkIDw+Hqan2/zZNTU3h7++Py5cv61zT0vfOrVu37nscd48JAHr16qVzLCAgAGKxWPMZenl5Yf78+Vi/fj1iYmIQGhqKwYMHIy4uDn369NFc98Ybb+DVV1/FzJkz4ebmhoEDB+KJJ57AmDFj2rSGgog6FgN4IurSrKysmm3/+uuv8c9//hMxMTGYPXs23NzcYGZmhoKCAixduhSCIOjVf2u7kTxoH/pery/1ossBAwYAUAX/a9euxcsvv4xly5ZBoVAgJCQEqamp+OCDD/Tq08LCAhMmTMCWLVtw/vx5REZGIikpCT169MBjjz2mOa+9Pu8H8X//9384ceIEpk6digEDBsDR0REmJib46aef8M033+j8UNHROmtLTH29/vrriI+Px4kTJ3D27Fns3LkTX331FebNm4e//OUvAIDo6GgcOXIEJ0+exKlTp3Dq1Cn88MMPWLduHbZs2aL54ZWIDIsBPBF1S3v37oWXlxc2bNigFUj9/PPPBhxVy7y8vPDbb7+hqqpKaxZeLpcjNzdXr2JD6ue8desWPDw8AKiC+M8//xzz58/H22+/DS8vLwQFBeHpp5/We2zx8fHYsmULdu3aBZlMBqlUivnz52t9rh3xeatnsLOzs+Hr66t1LCsrS+vr8vJynDhxAhMnTsSKFSu0jv366686fYtEojaP5Y8//oBCodCahVcoFLh+/Xqzs+0dTZ3ade3aNZ1j2dnZUCqVOuPy8fHBrFmzMGvWLNTV1eH555/Hl19+iblz58LFxQUAYGNjgzFjxmDMmDEAVL9ZWbFiBXbu3Il58+Z18FMRkT6Ma3qAiKidiMViiEQirZlfhUKBDRs2GHBULRsxYgQaGhrwn//8R6t9+/btqKio0KuPYcOGAVDtftI0v93CwgL/+te/YG9vj9zcXIwZM0YnFaQ1YWFhCA0NxYEDB7B582aIRCKdvd874vMeMWIERCIRvv76a60tES9duqQTlKt/aLh7pr+wsFBnG0ngTr68vqk9o0aNQklJiU5f27dvR0lJCUaNGqVXP+3JxcUF0dHROH78OK5evappFwQB69evBwCMHj0agGoXnbu3gbSwsNCkJ6k/h5KSEp37hIWFaZ1DRIbHGXgi6pbi4uLw8ccf44UXXsDo0aNRWVmJH374oU2Ba2eaMmUKtm3bhtWrV+PmzZuabSQPHjwIPz8/nX3nmzN06FDEx8dj586dGD9+PCZOnIgePXogJycHe/fuBaAKxj777DMEBgZi7Nixeo8vPj4ef/vb3/DLL79g4MCBOjO7HfF5BwYGYubMmfjuu+/w7LPPIjY2FsXFxdi8eTNCQkK08s5tbW0xdOhQJCUlwdLSEhEREbh16xa+//57eHt7a603AIDIyEgAwKpVq/Dkk0/CwsICjzzyCIKCgpody7x583Dw4EGsWLECly9fRmhoKDIyMrBz50707Nmzw2am09PT8fnnn+u0m5qa4sUXX8Ty5csxa9YszJw5EzNmzIBEIsHx48dx8uRJTJgwAUOGDAGgSq96++23ERsbi549e8LGxgbp6enYuXMnIiMjNYH8uHHjEBUVhT59+sDNzQ1SqRTbt2+HmZkZxo8f3yHPSERtZ5z/JyMiekDPP/88BEHAzp078cEHH0AikWDs2LGYPHkyxo0bZ+jh6TA3N8e3336LlStXIjk5GT/++CP69OmDb775BsuXL0dtba1e/XzwwQcYOHAgtm3bhq+++gpyuRxeXl6Ii4vD3LlzYW5ujmnTpuEvf/kL7OzsEBMTo1e/Tz75JFauXIm6ujqdxatAx33ey5cvh6urK7Zv346VK1fC398f77zzDm7cuKGzcPSjjz7Cxx9/jGPHjmH37t3w9/fH66+/DlNTUyxbtkzr3H79+mHx4sXYtm0b3n77bSgUCixYsKDFAN7Ozg5bt27FmjVrcOzYMezatQsuLi6YPn06XnvttTZX/9VXampqszv4mJub48UXX0RERAS2bduGNWvWYOvWraiuroaPjw8WL16MuXPnas4PDg7G6NGjcfr0aezbtw9KpRIeHh546aWXtM6bO3cufvrpJ2zatAkVFRVwcXFBZGQkXnrpJa2dbojIsERCZ6wsIiKi+9LQ0IDBgwejT58+910MiYiIuhfmwBMRGYnmZtm3bduG8vLyZvc9JyKihxNTaIiIjMRbb72F+vp6REdHw9zcHBcuXMAPP/wAPz8/TJ061dDDIyIiI8EUGiIiI7Fnzx5s3rwZ169fR3V1NVxcXDBs2DAsWrQIrq6uhh4eEREZCQbwRERERERdCHPgiYiIiIi6EAbwRERERERdCBextlFpaRWUys7POnJxsUVxcWWn35eoq+G7QqQfvitE+jHEuyIWi+DkZNPicQbwbaRUCgYJ4NX3JqJ747tCpB++K0T6MbZ3hSk0RERERERdCAN4IiIiIqIuhAE8EREREVEXwgCeiIiIiKgLYQBPRERERNSFcBeaDlBTU4XKShkaGuTt1mdhoRhKpbLd+iPDMjExg62tA6ysWt4iioiIiKg5DODbmVxej4qKUjg6usLMzAIikahd+jU1FUOhYADfHQiCALm8DmVlRTA1NYOZmbmhh0RERERdCFNo2llFRRlsbR1gbm7ZbsE7dS8ikQjm5pawsXFAZWWZoYdDREREXQwD+HamUNTDwsLK0MOgLsDS0gpyeb2hh0FERERdDFNo2plS2QCx2MTQw6AuQCw2gVLZYOhhEBERUTNO3z6PpKyDKKsrg6OFI54KjMPAHn0NPSwADOA7BFNnSB/8PiEiIjJOp2+fx5bMRMiVqg1JSuvKsCUzEQCMIohnCg0RERERUaPy+gok/r5PE7yryZVyJGUdNNCotHEGnozCggUvAgDWrl3fqdcSERHRw0spKFFQLUV22XVkyVT/FNUUt3h+aZ1xbD7BAJ5aFRPTX6/zduxIgoeHZwePhoiIiOj+1TfIcaM8B9my643/3EC1ogYAYGtmg0AHfzzmNRhHb/6EivpKneudLBw7e8jNYgBPrXr77RVaX2/fvhUFBfl47bU3tNodHZ0e6D4JCZ8Z5FoiIiLqvirqK5Elu66ZYc+puIUGQbWBhLu1G6IkEQhw9Eeggx8kVq6a9Wn25nZaOfAAYCY2w1OBcQZ5jrsxgKdWjRkzTuvrEyeSIZOV6bTfrba2FpaWlnrfx8zM7L7G96DXEhERUfdwdzpMtuw6pI3pMKZiU/jZeWOEz2MIdPRHTwc/2Jq1XA1dvVCVu9BQt7VgwYuorKzEm2/+FZ9+moArVzIxc+ZsPP/8S/jllxNIStqNq1evoLxcBonEDePGPYlZs+bAxMREqw/gTh77+fNnsXDhfHzwwUr88Uc29uxJRHm5DBERkfjLX/4Kb2+fdrkWABITt2Pbts0oLi5CYGAgFix4HRs2rNPqk4iIiIxLfYMcNytyNQH7H7IbqFJUA1ClwwQ4+GOo5yAEOvaEj50XzMRtC3sH9uiLgT36QiKxg1Ra0RGPcN8MGsDX19fjk08+wd69e1FeXo6QkBC8/vrrGDJkSKvXJSUlYefOncjKyoJMJoObmxsGDRqEBQsWwMvLS+f8HTt2YOPGjcjNzYWnpydmz56NmTNndtRjtbvfLt3Grp+zUSyrhYu9BSYNC8SQsB6GHpaWsrJSvPnm64iNjUNc3Hi4u6vGd+DAD7Cyssa0aTNhbW2Fc+fO4ssv/42qqiq8+uqie/b77bdfQSw2wYwZs1FRUY6tWzfh/fffwoYN37bLtbt370RCwkpERfXFtGl/Qn5+PpYtWww7OztIJG73/4EQERFRu2qaDpMtu46bd6XDRErCEODgjwBHf7g1SYfpjgwawC9duhSHDx/G7Nmz4efnh927d+OFF17Apk2bEB0d3eJ1mZmZcHd3x7Bhw+Dg4IC8vDxs374dJ06cQFJSEiQSiebcbdu24d1330VcXBzmzJmDs2fPYsWKFairq8PcuXM74zEfyG+XbuPbHzNRr1ACAIrL6/Dtj5kAYFRBfFGRFEuXvo0JEyZqtb/33t9hYXEnlebpp+Px0Uf/wO7dO/DCCy/D3Ny81X4VCgU2bvwWpqaqb1V7ewd88skqZGdfQ0BArwe6Vi6X48sv1yEsLAKrV3+uOa9Xr0fwwQfvMYAnIiIyEEEQUFBd2Biw30C27DoKa4oAAKYiE/ja+2CEz2MIcPBDgIM/bM1bTofpjgwWwF+8eBH79+/HsmXL8NxzzwEAnn76aUyYMAGrVq3C5s2bW7z2zTff1GkbOXIkJk2ahKSkJDz//PMAVHnYCQkJGDlyJD755BMAwNSpU6FUKrF27VpMmTIFdnZ27f9wd/lvWj5OXsy/r2uz8mRQNAhabfUKJb4+kIGfU/La1FdMHw8MjfC4r3Hci6WlJeLixuu0Nw3eq6urUF8vR2RkNPbu3YUbN67jkUeCWu13/PinNIE1AERGRgEA8vJu3TOAv9e1mZmXIZPJ8Morz2idN3p0HNas+VerfRMREVH7kTfIceMe6TCPeg5EoKM/fOy825wO090Y7OkPHjwIMzMzTJkyRdNmYWGB+Ph4JCQkoLCwEG5u+s+AenqqtjAsLy/XtJ06dQplZWWYMWOG1rkzZ87Evn378PPPP2P8eN2g05jcHbzfq91QJBI3rSBYLTs7Cxs2rMP582dQVVWldayqSnd7prupU3HU7OzsAQAVFffORbvXtbdvq36oujsn3tTUFB4eHfODDhEREanSYbIb913PLruBnIpcKDTpMBL0aUyHCXTwg5u1pFunw9wPgwXwGRkZ6NmzJ2xstH/l0adPHwiCgIyMjHsG8GVlZWhoaEBeXh4++0y1lWDT/PnLly8DAMLDw7WuCwsLg1gsxuXLlzslgB8acf8z33/5/L8oLq/TaXext8CSmcaxEhrQnmlXq6iowGuvvQhra1s8//x8eHl5w9zcHFevZmLduk+hVCrv2a9YbNJsuyDc+weYB7mWiIiI2ocqHUbaJGC/Ox3GG8Mf4nSY+2GwAF4qlcLd3V2nXZ2/XlhYeM8+xowZg7IyVUUsR0dHvPPOOxg8eLDWPczNzeHoqL3pvrpNn3sY2qRhgVo58ABgbirGpGGBBhyVfi5cOAeZTIYPPvgIUVF3ftjIz29b6k9H6dFD9UNVbm4OIiPvrLlQKBTIz89HYGDrKTpERESkS5MO06RYUpVclQ5jY2atSYcJcPCHr50XzEy4HXRbGSyAr62tbXb/bgsLCwBAXZ3urPPd1q5di+rqavzxxx9ISkrSSdFo6R7q++hzj7u5uNi2erywUAxTU3Gb+23JY5GeMDERYcfxLNUuNA6WmDI8EI+GGybFQ/0rrKbPKBKJIBJB57nNzFQz4GKxSHNMLpdjz56dAAATkzuf1d39mpio/y3S6lfd3rTP+702PDwcDg6O2LdvN8aPn6BJATp06BAqKsohEona9c+yOWKxGBJJx6/DeNjwMyXSD98Vag/ltRW4UpyNK0VZyJRmIbv0JhRKBQDAw84NA7wjEeLaCyGuAfCwc++S6TDG9q4YLIC3tLSEXC7XaVcH1epAvjUDBgwAAAwbNgwjR47Ek08+CWtra/z5z3/W3KO+vr7Za+vq6vS6x92KiyuhVLacgqFUKqFQ3Ds1pC0Ghrjj0XAPrX7b+x76UqefNL2/IAgQBN0x9e4dATs7e6xY8Q7i46dBJBLh0KEDms+voeHOZ3V3vw0N6n8LWv2q25VK4YGvFYlMMHfuC0hI+AgLFszH8OEjkZ+fjx9/3AcvL2+d5+wISqXS6PaW7eqMcb9eImPEd4Xuh046jOw6Cqu102Ge8B6q2s7RwQ925k0mPuuAorp7r38zNoZ4V8RiUauTxgYL4CUSSbMpLFKpFADatIAVAHx8fBAWFoZ9+/ZpAniJRAK5XI6ysjKtNJr6+nqUlZW1+R7UNg4Ojli5MgFr167Ghg3rYGdnj9jYsejffyDeeGOBoYcHAJg8eRoEQcC2bZvx2WefIDDwEfzzn//C6tWrYG7e9h/wiIiIuhN5gxw3K25pAvY/ZDdQKVdlPKjSYfwwxGMAAhz84WfnzXSYTmKwAD4kJASbNm1CVVWV1kLW1NRUzfG2qq2tRU1Njebr0NBQAEB6ejpiYmI07enp6VAqlZrjpL8PP/xYp621aqUREZH44ouvddpPnjzbah99+/bXOQcAPDw82/VaAIiPn474+Omar5VKJfLz8xAUFNzMExEREXVflfVVmpn1bNl13Cy/szuMm7Urwl1DEejgjwAHf7hzdxiDMVgAHxcXh40bN2LHjh2afeDr6+uxa9cu9O3bV7PANS8vDzU1NQgMvLNos6SkBM7Ozlr9paenIzMzE+PGjdO0DR48GI6OjtiyZYtWAL9161ZYW1vj8ccf78AnpK6guVSqgwf3o7xchujofgYaFRERUccTBAGF1VJkyW5oAvaCalUmhInIBL523hjmM1QTsGulw5BBGSyAj4yMRFxcHFatWgWpVApfX1/s3r0beXl5+PDDDzXnLVmyBKdPn8aVK1c0bcOHD8fYsWMRFBQEa2trXLt2DYmJibCxscErr7yiOc/S0hILFy7EihUrsGjRIsTExODs2bNISkrC4sWLYW9v36nPTMbn4sUUrFv3KZ54YgTs7R1w9Wom9u9PQkBAIIYPH2Xo4REREbUbuVKBnIpcZJU1kw5jao0ARz8M9ujPdJguwKBlrFauXInVq1dj7969kMlkCA4Oxvr169GvX+sznzNmzMBvv/2Go0ePora2FhKJBHFxcXjllVfg46NdlGfmzJkwMzPDxo0bkZycDA8PDyxfvhyzZ8/uyEejLsLT0wuurhLs3Pk9ystlsLd3QFzceMyfv6DFHYyIiIi6gsr6Ks02jlmy67hZkavZHcbNyhXhLqEIcPRDoIM/3KwlEIs6duc1aj8igVVt2uReu9Dcvn0DPXr4tft9TU3FBtt5hjpOR32/PMy4swaRfviudC+CIKCwpgjZZdc1C06102G8EODoz3SY+8BdaIiIiIjogTVNh8luzGFvmg7T08EPg3v0R4CjP3ztvGHOdJhuhQE8ERERkZGrlFfhD9mNxoD9Om40SYeRWLkgzCUEgY0z7EyH6f4YwBMREREZEUEQIK0p0gTrWbIbKKhW1c5Rp8MM83oUAY6qYkn25sZVJZQ6HgN4IiIiIgNSpcOoiiVlN+4Qo06HsTa1QoCDHwb16KvaHcbeh+kwxACeiIiIqDNVyatVM+utpcM4+CPAUVUsiekwdDcG8EREREQdRJMOI7uh2SHmdpN0GB87LzzuNUQTsDMdhvTBAJ6IiIionSga02GyGtNhsmU3UCGvBABYmVoh0MEPA5kOQw+IATx1ugMH9uEf/3gfO3YkwcPDEwAQH/8koqP7Yfny99p87YM6f/4sFi6cjzVr/o2+ffu3S59ERPRwUKfDZDfuEHOzIgfyxnQYVysX9HYJRoCDHwIc/NHDxo3pMNQuGMDTPb355us4f/4M9u07Aisrq2bPeeONBbh0KQ1JSYdhYWHRySPUz9Gjh1BSUoypU2cYeihERNQFqdJhihsDdtXuMLerCgAAYpEYPnZeeKwxHaangz8cLJgOQx2DATzd0+jRY/Drr7/g5MmfMHp0nM7x0tISnDt3BrGxY+87eN+yJRFiccfOSiQnH8bvv1/VCeCjovoiOfm/MDPjrzGJiOgOrXSYxmJJFfV30mECHPwwwD0agQ5+jekw5gYeMT0sGMDTPXyS+qQAACAASURBVD322BOwsrLG0aOHmg3gjx07ioaGBsTG6h7Tl7m54f7SE4vFRvtbAyIi6jzV8mpVKkzjDPuN8ibpMJbOCHUOQoCDqlgS02HIkBjA0z1ZWlrisceG4fjxoygvL4e9vb3W8aNHD8HFxQU+Pn5YteqfOHfuNAoKCmBpaYm+ffvj1VcX3TNfvbkc+OzsLKxe/RHS09Pg4OCAiRMnwdVVonPtL7+cQFLSbly9egXl5TJIJG4YN+5JzJo1ByYmJgCABQteRErKeQBATIwqz71HDw/s3LmvxRz45OTD+O67b3DjxnVYW9tg6NDH8PLLC+Ho6Kg5Z8GCF1FZWYl33lmBf/1rJTIyLsHOzh5TpkzHzJnPtu2DJiKiTiMIAopqShpTYf5oMR0mwEFVLMnBwv4ePRJ1HgbwXcDp2+exL/sgSmrL4GThiKcC4zCwR99OHcPo0XE4fPhHnDiRjKeeekbTfvt2PtLTLyI+fjoyMi4hPf0iRo0aA4nEDfn5edizJxGvvfYSvvtuBywtLfW+X3FxERYunA+lUok///lZWFpaISlpd7Mz5QcO/AArK2tMmzYT1tZWOHfuLL788t+oqqrCq68uAgA8++xc1NTUoKAgH6+99gYAwMrKusX7qxfLhoVF4OWXF6KwsACJid8jI+MSNmz4j9Y4ystl+L//W4jhw0di5MhYHD9+FOvWfYqAgF4YMmSo3s9MREQdR5UOk9ckf71pOowlejr4YYB7FAIc/OHPdBgycgzgjdzp2+exJTMRcqUcAFBaV4YtmYkA0KlB/IABg+Do6ISjRw9pBfBHjx6CIAgYPXoMAgN7YfjwUVrXDR36OObPn4MTJ5IRFzde7/tt3vwtZLIyfPnlJgQHhwAAxo6dgD/96Rmdc9977++wsLjzw8HTT8fjo4/+gd27d+CFF16Gubk5BgwYjF27dkAmK8OYMeNavbdCocC6dZ+iV68gfPrpF5r0nuDgELz33nLs27cb8fHTNecXFhbg3Xf/rkkvmjBhIuLjJ2D//r0M4ImIDKS1dBgXpsNQF8cAvhOcyj+H3/LP3Ne1f8huQiEotNrkSjk2Z+zEr3mn29TXEI8BGOTR777GYWpqihEjRmHPnkQUFRXB1dUVAHD06GF4e/ugd+9wrfMVCgWqqirh7e0DW1s7XL2a2aYA/rff/ouIiEhN8A4ATk5OGD16LHbv3qF1btPgvbq6CvX1ckRGRmPv3l24ceM6HnkkqE3Pmpl5GaWlJZrgX23EiNH47LNP8Ouv/9UK4G1tbTFq1BjN12ZmZggNDUNe3q023ZeIiO6PdjqMKmDPb5oOY+uFGK/BmoCd6TDU1TGAN3J3B+/3au9Io0fHYdeuHTh27DCmTp2B69f/wLVrVzFnzgsAgLq6Wmza9A0OHNgHqbQQgiBorq2srGzTvQoKbiMiIlKn3dfXT6ctOzsLGzasw/nzZ1BVVaV1rKqqbfcFVGlBzd1LLBbD29sHBQX5Wu1ubu4QiURabXZ29sjKutbmexMR0b01KBuQU3kL2WWqrRyzZddRXl8BoDEdxt4P/RvTYfzsfWDBdBjqZhjAd4JBHv3ue+b7rf/+A6V1ZTrtThaO+P/6zn/QobVJREQkPDy8cOTIQUydOgNHjhwEAE3qSELCRzhwYB+mTPkTwsMjYGtrC0CE9977q1Yw354qKirw2msvwtraFs8/Px9eXt4wNzfH1auZWLfuUyiVyg65b1NisUmz7R31zEREDxt1Oox6K8fr5Tma1FIXS2cEOz2CQEdVsSQPG3emw1C3xwDeyD0VGKeVAw8AZmIzPBV4/1s2PohRo2KxadPXyM3NQXLyYQQHh2pmqtV57q+99rrm/Lq6ujbPvgOAu3sP5Obm6LTfvHlD6+sLF85BJpPhgw8+QlTUnTUB+fl5zfQqaqZNV48eHpp7Ne1TEATk5uagZ89AvfohIqK2EwQBxbUlyCq7rqlwml9VAAECxCIxvG09EeM5CAGOqt1hHC0cDD1kok7HAN7IqReqGnoXGrXY2LHYtOlrrF2bgNzcHK1gvbmZ6MTE79HQ0NDm+wwZMhQ7dmzDlSuZmjz40tJSHDnyo9Z56uJPTWe75XK5Tp48AFhZWen1w0RISG84OTljz56dGDt2gqbA0/HjyZBKCzFz5uw2Pw8RETVPn3SYvm6RCHT0g5+9L9NhiMAAvksY2KMvHvXuD4Wi49NB7qVnzwD06hWEkyd/hlgsxsiRdxZvPvpoDA4dOgAbG1v4+/fEpUtpOHv2NBwc2j47MmPGszh06ADeeONVxMdPh4WFJZKSdsPd3QOVlb9rzouI6AM7O3t88MF7iI+fBpFIhEOHDqC57JXg4BAcPvwjPv30XwgJ6Q0rK2vExDyuc56pqSlefvk1/OMf7+O1117CqFGxKCwswM6d3yMgIBBPPqm7Ew4REemnWl6DP8pvNAbsd6fDOCHYqZdqsakj02GIWsIAntosNjYO165dRXR0P81uNACwaNFiiMViHDnyI+rq6hEREYnVqz/DG2+81uZ7uLq6Ys2aL5CQsBKbNn2jVcjpn//8m+Y8BwdHrFyZgLVrV2PDhnWws7NHbOxY9O8/EG+8sUCrz4kTJ+Pq1UwcOPADvv9+C3r08Gg2gAeAceOehLm5OTZv/hafffYJbGxsMHp0HObPf41VW4mI9KRKhylV7Q5T9gfTYYjaiUjgSrs2KS6uhFLZ8kd2+/YN9Oihu1PKgzI1FRvFDDy1r476fnmYSSR2kEorDD0MIqPXEe9Kg7IBuZV5qq0cG3PYZY3pMJYmlujp4ItAB3/N7jCWppwQIeNniP+viMUiuLjYtnicM/BERER0X2oUNciW3dTMsN8oz0F9YzqMs6UTgpgOQ9QhGMATERHRPWmlwzTOsGunw3jgUc+BmoCd6TBEHYcBPBEREelQp8Nky25oAnZZfTkAwNLEAj0d/BDtFoFAh55MhyHqZAzgiYiIHiKnb59HUtZBlNWVwbHJ1sRN02Gyy67jevlNrXSYR5wCNPnrnrY9mA5DZEAM4ImIiB4Sp2+f1yoOWFpXhk0Z27E360fI6sohQIAIInjbeWKI50AEOqiqmzpZOhp45ETUFAN4IiKih4BSUGLX7z9oVfZWt1fKqzCu5ygEOPjD396X6TBERo4BPBERUTelUCpwpTQLqdI0pEovoVJe1eJ543qO7uTREdH9YgDfAQRBgEgkMvQwyMixBAMRdYT6BjkySq7gQmE60osvo0ZRCwsTc4S7hCKz9BqqmgninSyYIkPUlTCAb2cmJqaQy+thbs5fP1Lr5PJ6mJjwFSSiB1erqEV6cSZSCtNwqTgT9Uo5rE2tEOkajii3cIQ4PQIzEzOdHHgAMBOb4anAOAOOnojaitFDO7O1dURZmRSOjhKYmZlzJp50CIIAubweZWVS2Nk5GXo4RNRFVcqrkFaUgZTCNGSW/g6FUgE7c1sM9OiHaEkEHnEMgInYROuagT36AkCzu9AQUdfBAL6dWVnZAABksiI0NCjarV+xWAylUtlu/ZFhmZiYws7OSfP9QkSkD1ldBS4WpSOlMB1Xy7KgFJRwsnDEY16DESWJQICD3z23dxzYoy8G9uhrkPLwRNQ+GMB3ACsrm3YPzPgXLRHRw6m4phSp0jSkSNORLbsBAQLcrF0xyncYoiTh8LXz5m97iR4yDOCJiIiMTEG1FCmFaUiRpuFmxS0AgJetB8b1HIUoSQQ8bNwZtBM9xBjAExERGZggCMiruo0LjUF7flUBAMDf3hdPB45DpCQcbtauBh4lERkLBvBEREQGIAgCrpfnIFWajhRpGqQ1xRBBhF6OPRH/yFOIkoSzAioRNYsBPBERUSdRCkpklf2BFGk6UqTpKKuTQSwSI9ipF0b5DkOkJBx25raGHiYRGTkG8ERERB1IoVTgamkWUppUQzUTmyLUORhPBcQhwjUU1mbWhh4mEXUhDOCJiIjamaoa6lWkSNOQVpSBGkWNphpqlFsEejsHw9KUBf+I6P4wgCciImoHtYpaXCrOxAVpuqoaakN9YzXUMK1qqERED4oBPBER0X2qklfjYtFlpErTkFHSpBpqj76IkoQjyDFQpxoqEdGDYgBPRETUBu1RDZWI6EEwgCciIrqHktpS1c4xhWl3qqFasRoqERkGA3giIqJmFFRLkVqYjgvSNNysyAWgqoY6tucoRLMaKhEZkEED+Pr6enzyySfYu3cvysvLERISgtdffx1Dhgxp9brDhw/jwIEDuHjxIoqLi+Hh4YHhw4fjlVdegZ2dnda5wcHBzfbx3nvv4U9/+lO7PQsREXVt6mqoKYVpSJGmI6/qNgDAz96nsRpqGNysJQYeJRGRgQP4pUuX4vDhw5g9ezb8/Pywe/duvPDCC9i0aROio6NbvO7tt9+Gm5sbJk6cCE9PT1y5cgWbNm3CL7/8gsTERFhYaG/NFRMTg6eeekqrLTIyskOeiYiIug5BEHCjIgcphdrVUAMd/VkNlYiMlsEC+IsXL2L//v1YtmwZnnvuOQDA008/jQkTJmDVqlXYvHlzi9euWbMGgwYN0moLDw/HkiVLsH//fkyaNEnrWEBAACZOnNjuz0BERF2PqhrqdaRI05qthtpHEgZ7c7t7d0REZCAGC+APHjwIMzMzTJkyRdNmYWGB+Ph4JCQkoLCwEG5ubs1ee3fwDgCjRo0CAGRlZTV7TW1tLUQikc7sPBERdX8NygZcLc3CBWkaLkovoUJeCVOxKXqzGioRdUEGC+AzMjLQs2dP2NjYaLX36dMHgiAgIyOjxQC+OUVFRQAAJycnnWM7d+7Epk2bIAgCgoKCsHDhQowePfrBHoCIiIyauhpqqjQdF4sua1VDjZSEI8wlhNVQiahLMlgAL5VK4e7urtMukagWCBUWFrapvw0bNsDExASxsbFa7dHR0Rg3bhy8vb2Rn5+P//znP1iwYAE+/vhjTJgw4f4fgIiIjI66GmqKNB3pTaqh9nHtjWi3CFZDJaJuwWABfG1tLczMdP8SVae41NXV6d3Xvn37sHPnTrz00kvw9fXVOrZt2zatr5955hlMmDABH330EcaPH9/mLcBcXGzbdH57kkiYk0mkD74rD5fK+iqcvXURp3NTkHr7MuRKBRws7PC4/yAM9o5Gb7cgmLIaarP4rhDpx9jeFYMF8JaWlpDL5Trt6sBd31z1s2fPYvny5XjiiSewaNGie55vbW2N6dOn4+OPP0Z2djYCAwPbNO7i4koolUKbrmkPEokdpNKKTr8vUVfDd+XhUF5fgVTpJaQUpmlVQ43xHIwoN+1qqKXF1QYerXHiu0KkH0O8K2KxqNVJY4MF8BKJpNk0GalUCgB65b9nZmbi5ZdfRnBwMBISEmBiot8Mi4eHBwBAJpO1YcRERGRIJbWlSJVewoXCNGTLrrMaKhE9tAwWwIeEhGDTpk2oqqrSWsiampqqOd6amzdvYt68eXB2dsYXX3wBa2v9dw/IyckBADg7O9/HyImIqLMUVksb92hPx40K1d/dnjY9MLbnKERJwuFp04NBOxE9dAwWwMfFxWHjxo3YsWOHZh/4+vp67Nq1C3379tUscM3Ly0NNTY1WqotUKsXcuXMhEonw1VdftRiIl5SU6BwrLS3Fli1b4O3tDX9//w55NiIiuj8tVkO188HEwLGIkoSzGioRPfQMFsBHRkYiLi4Oq1atglQqha+vL3bv3o28vDx8+OGHmvOWLFmC06dP48qVK5q2efPmIScnB/PmzcO5c+dw7tw5zTFfX19NFdfNmzcjOTkZTzzxBDw9PVFQUIDvv/8eJSUl+OyzzzrvYYmIqEWCIOBmRS4uFKYhVZqOwpoiiCBCgIOqGmqkJAzOlrpbBBMRPawMFsADwMqVK7F69Wrs3bsXMpkMwcHBWL9+Pfr169fqdZmZmQCAL7/8UufYM888owngo6Ojcf78eezYsQMymQzW1taIiorCSy+9dM97EBFRx1FXQ02VqtJjSuvKNNVQR/g+jkhWQyUiapFIEITO31KlC+MuNETGje+K8VJXQ02RpiG16BIq6lXVUEOdgxAtiUC4ayhsWA210/BdIdIPd6EhIqKHiryxGmpKk2qo5ibmCHcJQZQkAmEuwbA0tTT0MImIuhQG8ERE1K5qFXWN1VDTcKk4E3UN9bBqrIYaJQlHiHMQzFkNlYjovjGAJyKiB1Ytr0ZaUQYuSNOQUXIVCqUCdma26O8ejWhJBIKcAmHCaqhERO2CATwREd0XdTXUVGk6rpRe01RDfcxzMCIl4Qh09NdUQyUiovbDAJ6IiPRWWluGFGk6UqRpyCpTVUOVWLlgpM/jiHaLYDVUIqJOwACeiIhaVVhdhBRpGlIK76qG6j8SUW4RrIZKRNTJGMATEZEWQRCQX1WAC9I0pBSmaVdDDRiLSLdwuLMaKhGRwTCAJyIiTTXUFGk6UgrTtKqhTn7kSURJwlkNlYjISDCAJyJ6SCkFJbJlN5BSmKZVDTXIMRAjfB9HH9cwOFiwGioRkbFhAE9E9BBpUDbgalkWUgrvrob6CCYExCLCtTeroRIRGTkG8ERE3Zy8QY7M0t9xoTANaUWXUd1YDTXMJQTRknCEuYSwGioRURfCAJ6IqBtSV0NNlaYjvThDqxpqpCQcoayGSkTUZTGAJyLqJtTVUFOk6cgouQJ5k2qoUZJwBDkFwlTMv/aJiLo6/k1ORNSFVdRXIlWajpQm1VAdLRww1HMQoiQRrIZKRNQNMYAnIupimquG6tpYDTXKLRx+dj4srERE1I0xgCci6gIKq4uQKk3HBWkabpTfqYYa5z8S0ayGSkT0UGEAT0RkhJpWQ02VpuNWZT4AwNfOm9VQiYgecgzgiYiMhFY1VGkaCqvV1VD9MPmRJxHpGg4XK1ZDJSJ62DGAJyIyIE01VGkaUgrvqobq8xj6uIazGioREWlhAE9E1MkalA34vSwbF6RpuCi9hPL6Ck011PEBsejDaqhERNQKBvBERJ1AXQ01pTAdaUWXUaWo1lRDjZKEI5zVUImISE8M4ImIOkitog6XS64gpTCtSTVUS0S49kaUJILVUImI6L4wgCciakfNVUO1NbNBf/coREkiWA2ViIgeGP8vYuR+u3Qbu37KQkl5HZztLTBpWCCGhPUw9LCIqImK+kpclF7CBWmaVjXURz0HIVoSjkDHnqyGSkRE7YYBvBH77dJtfPtjJuoVSgBAcXkdvv0xEwAYxBMZWGltGVKll5AiTcO1sj+0qqFGSsLhZ+/NoJ2IiDoEA3gjtuunLE3wrlavUGLXT1kM4IkMQFpdrNruUZqO6+U3AQAeNu6I8x+JKEk4vGw9WA2ViIg6HAN4I1ZcXtemdiJqX+pqqOqg/U41VC88FRCHKEk43G3cDDxKIiJ62DCAN2Iu9hbNBusiANuPXcPIft5wceC2c0TtqdVqqL0mIFISwWqoRERkUAzgjdikYYFaOfAAYGoigq+7LQ6fycHhMznoHyJB7ABfBHjaG3CkRF2buhpqqjQdKdJ0lNSWaqqhDvd+DJGSMDhY8B0jIiLjwADeiKnz3JvbhaZYVovkc7n4KfUWTmcUopeXA2IH+CA6yBUmYi6cI7oXdTXUFGk6UqXpqmqoIhOEOAdhXM/RiHANha2ZjaGHSUREpEMkCIJg6EF0JcXFlVAqO/8jk0jsIJVW6LTX1ClwMi0fR87koEhWC1cHS4zq543HIj1hZcGfz+jh09K7AgBypQKZJVe1q6GKzVTVUN0iEOYSAitWQ6WHRGvvChHdYYh3RSwWwcXFtsXjDODbyNgCeDWlUsCF34tw+MxN/J4rg6W5CR6P9MSoft5wdbTqxJESGcbp2+eRlHUQZXVlcLRwxFOBcRjYo69WNdRLxZmobaiDlaklwl16I9otvLEaqrmhh0/U6RjAE+mHAXw3YKwBfFN/5Jfj8JkcnMkohAAB/YLdEDvAB728HDp4lESGcfr2eWzJTIRcKde0mYhM4GnTA7erCzTVUCMlYYiURCCY1VCJGMAT6YkBfDfQFQJ4tZLyxjz5lDxU1ykQ6GmP2IG+6Ms8eepGlIISb/33H5DVl+scE0GEx70fRZQkHIEO/jARmxhghETGiQE8kX4YwHcDXSmAV6utV+C/abdx5EwOCstq4GJviVH9vfFYH09YW3IWkoyfIAiokFeisLqo8R8pCmuKIK0ugrSmCHKlosVrPxuxshNHStR1MIAn0o8xBvCM3h4CluamGNnPG8OjvZB6rQiHzuTg+2PXsPfkH3isjydG9feGhHnyZASq5TWQ1hShoFoKaXURCmsag/XqYtQ21GrOMxGZQGLlAom1K0JdgvBb3hlUK2p0+nOycOzM4RMREXUKBvAPEbFYhOggCaKDJLh+W5Unf+x8Lo6ey0HfIAnGDPBFoJc9S8FTh6pvkENaoz2Trv7vSnmV5jwRRHC2dIKbtSsGefjBzUoCN2tXuFm7wtnSCWLRnTQwb1tPnRx4M7EZngqM69RnIyIi6gxMoWmjrphC05rSirrGPPlbqKpVoKeHPWIH+KBfsASmJsyTp/vToGxAUW2Jaha9WoqCxnSXwuoilNaVaZ3rYG4HN2tVcC6xcoWbtQTu1q5wsXKBWRsWmra0Cw0RNY8pNET6McYUGgbwbdTdAni1uvoG/DddtZ98QWkNnO0tMLKfN4ZFesLa0qzD7ktdl1JQoqxOdicvvUaKwmpVoF5UWwKlcKeCsLWplSZIV82ku8DNWgKJlQss23nfdQYlRPrhu0KkHwbw3UB3DeDVlIKAi9eKcfjMTWTeLIOFmQke6+OBUf294eZk3eH3J+MiCAIq5VUoqG4MzjU56bqLR83FZqqg3NoV7o0z6ZLGlJfOrGjKoIRIP3xXiPRjjAE8c+BJi1gkQtQjroh6xBU3blfg8JkcHL9wC8nnchEdJEHsAB884u3APPlupkZRo7PDi/rruxePulq5wM3aFaHOQY056aqZdQdzrp8gIiLqDJyBb6PuPgPfnNKKOhw7n4sTF1R58v497BA70Af9g92YJ9+FqBePqnPRC2qkmv+ukFdqzlMtHnVsNi/dycLR6PdS56wikX74rhDpxxhn4BnAt9HDGMCr1ckb8Gv6bRw+k4OCkmo42VlgVD9vPB7lCRvmyRuFBmUDimtLGnPSm8yot7B4VKLJSXfVzKa7WjrDzKTr/nkaw7tC1BXwXSHSDwP4buBhDuDVlIKAtKxiHD6Tg4wbpbAwM0FMhAdGDfCGO/PkO9zdi0eb5qXfvXjUytQK7tYSSKxc4d4YpKuCdtd2XzxqLIzpXSEyZnxXiPRjjAE8c+CpzcQiESJ7uSKylytuFlTgyJkcnEi5hWPncxH1iCtiB/ggyMeR+dAPoOniUe2CRs0vHpVYu8LLzhPRbn3u5KVbucLGzJp/DkRERN2MQWfg6+vr8cknn2Dv3r0oLy9HSEgIXn/9dQwZMqTV6w4fPowDBw7g4sWLKC4uhoeHB4YPH45XXnkFdnZ2Oufv2LEDGzduRG5uLjw9PTF79mzMnDnzvsbMGfjmlVXW4dj5Wzhx4RYqa+Twc7dD7AAfDAhlnnxrtBaP3hWk1yjuXjzqrNmGUWKtnlGXcPHoXYz9XSEyFnxXiPRjjDPwBg3g33jjDRw+fBizZ8+Gn58fdu/ejfT0dGzatAnR0dEtXjdo0CC4ublh1KhR8PT0xJUrV7Bt2zb4+/sjMTERFhYWmnO3bduGd999F3FxcRg6dCjOnj2LvXv3YsmSJZg7d26bx8wAvnX18gb8euk2jpzJQX5xNRxtzVX7yUd5wdaq6+ZVP4j6BjmKaoo1wXnTQL25xaPqRaOavHQrCZwtjX/xqLHoKu8KkaHxXSHSDwP4Ji5evIgpU6Zg2bJleO655wAAdXV1mDBhAtzc3LB58+YWrz116hQGDRqk1bZnzx4sWbIEH374ISZNmgQAqK2txbBhw9CvXz98/vnnmnMXL16MY8eO4aeffmp2xr41DOD1oxQEpGeX4MiZm7h0vRTmZmIMjfDA6P4+6OHc/fLkm1s8Kq0uQkG1FGV1Mgi48z1jb27XGJi7au30IrFy6dKLR41FV3tXiAyF7wqRfowxgDdYDvzBgwdhZmaGKVOmaNosLCwQHx+PhIQEFBYWws3Nrdlr7w7eAWDUqFEAgKysLE3bqVOnUFZWhhkzZmidO3PmTOzbtw8///wzxo8f3x6PQ3cRi0ToE+iCPoEuyC2sxOEzOfglNQ8nzt9CZC9XjB7ggxDfrpUnrxSUkNWVq/LSa7T3TC+q0V086mbtil6OAZqqo+pA3aqbLh4lIiKizmGwAD4jIwM9e/aEjY12hcY+ffpAEARkZGS0GMA3p6ioCADg5OSkabt8+TIAIDw8XOvcsLAwiMViXL58mQF8J/B2s8Xc8aGY/EQgjp/PxbHzt5ByrQi+braIHeiDgaHuRpMnr148qlvQSAppTTHkSrnmXDOxGdysXeFl44FoSZ87eelWEi4eJSIiog5jsABeKpXC3d1dp10ikQAACgsL29Tfhg0bYGJigtjYWK17mJubw9HRUetcdVtb7wGg1V9ndDSJpG3pPsZGIgF6+btg9pPhOHEuF3t/zsKXP2Rg18/ZGD80AHFD/GFvY94pY6mW1+B2RSHyKgqRX1GA/Eqp6t8VhaiW12jOMxGJ4W4rQQ8HN/T1CoOHnTs87CTwsHOHk5UDxCLj+MGDtHX1d4Wos/BdIdKPsb0rBgvga2trYWamm++rXoBaV1end1/79u3Dzp078dJLL8HX1/ee91Dfpy33UGMOfPvoG+iM6AAnXPqjBIfO5GDTjxn4/sgVPBrhgdH9veHhYnPvTu5Bs3i0yaJRVY66FBX12otHnSwd4Wbliv5uUVrpLi6WTs0uHlVWAcVVVQ88Rmp/3e1dIeoofFeI9MMc+CYsLS0hl8t12tVBddOdZFpz9uxZLF++HE888QQWLVqkc4/6+vpmr6urq9P7HtQxRCIRItxbnAAAIABJREFUwgNcEB7gglxpJY6cycHJi/k4ceEW+gS6IHaAD0L9nFpNRVEtHi3VpLg03emltLZMa/Gonbkt3KwkiHAJVRUzatwrnYtHiYiIqCsxWAAvkUiaTWGRSqUAoFf+e2ZmJl5++WUEBwcjISEBJibaM6USiQRyuRxlZWVaaTT19fUoKytrU449dSxviS3mjAvF5GGBOH7hFo6fz8WqbSnwlthidH9vhPSyQkldMQprtGfSdRePWsLNSoJAB3+4edzZ6UVizcWjRERE1D20SwCvUCiQnJwMmUyG4cOHa/LYWxMSEoJNmzahqqpKayFramqq5nhrbt68iXnz5sHZ2RlffPEFrK11tyYMDQ0FAKSnpyMmJkbTnp6eDqVSqTlOhqdePFqkKIJbYDGGuVfgSsEt5FUUYmthBUTFd4J09eJRTxsPREkiNDPpbtausDWz4eJRIiIi6tbaHMCvXLkSp06dQmJiIgBV4DVnzhycPXsWgiDA0dER27dv18pFb05cXBw2btyIHTt2aPaBr6+vx65du9C3b1/NAte8vDzU1NQgMDBQc61UKsXcuXMhEonw1VdfwdnZudl7DB48GI6OjtiyZYtWAL9161ZYW1vj8ccfb+vj0wOqUdRCevcOL43/rlHcWTwqFonhauWMUE9PmMhtcSsPuJUDmChsMSDYH7HhvvB0ffA8eSIiIqKups0B/C+//IJHH31U8/WxY8dw5swZzJs3D6Ghofjb3/6G9evX4+9//3ur/URGRiIuLg6rVq2CVCqFr68vdu/ejby8PHz44Yea85YsWYLTp0/jypUrmrZ58+YhJycH8+bNw7lz53Du3DnNMV9fX00VV0tLSyxcuBArVqzAokWLEBMTg7NnzyIpKQmLFy+Gvb19Wx+f9CBvkKvy0RsXj6oKGjW/eNTRwgHu1hIMcI9S5aU3prw0t3j0VlEVjpzJwW/pt/FzSj4iAlR58r39W8+TJyIiIupO2hzA3759G35+fpqvjx8/Dm9vbyxevBgA8Pvvv2Pfvn169bVy5UqsXr0ae/fuhUwmQ3BwMNavX49+/fq1el1mZiYA4Msvv9Q59swzz2gCeEBVtMnMzAwbN25EcnIyPDw8sHz5csyePVuvMVLz1ItHtQoatbp41BXhLqFaFUhdrVxg3obFo16uNnhubAgmDQvAiQu3cOz8LXz8fQq8JTYYPcAHg3u7w8xUd8cYIiIiou5EJAhCm/ZEjIqKwl//+ldMnToVADB69GgMGTIEK1asAAAkJibi/fffx8WLF9t/tEbgYdpGUl159E6aiypIlzZWHm0QGjTnqhePShqrjrpbuTbu9OIKK1OrDhmfXKHEqcsFOHzmJnKlVbC3NsOIvt54oq8X7K07Zz95Mj7cGo9IP3xXiPTTLbaR7NGjBy5cuICpU6fi99////buPDzq8t7//3MmmUyWyWSd7HsCCUs2ICKiiCKWIq6VY6tCta21Vduqp72s7ffb3++cnlbPEaseT+mCnONyqCgWpFKrbAoiVvYEZM0CZCUbJJCQdeb7x4SRmGAnkDCT5PW4Lq/L3PP5zOc96E3eufN+3/cRysvL+eEPf+h6vaGhod+GUvFODoeDls5Was/WcaK1vld9el1rPR29Th71xRYQSWxQDLm2ia6VdE81j5p8jVydE8v07BgOHDvJ2u3lvL2ljDWfHOOqidHMnpJIvM1zB2+JiIiIDIUBJ/A33XQTixcvprGxkSNHjmCxWLj22mtdrx84cOAfNrDK5dfW1fZ502hrHbWtn2/J2Kd51D+cqMBIMsMysPXs7hIdaCPEbPXKk0cNBgPjU8IZnxJOdUML63ZUsHVvNZsLq5mYGs6NBYlMSA1XnbyIiIiMCANO4B988EGqq6vZsGEDFouFf//3f3c1g54+fZqNGze6dpWRS7etZhd/KXmPU+2nCDWHckv6HK6ImdTvteeaR3vVpff8e3NH71/9hJlDiQqMZEp03nl16ZFE+If3e/LocBEbEcTCr2Ry+zWpfLinio07K/jNm4XERzrr5KdNUJ28iIiIDG8DroH/Mna7nZaWFvz9/TGZRubJlpezBn5bzS7+dPDPdPYqYzFxc+pXiLFEfX6gUWsddWfrafxi86jJQlSgsxY9OsDm+ndbQOSAmkeHs84uO9sOnGDd9nKO154hONDEdfnxXDcpgZAg1cmPRKrrFXGP5oqIe7yxBn5QE/iOjg78/EZ2UnQ5E/j/8/GvOdl+6kuv8ffxd66gn7e7S9QQN48ORw6Hg4PHT7F223EKSxrw9TFw5YQYbixIJEF18iOKkhIR92iuiLjHGxP4AZfQbNq0iaKiIn7wgx+4xpYtW8azzz5LW1sbX/3qV3n66adH7Ar85fRlyftjk75PdKBNJ4+6yWAwMC45jHHJYVQ3tLB+RwUf761mS1E1E1LCmF2QxMS0cIz6sxQREREvN+AEfunSpURERLi+Likp4de//jWJiYkkJCTw7rvvkp2drTr4QRBmDu03iQ8zh5IRmuqBiEaG2IggFnwlk9tnpLFpTyUbdlbw/IpCYiMCubEgkWkTYvAzqU5eREREvNOAtxQpLS1l4sSJrq/fffddzGYzb731Fi+99BJz587l7bffHtQgR6tb0udgMvb+TYbJaOKW9DkeimhksQSYuGlaCv/x/at4YN54/Hx9eOW9Q/x48VZWbS6l6Uy7p0MUERER6WPAK/BNTU2EhYW5vt66dStXXnklFouzTueKK65g06ZNgxfhKHZutxl3d6GRi+PrY2TaxBiunBDN4fJTrN1ezpqtR/nbp8eYOj6aGwuSSIxSnbyIiIh4hwEn8GFhYVRVVQFw5swZ9u7dy+OPP+56vauri+7u7gvdLgN0RcwkroiZpGajy8BgMJCZFEZmUhgnGltZv6OCj/ZW8fHeGsYlh3FjQSLZ6RGqkxcRERGPGnACn5eXx/Lly8nIyGDz5s10d3czY8YM1+vHjh0jKipqUIMUudyiwwO558ax3DYjlU17qtiws4IX3ioiJjyQ2QWJXDUxBrPq5EVERMQDBpzA//CHP2ThwoU8+uijANx+++1kZGQAzq361q9fz9SpUwc3ShEPCfI3MffKZG4sSGTHwVre317Oa+8fYuWmEq6bFM/1kxIItZg9HaaIiIiMIhe1D/ypU6fYtWsXwcHBFBQUuMabmpp4++23mTp1KllZWYMaqLe4nPvAn08lNN7B4XBwpKKJ97cdZ8+ReoxGQ0+dfCJJ0cGeDk/QXBFxl+aKiHu8cR/4QT3IaTRQAi/n1J5sZd2OCrYUVdPe2U1WUig3FiSRk6E6eU/SXBFxj+aKiHtGVAJ//PhxNmzYQHl5OQCJiYnMmjWLpKSki4t0mFACL1/U2tbJpsIq1u+o4OTpdqLDA7lxSgJXTYzF7Kc6+ctNc0XEPZorIu4ZMQn8888/z5IlS/rsNmM0GnnwwQf50Y9+NPBIhwkl8HIhXd12dh6qY+3245RVnybI35dr8+KZNTmBsGDVyV8umisi7tFcEXGPNybwA25ifeutt/j9739Pfn4+3/nOdxgzZgwAR44cYenSpfz+978nMTGRO+644+KjFhmGfH2MTB0fzRXjoiiubGLt9nL+9ukx3t92nIJxUXylIInkGNXJi4iIyKUZ8Ar8HXfcgclkYtmyZfj69s7/u7q6uOeee+js7GTlypWDGqi30Aq8DETdqbOs31HB5qIq2ju6yUwM5caCRHIzIjEaVSc/FDRXRNyjuSLiHm9cgTcO9A1LSkqYO3dun+QdwNfXl7lz51JSUjLQtxUZkWyhAXzjhjE8+9B0/um6DOqbzvLiyr38bMnf2bCzgraOLk+HKCIiIsPMgEtoTCYTra2tF3y9paUFk8l0SUGJjDSB/r7MmZrE7IKEnjr5cpatO8yqzaVcmx/HrEkJhFv9PR2miIiIDAMDTuCzs7N54403mD9/PpGRkb1ea2ho4M033yQ3N3fQAhQZSXyMRq4YF80V46KddfLbjvPep8dZu62cgqwoZhckkhpr9XSYIiIi4sUGXAO/fft27rvvPoKCgvja177mOoW1uLiYlStX0tLSwssvv8yUKVOGJGBPUw28DLb6U2dZv7OCzYVVtHV0MzYhhNkFSeSPUZ38xdBcEXGP5oqIe7yxBv6itpHcuHEjv/zlL6muru41HhcXxy9+8Qtmzpw54ECHCyXwMlTOtnfxUWEV63dWUN/Uhi3Un9lTErk6JxZ/vwH/smzU0lwRcY/mioh7RkwCD2C329m3bx8VFRWA8yCnCRMm8Oabb/Lqq6/y7rvvXlzEXk4JvAy1brud3YfrWbu9nOLKJgLMvlybG8esyQlEhKhO/h/RXBFxj+aKiHu8MYG/6GU9o9FITk4OOTk5vcZPnjxJWVnZxb6tyKjnYzQyJSuKKVlRlFQ1sW57OWt7/pmSZePGgiTS4lQnLyIiMlrp9/IiXiw9LoT0W0NomNnGhp0VbCqsZNuBWjISQrhxSiKTxtpUJy8iIjLKKIEXGQYiQvz5p+szuHl6CluKqlm3o5zFb+8jMsSfG6Ykck1OLAFmTWcREZHRQN/xRYaRALMvswsSmTU5gd1HnPvJL99whNVbSpnRUycfGRLg6TBFRERkCCmBFxmGjEYDkzOjmJwZRWlVM2u3H2fd9grWba9gcqaNGwsSSY8P8XSYIiIiMgTcSuD/53/+x+033LVr10UHIyIDlxZn5Xu3TqTxujbW76xg054qth+sJT3eyo0FSUwaG4mP0ejpMEVERGSQuLWNZFZW1sDe1GDgwIEDFx2UN9M2kuLt2jq6+HhvDeu2l1N76iwRVn9umJLANTlxBPqP/F+6aa6IuEdzRcQ9w3YbyVdffXXQAhKRoeXv58usyQlclx/PnmLnfvJvbCxm9ZYyrsmJ44YpCdhCVScvIiIyXF30QU6jlVbgZTg6WtPM2u3lbD9Qi93hYNJYG18pSCI93orBMLK2odRcEXGP5oqIe4btCryIDG8pMVa+e/ME7rw2nY27KvlwdyU7D9WRFmflxoJEJmfaVCcvIiIyTCiBFxlFwq3+3DkznXlXJTvr5HeU8/vVnxFhNTNrciIzcmMJ9Dd5OkwRERH5EiqhGSCV0MhIYnc4KCyuZ932cg4eP4XZz4drcmK5YUoiUcO0Tl5zRcQ9misi7lEJjYh4FaPBQP4YG/ljbByrOc3a7eV8sKuSDTsrmDTGxuyCRMYkhIy4OnkREZHhTAm8iACQHBPMAzeP586Z6WzcVeGskz9cR2psMLMLEpmSGYWvj+rkRUREPE0lNAOkEhoZLdo7u9m6r4a128s50dhKWLCZGyYnMCMvjiAvrpPXXBFxj+aKiHtUQiMiw4bZ5MN1+fFcmxdHUUkD67aXs+LDEv7y8VGuzo7lhoIEosMCPR2miIjIqKMEXkS+lNFgIC8jkryMSI6fOM267eV8uKeSjbsqyBsTyY0FiYxNDFWdvIiIyGWiBF5E3JYUHcy3543nazM/309+95F6kmOCubEgkYIs1cmLiIgMNdXAD5Bq4EU+197ZzSf7nPvJVzc46+SvnxTPtXnxWAI8UyevuSLiHs0VEfeoBl5ERhSzyYeZ+fHMyItjX2kDa7eX8+dNpbyz1VknP3tKItHhqpMXEREZTB5N4Ds6OnjhhRdYvXo1zc3NZGVl8dhjjzFt2rQvva+oqIiVK1dSVFTE4cOH6ezs5NChQ32uq6ioYNasWf2+x5IlS5gxY8agfA6R0c5oMJCTHklOeiTltWdYt72czYVVfLCrktwMZ518ZpLq5EVERAaDRxP4n/70p6xdu5aFCxeSnJzMqlWreOCBB3jttdfIz8+/4H2bNm1ixYoVZGZmkpiYSGlp6Zc+55ZbbuHqq6/uNZaVlTUon0FEekuMsvCtm8bxtWvT+GB3JRt3VbKnuJ6kaAs3FiRyxbho1cmLiIhcAo/VwBcVFTF//nyefPJJ7rvvPgDa29uZN28eUVFRLFu27IL31tfXY7FY8Pf351e/+hWvvvrql67An/+MS6UaeJGB6ejs5u/7T/D+tuNUN7QSYvFj1qQEZuYPTZ285oqIezRXRNyjGvjzvPfee5hMJubPn+8aM5vN3HnnnTz33HPU1tYSFRXV772RkZEDfl5rayu+vr74+flddMwiMnB+Jh9m5MZxTU4s+8oaWbu9nJWbS1mz9ShXZccye0oCsRFBng5TRERk2PBYAn/gwAFSU1MJCur9jTsnJweHw8GBAwcumMAP1AsvvMBTTz2FwWAgNzeXH//4xxQUFAzKe4uIewwGA9lpEWSnRVBR56yT31JUzYe7K8lJj+ArBYlkJYepTl5EROQf8FgCX1dXR3R0dJ9xm80GQG1t7SU/w2g0cvXVVzN79myioqI4duwYS5cu5f777+fll19mypQpl/wMERm4BJuF++eO42vXpvfUyVfwzPI9JEZ9Xidv8lWdvIiISH88lsC3tbVhMvWtfzWbzYCzHv5SxcXFsXTp0l5jc+fO5aabbmLRokUsX758wO/5ZfVIQ81mC/bYs0WGgs0G6SkRLJw3gQ93VbB6cwlL/3qAlZtLuWl6KnOmpRBiMV/E+2quiLhDc0XEPd42VzyWwPv7+9PZ2dln/Fzifi6RH2zR0dHcdNNNvPnmm5w9e5aAgIAB3a8mVpGhkZ8WTl5qGPuPnuT97cf53/cO8sb6w0yfGMPsgkS36+Q1V0Tco7ki4h41sZ7HZrP1WyZTV1cHMGj17/2JjY3FbrfT3Nw84AReRIaOwWBgQmo4E1LDqaxvcdbJ763hwz1V5KRHMLsgkfGqkxcRkVHOY0WmWVlZlJWV0dLS0mu8sLDQ9fpQKS8vx8fHh5CQkCF7hohcmvjIIO77ahaLHr6K265J5Wh1M88u38P/99/b+Kiois4uu6dDFBER8QiPJfBz5syhs7OTFStWuMY6OjpYuXIlkyZNcjW4VlVVUVJSclHPaGxs7DN27Ngx/vrXvzJlyhT8/f0vLngRuWysgX7cMj2VZx6azrfmjgPgf949yE9+t5W/bCmjubUDgE8+q+Eniz/mln9ezU8Wf8wnn9V4MmwREZEh47ESmtzcXObMmcOiRYuoq6sjKSmJVatWUVVVxVNPPeW67oknnmDbtm29DmqqrKxk9erVAOzduxeAxYsXA86V++uvvx6AZ555hvLycq688kqioqI4fvy4q3H1iSeeuCyfU0QGh8nXyNU5sUzPjmH/sZOs217O21vKWPPJMdLjgimtOk1nt3NVvqG5nVf+dhCAaRNiPBm2iIjIoPNYAg/wH//xHzz//POsXr2apqYmMjMz+eMf/8jkyZO/9L6KigpeeOGFXmPnvr799ttdCfz06dNZvnw5//u//8vp06exWq1Mnz6dRx55hDFjxgzNhxKRIWUwGJiQEs6ElHCq6ltYv6OcD/dU9bmuo8vOyk0lSuBFRGTEMTgcjsu/pcowpl1oRLzPt57eeMHX/u07U4mNCFTjq8gX6PuKiHu0C42IyBCIsJppaO7/7Ij/89Kn2EL9yUmPJDcjgszEMB0SJSIiw5oSeBEZ9u64Np1X/naQjvN2pvHzNfK1memYfIwUFtfzUWEVG3ZWYDb5MD4ljNyMSLLTIggLHpozJ0RERIaKEngRGfbO1bmv3FRCY3M74VYzd1yb7hqfmR9PR2c3B4+fpLC4gaKSenYfqQcgOTqY3IwIctIjSYkNxqhSGxER8XKqgR8g1cCLeDd35orD4aCyroXCknoKSxooqWzC4QBrkB/ZaeHkpkcyITWcALPWOGTk0vcVEfeoBl5ExAsYDAYSoiwkRFm4aVoKZ852sre0gaKSBnYfrufjvTX4GA2MTQwlNyOS3PQIosMDPR22iIgIoBX4AdMKvIh3u9S50m23U1LZTGFxPUUlDVTWO0+Ljg4PJDc9gtz0CMYkhuLro0ZYGd70fUXEPd64Aq8EfoCUwIt4t8GeK/WnzlJY0kBhST0Hj52iq9uOv58PE1PDyUmPJCc9AmuQ36A9T+Ry0fcVEfd4YwKvEhoRkS8RGRrArMkJzJqcQHtHN/uPNboaYXccqsMApMRayc2IIDc9kqRoi/acFxGRIaUEXkTETWY/H/LH2MgfY8PhcHD8xBmKehphV39UxtsflRFq8XPuOZ8ewfiUcMx+Pp4OW0RERhgl8CIiF8FgMJAcE0xyTDA3T0+luaWDvaUNFJY0sP3gCTYXVuHrYyQrydkIm5MegS00wNNhi4jICKAa+AFSDbyId/OGudLVbedI+ame2vkGTjS2AhAXGUROTyNsRkIIPkY1worneMNcERkOvLEGXgn8ACmBF/Fu3jhXTjS2UljirJs/dPwU3XYHgWZfJvbsOZ+dHoElwOTpMGWU8ca5IuKNvDGBVwmNiMgQiw4P5MbwQG4sSORsexf7j/Y0wpY2sO1ALQYDpMeFuE6ETbAFqRFWREQuSAm8iMhlFGD2ZXJmFJMzo7A7HByrOU1hsbMR9s+bSvnzplLCrWZye7aoHJcchp9JjbAiIvI5JfAiIh5iNBhIjbWSGmvltmvSOHWmnaKSBgqL69m6r4YPdlfi52skKznMdSJsuNXf02GLiIiHKYEXEfESoRYzM3LjmJEbR2eXnUPlJykqdh4iVVTSwGtAgs3i2nM+Lc6K0ahSGxGR0UZNrAOkJlYR7zYS54rD4aCmsdV1gNTh8ibsDgeWABPZac4TYSemhRPkr0ZYcd9InCsiQ0FNrCIiMmAGg4HYiCBiI4KYMzWJ1rZO9pU5G2H3ljbwyWcnMBoMjEkIIadndT42IlCNsCIiI5RW4AdIK/Ai3m20zRW73UFpdbPzRNjiBsprzwAQGeLvqpvPTArF5KtGWOlttM0VkYulFXgRERlURqOBjPgQMuJDuGNGOo3NbRSVNFBU0sBHhVVs2FmB2eTD+BRnI2x2WgRhwWZPhy0iIpdACbyIyAgSbvVnZn48M/Pj6ejs5uDxk85DpIrr2X2kHoDk6GDnibAZkaTEBmNUqY2IyLCiEpoBUgmNiHfTXOmfw+Ggsr6FwmLnjjbFlU04HGANNJGd7qybn5AaToBZ6zqjheaKiHtUQiMiIh5hMBhIsFlIsFm4aVoKZ852sq+0gcKSBnYfrufjvTX4GA2MTQwlt2d1Pjo80NNhi4hIP7QCP0BagRfxbporA9dtt1NS2exana+sbwEgOiyA3AznibBjE0Px9TF6OFIZTJorIu7xxhV4JfADpARexLtprly6+lNnKSxxHiB18Ngpurrt+Pv5MCE1nNz0SLLTIwgJ8vN0mHKJNFdE3OONCbxKaEREpJfI0ABmTU5g1uQE2ju62X+skaKSBgqL69l5qA6A1Firq9QmKdqiPedFRC4jJfAiInJBZj8f8sfYyB9jw+FwUF57xlVqs3pLGW9vKSPU4ufc1SY9knEpYfj76VuLiMhQ0t+yIiLiFoPBQFJ0MEnRwdw8PZXmlg729jTCbj9Yy+bCanx9DGQlhZGTHkFORiRRoQGeDltEZMRRDfwAqQZexLtprnhGV7edIxVNrtX5msZWAGIjAl0nwqbHh6gR1otoroi4xxtr4JXAD5ASeBHvprniHU6cbKWo2NkIe+j4KbrtDgLNvkxMczbCTkwLJzhQjbCepLki4h5vTOBVQiMiIoMuOiyQ2QWBzC5I5Gx7F/uPNjpPhC1pYNuBWgwGSI8LcZ0Im2ALUiOsiIiblMCLiMiQCjD7MjkzismZUdgdDo7VnKawuJ7CkgZWbi5l5eZSwq1mctKde86PSw7DbPLxdNgiIl5LCbyIiFw2RoOB1FgrqbFWbrsmjVNn2inqWZn/ZF8NH+6uxORrZFxyGLnpEeSkRxIR4u/psEVEvIoSeBER8ZhQi5kZuXHMyI2js8vO4fJTPavzzmZYOEyCLch1Imx6XAhGo0ptRGR0UxPrAKmJVcS7aa6MDA6Hg5rGVgqLGygqqedIRRPddgeWAFOvRtggf5OnQx22NFdE3KMmVhERETcYDAZiI4KIjQhiztQkWts62VfWSGFxA3tLG/j7ZycwGgxkJIQ4S20yIomLCFQjrIiMClqBHyCtwIt4N82Vkc9ud1Ba3UxRST2FxQ2U154BIDLEn9z0SHIyIshKCsXkq0bYL6O5IuIercCLiIhcIqPRQEZ8CBnxIdwxI53G5jZXI+xHRVVs2FWBn8nI+ORwcjOcjbBhwWZPhy0iMmiUwIuIyLAWbvVnZn48M/Pj6ejs5uDxU84m2OJ69hTXA4dIira4VudTY60YVWojIsOYSmgGSCU0It5Nc0XOcTgcVNa3UFTSQGFxPcWVTTgcYA00kZ3mPEBqQmo4AebRuZaluSLiHpXQiIiIXCYGg4EEm4UEm4W5VyZz5mwn+0obKCxpYE9xPR/vq8HHaGBsYqjrRNiY8EBPhy0i8g9pBX6AtAIv4t00V8Qd3XY7JZXNPaU2DVTWtwAQFRZAbnokuRkRjE0MxdfH6OFIh47mioh7tAIvIiLiBXyMRsYmhjI2MZT5MzOoP3WWwp5G2A92V7JuRzn+fj5MSA0np+dE2JAgP0+HLSICKIEXEREhMjSAWZMTmDU5gfaObg4cO0lhST2FxfXsPFQHQGpssKsRNik6WI2wIuIxSuBFRETOY/bzIW9MJHljInE4HJTXnqGwuJ6ikgZWbynj7S1lhFj8yOlphB2fEoa/n76disjl49G/cTo6OnjhhRdYvXo1zc3NZGVl8dhjjzFt2rQvva+oqIiVK1dSVFTE4cOH6ezs5NChQ/1ea7fbWbp0Ka+//jp1dXWkpKTw/e9/n7lz5w7FRxIRkRHEYDCQFB1MUnQwN09Ppbm1g70lzkbYHYdq+aioGl8fA5lJYa5G2KjQAE+HLSIjnEcT+J/+9KesXbuWhQsXkpyczKpVq3jggQd47bXXyM/Pv+B9mzZtYsWKFWRmZpKYmEhpaekFr33uuef44x//yF133cXEiRPZsGEDjz32GEajkTlz5gwx2B1PAAAdP0lEQVTFxxIRkRHKGujH9OxYpmfH0tVt50hFk+tE2NfXH+H19UeIjQh0ltqkR5CREDKiG2FFxDM8tgtNUVER8+fP58knn+S+++4DoL29nXnz5hEVFcWyZcsueG99fT0WiwV/f39+9atf8eqrr/a7An/ixAlmzZrFN77xDX7+858Dzn2B7733Xqqrq1m/fj1G48D+YtUuNCLeTXNFPOXEyVaKihsoKqnn4PFTdNsdBJh9yU5zNsJmp0UQHOg9jbCaKyLu0S4053nvvfcwmUzMnz/fNWY2m7nzzjt57rnnqK2tJSoqqt97IyMj3XrG+vXr6ezs5O6773aNGQwGvvGNb/DP//zPFBUVkZeXd2kfREREBIgOC2R2QSCzCxI5297F/qPORtiikga2HajFAKTFW12r84lRFgxqhBWRi+CxBP7AgQOkpqYSFBTUazwnJweHw8GBAwcumMAP5BkWi4XU1NQ+zwDYv3+/EngRERl0AWZfJmfamJxpw+5wcKzmtKsRduXmUlZuLiUs2ExuegQ5GZGMSw7DbPLxdNgiMkx4LIGvq6sjOjq6z7jNZgOgtrZ2UJ7R32r9YD5DRETkyxgNBlJjraTGWrntmjROnWl3NcJ+sv8EH+6pwuRrZFxyWM+e8xFEhqgRVkQuzGMJfFtbGyaTqc+42WwGnPXwg/EMP7++9YaX8owvq0caajZbsMeeLTKcaK6IN7PZghmTGskdN0BnVzf7ShrYceAE2/bX8L9rDwOQEmtlyrhoCsZHk5kcjo9xaEptNFdE3ONtc8VjCby/vz+dnZ19xs8l1eeS7Et9RkdHx6A+Q02sIt5Nc0WGm4TwABKmp3DrVcnUNLZS2NMIu+rDYt7aeIQgf1+ye1bmJ6ZGYAnou/h1MTRXRNyjJtbz2Gy2fktY6uqcJ95dav37uWfs2LFjSJ8hIiIyGAwGA7ERQcRGBDFnahKtbZ3sK2ukqKSBopIG/v7ZCYwGAxnxVnIznI2wcZFBaoQVGYU8lsBnZWXx2muv0dLS0quRtbCw0PX6pRo3bhwrVqygrKysVyPruWeMGzfukp8hIiIyFAL9TVwxLporxkVjtzsoq2527mpT3MCKD0tY8WEJkSH+rgOkspJCMfmqEVZkNPDY6RJz5syhs7OTFStWuMY6OjpYuXIlkyZNcjW4VlVVUVJSclHPmDVrFiaTiT/96U+uMYfDwfLly4mLiyM3N/fSPoSIiMhlYDQaSI8P4Y4Z6fz/37qCRQ9dxcI5mSTYLGwpqua5Nwv5wQsf8Z9vFfHhnkpOnr70PjIR8V4eW4HPzc1lzpw5LFq0iLq6OpKSkli1ahVVVVU89dRTruueeOIJtm3b1uugpsrKSlavXg3A3r17AVi8eDHgXLm//vrrAYiJiWHhwoX893//N+3t7WRnZ7N+/Xp27NjBc889N+BDnERERLxBuNWfmXnxzMyLp6Ozm4PHT7lW5/cU1wOHSIqykJMRSW5GBKmxVow9pTaffFbDyk0lNDa3E241c8e16UybEOPZDyQiA+Kxk1jB2Uz6/PPP884779DU1ERmZiaPP/44V111leuaBQsW9EngP/30UxYuXNjve95+++08/fTTrq/tdjtLlizhjTfeoLa2ltTUVB588EHmzZt3UTGriVXEu2muyGjmcDiorG+hqKSBwuJ6iiubcDggONBETloE/mZfNhdW0dlld93j52vkm1/NUhIvcgHe2MTq0QR+OFICL+LdNFdEPnfmbCf7Sp1NsHtLG2hp6+r3ugirmWcemn6ZoxMZHrwxgfdYCY2IiIgMLUuAiSsnxHDlhBi67XYe+I8P+72uobmdxW/vIzU2mLRYK0nRwQSYlSKIeCvNThERkVHAx2gkwmqmoblvg6ufr5Gj1c3sOOjc3tkAxEUGkdKT0KfEWkmMsuDro94xEW+gBF5ERGSUuOPadF7520E6LlAD39zawdHqZkqrmjlac5qikgY+3lsDgK+PgcSocwl9MGlxVqLDA13NsSJy+SiBFxERGSXONapeaBcaa6AfOemR5KRHAs6m2IamNspqTlNW1UxZdTNb9lazYVcFAAFmH5Kjg0mNs5IaYyUtzkpYsFmHS4kMMTWxDpCaWEW8m+aKiHsudq7Y7Q6qG1ooqz5NWbUzqS+vPUN3z/dGa5Df56v0PeU3lgDTYIcvctmoiVVERESGNaPRQLzNQrzNwtU5sQB0dnVTXtviSujLqpspLK7n3HJXVGgAKbHBpMZaSY21khwdjNlPp8aKXCwl8CIiInJJTL4+pMU5S2jOaW3r4lhNs7P8prqZ4somth3oaZI1QHykhdTYz8tv4m1BapIVcZMSeBERERl0gf6+jEsJZ1xKuGus6Ux7r9KbXYfr+KioGgCTr5GkaItrlT411kpUWICaZEX6oQReRERELosQi5m8MWbyxnzeJFt36myvpH5zYRXrdzibZAPNvr1Kb1JjnU2yIqOdEngRERHxCIPBQFRYIFFhgUwdHw1At91OVX1rr3r6v/39OPaePTdCLX6fJ/RxVlJjggn0V5OsjC5K4EVERMRr+BiNJEZZSIyyMCM3DoCOzm6O155xbmVZ00xZVTO7j9S77okOD3TW0/ck9klRFvxMapKVkUsJvIiIiHg1P5MPGfEhZMSHuMZa2jo5et7+9AePneTvn50AwMdoIN4W1Kv0Ji4yEB+jmmRlZFACLyIiIsNOkL+JCSnhTDivSfbk6fZepTfbD9SyaU8VAH4mo/PQqfPKb2wh/jp0SoYlJfAiIiIyIoQFmwkLtjFprA0Au8NB7cmzvZL6D3ZXsnZ7OQCWAJOzSTbm86Q+JMjPkx9BxC1K4EVERGREMhoMxIQHEhMeyLQJMQB0dduprGtx1dKXVZ9mTdlRzp1LH2419yq9SYkJJsCsdEm8i/6PFBERkVHD18dIckwwyTHBzMyLB6C9o5tjJz7fyvJo9Wl2HqoDwADERAT2SuoToyyYfFVPL56jBF5ERERGNbOfD2MTQxmbGOoaO3O2k6PVzZT2JPT7yhrZuq8GcDbJJkadf+hUMLERQRiNqqeXy0MJvIiIiMgXWAJMTEyLYGJaBOA8dOrk6XZKz9vK8pPPavhgdyXg/CEgJTrYuTd9T1IfYVWTrAwNJfAiIiIi/4DBYCDc6k+41Z8pWVGAs0n2RGMrpVXOVfrS6mbW7yinq9tZUB8caOpVepMaG0xwoJpk5dIpgRcRERG5CEaDgdiIIGIjgpieHQs4m2TLa8/0Kr/ZW9JAT48skSH+vRL65Jhg/P2UjsnA6P8YERERkUHi62N0JejX9Yydbe/i+InTlPWs0pdVN7P9YC0ABgPERQY5t7KMcyb1CTYLvj5qkpULUwIvIiIiMoQCzL5kJoWRmRTmGmtu6eBoTbOz/KbmNHuK69mytxpw/hCQFG3pSeqdh09FhwdiVD299FACLyIiInKZWYP8yEmPJCc9EnA2yTY0tbnKbkqrm9myt5oNuyoACDD7kBLzeelNaqyVsGCzmmRHKSXwIiIiIh5mMBiIDA0gMjSAK8ZFA2C3O6huaOmV1L+/7TjddmdFfUiQX6+EPiXWiiXA5MmPIZeJEngRERERL2Q0Goi3WYi3WbgmxznW2dXN8dozzoS+qpmjNc3sKa533RMVGuCspY9xbmmZFB2M2eTjoU8gQ0UJvIiIiMgwYfL1IT0uhPS4EGZNdo61tnVxrKaZsprTlFU1c6TiFJ/uPwE4d8qJtwW5VulTY63E24LwMapJdjhTAi8iIiIyjAX6+zIuJZxxKeGusaYz7a5db45WN7PzUB2bC51Nsn6+RpKigz8vv4mzEhUaoHr6YUQJvIiIiMgIE2IxkzfGTN6Yz5tk606dddXTl1U3s2lPJet22AEI8vclpafs5tyWlqEWsyc/gnwJJfAiIiIiI5zBYCAqLJCosECuHB8DQLfdTlV9K2U9e9OXVTXz7ifHsTucTbJhwebeTbIxVgL9lTp6A/1XEBERERmFfIxGEqMsJEZZmJEbB0B7ZzflJ858ntRXN7PrcJ3rnpjwQFJjg0mJtZIWayUp2oLJV02yl5sSeBEREREBwGzyISMhhIyEENdYS1unq+ymrLqZ/cdO8slnziZZH6OBBJulV5NsXGQQRqPq6YeSEngRERERuaAgfxMTUsOZkPp5k+zJ0+29Vuk/PVDLh3uqAOcPAcnRFmc9fc/+9LYQfzXJDiIl8CIiIiIyIGHBZsKCbUwaawPA7nBQe/Ksq5a+rKaZDTsr6eouB8ASYCIlNpi0noQ+NdZKSJCfJz/CsKYEXkREREQuidFgICY8kJjwQKZNcDbJdnXbqaxr6bVS/07ZUXp6ZImwml1lN6mxVpJjggkwKzV1h/6URERERGTQ+foYSY4JJjkmmJn58QC0d3Rz7MTpXkn9jkPOJlkDEBsZ5DpFNjXWSoLNgslXh059kRJ4EREREbkszH4+jE0MZWxiqGvsdGsHR2tOu8pv9pY18vG+GgB8fQwkRllcu96kxFqJjQjEOMrr6ZXAi4iIiIjHBAf6kZ0WQXZaBOA8dKqxuXeT7Cf7avhgVyUA/n4+zkOnziu/CbeaR1WTrBJ4EREREfEaBoOBiBB/IkL8mZIVBTibZGsazjt0qvo063aU09XtLKi3Bpp6rdKnxgYTHDhym2SVwIuIiIiIVzMaDMRFBhEXGcT07FjA2SRbXnuGo9XNlFY3c7T6NHtLGujpkSUyxJ+0OOcJsmlxVpKjgzH7jYxDp5TAi4iIiMiw4+tjdJXQXNczdra9i+MnTlPas0pfUtnMtgO1ABgMEB8Z5NrGMi3WSrwtCF+f/ptkP/mshpWbSmhsbifcauaOa9NdO+x4mhJ4ERERERkRAsy+ZCaFkZkU5hprbulwld4crTnNniP1bCmqBnp2yok+v0k2mOjwQD7df4JX/naQji47AA3N7bzyt4MAXpHEK4EXERERkRHLGuRHbkYkuRmRgLNJtqGpzVV2U1rdzJaiajbsrACcPwR0dtnp6rb3ep+OLjsrN5UogRcRERERuZwMBgORoQFEhgZwxbhoAOx2B1UNLa4G2Q93V/Z7b0Nz++UM9YKUwIuIiIjIqGY0GkiwWUiwWbgmB/aW1PebrEdYzR6Iri8dbSUiIiIicp47rk3H7wsnwPr5Grnj2nQPRdSbRxP4jo4OnnnmGa6++mpycnL4p3/6Jz755BO37j1x4gQ/+tGPmDJlCpMmTeKhhx6ivLy8z3WZmZn9/vP6668P9scRERERkRFg2oQYvvnVLCKsZgw4V96/+dUsr6h/BzA4HA7HP75saDz++OOsXbuWhQsXkpyczKpVq9i3bx+vvfYa+fn5F7yvpaWFO+64g5aWFu677z58fX15+eWXMRgMvP3224SEhLiuzczM5Oqrr+aWW27p9R65ubmkpKQMOOaGhjPY7Zf/j8xmC6au7vRlf67IcKO5IuIezRUR93hirhiNBiIiLBd83WM18EVFRfz1r3/lySef5L777gPgtttuY968eSxatIhly5Zd8N4//elPHDt2jJUrVzJ+/HgArrnmGm6++WZefvllfvSjH/W6Pi0tjVtvvXXIPouIiIiIyOXisRKa9957D5PJxPz5811jZrOZO++8k507d1JbW3vBe99//33y8vJcyTtAeno606ZN429/+1u/97S1tdHe7h2dwyIiIiIiF8tjCfyBAwdITU0lKCio13hOTg4Oh4MDBw70e5/dbufQoUNMnDixz2vZ2dkcPXqUs2fP9hp/6623yMvLIycnh5tvvpl169YN3gcREREREbmMPJbA19XVERUV1WfcZrMBXHAF/tSpU3R0dLiu++K9DoeDuro611h+fj6PPfYYixcv5he/+AUdHR088sgjrFmzZpA+iYiIiIjI5eOxGvi2tjZMJlOfcbPZub/mhcpdzo37+fld8N62tjbX2PLly3tdc/vttzNv3jyeeeYZbrrpJgwGw4Di/rKGgqFmswV77Nkiw4nmioh7NFdE3ONtc8VjCby/vz+dnZ19xs8l6OeS8S86N97R0XHBe/39/S/43MDAQL7+9a/z7LPPUlpaSnr6wPbz1C40It5Nc0XEPZorIu7xxl1oPFZCY7PZ+i2TOVf+0l95DUBoaCh+fn69ymTOv9dgMPRbXnO+2NhYAJqamgYatoiIiIiIR3ksgc/KyqKsrIyWlpZe44WFha7X+2M0Ghk7diz79u3r81pRURHJyckEBAR86bPPHfgUHh5+MaGLiIiIiHiMxxL4OXPm0NnZyYoVK1xjHR0drFy5kkmTJhEdHQ1AVVUVJSUlve79yle+wp49e9i/f79rrLS0lL///e/MmTPHNdbY2NjnuSdPnuRPf/oTCQkJF3WQk4iIiIiIJ3msBj43N5c5c+awaNEi6urqSEpKYtWqVVRVVfHUU0+5rnviiSfYtm0bhw4dco3dfffdrFixgu9+97vcf//9+Pj48PLLL2Oz2VyHQgEsW7aMDRs2MHPmTOLi4jhx4gRvvPEGjY2N/Pa3v72ouI3GgTW9DiZPPltkONFcEXGP5oqIey73XPlHzzM4HI7L35HZo729neeff5533nmHpqYmMjMzefzxx7nqqqtc1yxYsKBPAg9QU1PDr3/9az7++GPsdjtTp07l5z//OYmJia5rtmzZwtKlSzl8+DBNTU0EBgaSl5fHgw8+yOTJky/b5xQRERERGSweTeBFRERERGRgPFYDLyIiIiIiA6cEXkRERERkGFECLyIiIiIyjCiBFxEREREZRpTAi4iIiIgMI0rgRURERESGESXwIiIiIiLDiBJ4EREREZFhRAm8iIiIiMgw4uvpAKR/tbW1vPrqqxQWFrJv3z5aW1t59dVXmTp1qqdDE/EqRUVFrFq1ik8//ZSqqipCQ0PJz8/n0UcfJTk52dPhiXiNvXv38vvf/579+/fT0NBAcHAwWVlZPPzww0yaNMnT4Yl4rSVLlrBo0SKysrJYvXq1p8MBlMB7rbKyMpYsWUJycjKZmZns3r3b0yGJeKWXXnqJXbt2MWfOHDIzM6mrq2PZsmXcdtttvPXWW6Snp3s6RBGvUF5eTnd3N/Pnz8dms3H69Gneeecd7r33XpYsWcL06dM9HaKI16mrq+N3v/sdgYGBng6lF4PD4XB4Ogjp68yZM3R2dhIWFsb69et5+OGHtQIv0o9du3YxceJE/Pz8XGNHjx7l5ptv5qabbuLpp5/2YHQi3u3s2bPccMMNTJw4kT/84Q+eDkfE6/z0pz+lqqoKh8NBc3Oz16zAqwbeS1ksFsLCwjwdhojXmzRpUq/kHSAlJYUxY8ZQUlLioahEhoeAgADCw8Npbm72dCgiXqeoqIi//OUvPPnkk54OpQ8l8CIy4jgcDurr6/VDsEg/zpw5Q2NjI6WlpfzmN7/h8OHDTJs2zdNhiXgVh8PBL3/5S2677TbGjRvn6XD6UA28iIw4f/nLXzhx4gSPPfaYp0MR8To/+9nPeP/99wEwmUx8/etf53vf+56HoxLxLm+//TbFxcX89re/9XQo/VICLyIjSklJCf/6r//K5MmTufXWWz0djojXefjhh7nrrruoqalh9erVdHR00NnZ2acUTWS0OnPmDM8++yzf/e53iYqK8nQ4/VIJjYiMGHV1dTz44IOEhITwwgsvYDTqrziRL8rMzGT69Ol87WtfY+nSpXz22WdeWeMr4im/+93vMJlM3H///Z4O5YL03U1ERoTTp0/zwAMPcPr0aV566SVsNpunQxLxeiaTiVmzZrF27Vra2to8HY6Ix9XW1vLKK69w9913U19fT0VFBRUVFbS3t9PZ2UlFRQVNTU2eDlMlNCIy/LW3t/O9732Po0eP8vLLL5OWlubpkESGjba2NhwOBy0tLfj7+3s6HBGPamhooLOzk0WLFrFo0aI+r8+aNYsHHniAH//4xx6I7nNK4EVkWOvu7ubRRx9lz549LF68mLy8PE+HJOKVGhsbCQ8P7zV25swZ3n//fWJjY4mIiPBQZCLeIyEhod/G1eeff57W1lZ+9rOfkZKScvkD+wIl8F5s8eLFAK69rFevXs3OnTuxWq3ce++9ngxNxGs8/fTTbNy4keuuu45Tp071OmQjKCiIG264wYPRiXiPRx99FLPZTH5+PjabjerqalauXElNTQ2/+c1vPB2eiFcIDg7u9/vGK6+8go+Pj9d8T9FJrF4sMzOz3/H4+Hg2btx4maMR8U4LFixg27Zt/b6muSLyubfeeovVq1dTXFxMc3MzwcHB5OXl8a1vfYsrrrjC0+GJeLUFCxZ41UmsSuBFRERERIYR7UIjIiIiIjKMKIEXERERERlGlMCLiIiIiAwjSuBFRERERIYRJfAiIiIiIsOIEngRERERkWFECbyIiIiIyDCiBF5ERLzeggULuP766z0dhoiIV/D1dAAiIuIZn376KQsXLrzg6z4+Puzfv/8yRiQiIu5QAi8iMsrNmzePGTNm9Bk3GvVLWhERb6QEXkRklBs/fjy33nqrp8MQERE3aXlFRES+VEVFBZmZmbz44ousWbOGm2++mezsbGbOnMmLL75IV1dXn3sOHjzIww8/zNSpU8nOzmbu3LksWbKE7u7uPtfW1dXxb//2b8yaNYuJEycybdo07r//fj7++OM+1544cYLHH3+cgoICcnNz+fa3v01ZWdmQfG4REW+lFXgRkVHu7NmzNDY29hn38/PDYrG4vt64cSPl5eXcc889REZGsnHjRv7rv/6LqqoqnnrqKdd1e/fuZcGCBfj6+rqu/eCDD1i0aBEHDx7k2WefdV1bUVHBN77xDRoaGrj11luZOHEiZ8+epbCwkK1btzJ9+nTXta2trdx7773k5uby2GOPUVFRwauvvspDDz3EmjVr8PHxGaI/IRER76IEXkRklHvxxRd58cUX+4zPnDmTP/zhD66vDx48yFtvvcWECRMAuPfee3nkkUdYuXIld911F3l5eQD86le/oqOjg+XLl5OVleW69tFHH2XNmjXceeedTJs2DYB/+Zd/oba2lpdeeolrrrmm1/Ptdnuvr0+ePMm3v/1tHnjgAddYeHg4zzzzDFu3bu1zv4jISKUEXkRklLvrrruYM2dOn/Hw8PBeX1911VWu5B3AYDDwne98h/Xr17Nu3Try8vJoaGhg9+7dzJ4925W8n7v2+9//Pu+99x7r1q1j2rRpnDp1io8++ohrrrmm3+T7i020RqOxz645V155JQDHjh1TAi8io4YSeBGRUS45OZmrrrrqH16Xnp7eZywjIwOA8vJywFkSc/74+dLS0jAaja5rjx8/jsPhYPz48W7FGRUVhdls7jUWGhoKwKlTp9x6DxGRkUBNrCIiMix8WY27w+G4jJGIiHiWEngREXFLSUlJn7Hi4mIAEhMTAUhISOg1fr7S0lLsdrvr2qSkJAwGAwcOHBiqkEVERiQl8CIi4patW7fy2Wefub52OBy89NJLANxwww0AREREkJ+fzwcffMDhw4d7XfvHP/4RgNmzZwPO8pcZM2awefNmtm7d2ud5WlUXEemfauBFREa5/fv3s3r16n5fO5eYA2RlZfHNb36Te+65B5vNxoYNG9i6dSu33nor+fn5rut+/vOfs2DBAu655x7uvvtubDYbH3zwAVu2bGHevHmuHWgA/u///b/s37+fBx54gNtuu40JEybQ3t5OYWEh8fHx/OQnPxm6Dy4iMkwpgRcRGeXWrFnDmjVr+n1t7dq1rtrz66+/ntTUVP7whz9QVlZGREQEDz30EA899FCve7Kzs1m+fDn/+Z//yeuvv05rayuJiYn8+Mc/5lvf+lavaxMTE/nzn//Mb3/7WzZv3szq1auxWq1kZWVx1113Dc0HFhEZ5gwO/Y5SRES+REVFBbNmzeKRRx7hBz/4gafDEREZ9VQDLyIiIiIyjCiBFxEREREZRpTAi4iIiIgMI6qBFxEREREZRrQCLyIiIiIyjCiBFxEREREZRpTAi4iIiIgMI0rgRURERESGESXwIiIiIiLDiBJ4EREREZFh5P8BlAPbXBqai5MAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 864x432 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mkyubuJSOzg3"
      },
      "source": [
        "# 5. Performance On Test Set"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DosV94BYIYxg"
      },
      "source": [
        "Now we'll load the holdout dataset and prepare inputs just as we did with the training set. Then we'll evaluate predictions using [Matthew's correlation coefficient](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.matthews_corrcoef.html) because this is the metric used by the wider NLP community to evaluate performance on CoLA. With this metric, +1 is the best score, and -1 is the worst score. This way, we can see how well we perform against the state of the art models for this specific task."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tg42jJqqM68F"
      },
      "source": [
        "### 5.1. Data Preparation\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xWe0_JW21MyV"
      },
      "source": [
        "\n",
        "We'll need to apply all of the same steps that we did for the training data to prepare our test data set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 402
        },
        "id": "PVbtWms3pC9s",
        "outputId": "e1c6357f-1123-446d-9383-db09629f4714"
      },
      "source": [
        "df = pd.read_csv(dev_df_path)\r\n",
        "df"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Thats it.... like....I dont like that statue</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>@Generation X Counting money that she been giv...</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>@Paola Hernandez i never said to be intolerant...</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>@Firstlast300 Wow An opinion is that I don't l...</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>WOW!!!!!!!That was so so inspiring and incredi...</td>\n",
              "      <td>Hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2838</th>\n",
              "      <td>Such fake sentiment. .</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2839</th>\n",
              "      <td>@A G black lives arent undervalued compared to...</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2840</th>\n",
              "      <td>People who pulled it down can and will be arre...</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2841</th>\n",
              "      <td>@Aaron Castellanos It will be a two hour movie...</td>\n",
              "      <td>Non_hope_speech</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2842</th>\n",
              "      <td>why is there no footage of the riots and the v...</td>\n",
              "      <td>Hope_speech</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>2843 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                   text            label\n",
              "0          Thats it.... like....I dont like that statue  Non_hope_speech\n",
              "1     @Generation X Counting money that she been giv...  Non_hope_speech\n",
              "2     @Paola Hernandez i never said to be intolerant...  Non_hope_speech\n",
              "3     @Firstlast300 Wow An opinion is that I don't l...  Non_hope_speech\n",
              "4     WOW!!!!!!!That was so so inspiring and incredi...      Hope_speech\n",
              "...                                                 ...              ...\n",
              "2838                             Such fake sentiment. .  Non_hope_speech\n",
              "2839  @A G black lives arent undervalued compared to...  Non_hope_speech\n",
              "2840  People who pulled it down can and will be arre...  Non_hope_speech\n",
              "2841  @Aaron Castellanos It will be a two hour movie...  Non_hope_speech\n",
              "2842  why is there no footage of the riots and the v...      Hope_speech\n",
              "\n",
              "[2843 rows x 2 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mAN0LZBOOPVh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2cb99119-50ce-48a3-ba61-a59816bae7b8"
      },
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Load the dataset into a pandas dataframe.\n",
        "df = pd.read_csv(dev_df_path)\n",
        "\n",
        "# Report the number of sentences.\n",
        "print('Number of test sentences: {:,}\\n'.format(df.shape[0]))\n",
        "\n",
        "# Create sentence and label lists\n",
        "sentences = df.text.values\n",
        "df['label'] = le.transform(df['label'])\n",
        "labels = df.label.values\n",
        "\n",
        "# Tokenize all of the sentences and map the tokens to thier word IDs.\n",
        "input_ids = []\n",
        "attention_masks = []\n",
        "\n",
        "# For every sentence...\n",
        "for sent in sentences:\n",
        "    # `encode_plus` will:\n",
        "    #   (1) Tokenize the sentence.\n",
        "    #   (2) Prepend the `[CLS]` token to the start.\n",
        "    #   (3) Append the `[SEP]` token to the end.\n",
        "    #   (4) Map tokens to their IDs.\n",
        "    #   (5) Pad or truncate the sentence to `max_length`\n",
        "    #   (6) Create attention masks for [PAD] tokens.\n",
        "    encoded_dict = tokenizer.encode_plus(\n",
        "                        sent,                      # Sentence to encode.\n",
        "                        add_special_tokens = True, # Add '[CLS]' and '[SEP]'\n",
        "                        max_length = 64,           # Pad & truncate all sentences.\n",
        "                        pad_to_max_length = True,\n",
        "                        return_attention_mask = True,   # Construct attn. masks.\n",
        "                        return_tensors = 'pt',     # Return pytorch tensors.\n",
        "                   )\n",
        "    \n",
        "    # Add the encoded sentence to the list.    \n",
        "    input_ids.append(encoded_dict['input_ids'])\n",
        "    \n",
        "    # And its attention mask (simply differentiates padding from non-padding).\n",
        "    attention_masks.append(encoded_dict['attention_mask'])\n",
        "\n",
        "# Convert the lists into tensors.\n",
        "input_ids = torch.cat(input_ids, dim=0)\n",
        "attention_masks = torch.cat(attention_masks, dim=0)\n",
        "labels = torch.tensor(labels)\n",
        "\n",
        "# Set the batch size.  \n",
        "batch_size = 32  \n",
        "\n",
        "# Create the DataLoader.\n",
        "prediction_data = TensorDataset(input_ids, attention_masks, labels)\n",
        "prediction_sampler = SequentialSampler(prediction_data)\n",
        "prediction_dataloader = DataLoader(prediction_data, sampler=prediction_sampler, batch_size=batch_size)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of test sentences: 2,843\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py:2179: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).\n",
            "  FutureWarning,\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "16lctEOyNFik"
      },
      "source": [
        "## 5.2. Evaluate on Test Set\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rhR99IISNMg9"
      },
      "source": [
        "\n",
        "With the test set prepared, we can apply our fine-tuned model to generate predictions on the test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hba10sXR7Xi6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fed781d0-564e-437e-ab9a-d40b9332f5a2"
      },
      "source": [
        "# Prediction on test set\n",
        "\n",
        "print('Predicting labels for {:,} test sentences...'.format(len(input_ids)))\n",
        "\n",
        "# Put model in evaluation mode\n",
        "model.eval()\n",
        "\n",
        "# Tracking variables \n",
        "predictions , true_labels = [], []\n",
        "\n",
        "# Predict \n",
        "for batch in prediction_dataloader:\n",
        "  # Add batch to GPU\n",
        "  batch = tuple(t.to(device) for t in batch)\n",
        "  \n",
        "  # Unpack the inputs from our dataloader\n",
        "  b_input_ids, b_input_mask, b_labels = batch\n",
        "  \n",
        "  # Telling the model not to compute or store gradients, saving memory and \n",
        "  # speeding up prediction\n",
        "  with torch.no_grad():\n",
        "      # Forward pass, calculate logit predictions\n",
        "      outputs = model(b_input_ids, token_type_ids=None, \n",
        "                      attention_mask=b_input_mask)\n",
        "\n",
        "  logits = outputs[0]\n",
        "\n",
        "  # Move logits and labels to CPU\n",
        "  logits = logits.detach().cpu().numpy()\n",
        "  label_ids = b_labels.to('cpu').numpy()\n",
        "  \n",
        "  # Store predictions and true labels\n",
        "  predictions.append(logits)\n",
        "  true_labels.append(label_ids)\n",
        "\n",
        "print('    DONE.')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Predicting labels for 2,843 test sentences...\n",
            "    DONE.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-5jscIM8R4Gv"
      },
      "source": [
        "Accuracy on the CoLA benchmark is measured using the \"[Matthews correlation coefficient](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.matthews_corrcoef.html)\" (MCC).\n",
        "\n",
        "We use MCC here because the classes are imbalanced:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hWcy0X1hirdx",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fe882729-91d2-40fc-a801-79758334baae"
      },
      "source": [
        "print('Positive samples: %d of %d (%.2f%%)' % (df.label.sum(), len(df.label), (df.label.sum() / len(df.label) * 100.0)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Positive samples: 2573 of 2843 (90.50%)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cRaZQ4XC7kLs",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6214457e-9ed8-43f4-d5d6-2878f5293cdf"
      },
      "source": [
        "from sklearn.metrics import matthews_corrcoef\n",
        "\n",
        "matthews_set = []\n",
        "\n",
        "# Evaluate each test batch using Matthew's correlation coefficient\n",
        "print('Calculating Matthews Corr. Coef. for each batch...')\n",
        "\n",
        "# For each input batch...\n",
        "for i in range(len(true_labels)):\n",
        "  \n",
        "  # The predictions for this batch are a 2-column ndarray (one column for \"0\" \n",
        "  # and one column for \"1\"). Pick the label with the highest value and turn this\n",
        "  # in to a list of 0s and 1s.\n",
        "  pred_labels_i = np.argmax(predictions[i], axis=1).flatten()\n",
        "  \n",
        "  # Calculate and store the coef for this batch.  \n",
        "  matthews = matthews_corrcoef(true_labels[i], pred_labels_i)                \n",
        "  matthews_set.append(matthews)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Calculating Matthews Corr. Coef. for each batch...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:900: RuntimeWarning: invalid value encountered in double_scalars\n",
            "  mcc = cov_ytyp / np.sqrt(cov_ytyt * cov_ypyp)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IUM0UA1qJaVB"
      },
      "source": [
        "The final score will be based on the entire test set, but let's take a look at the scores on the individual batches to get a sense of the variability in the metric between batches. \n",
        "\n",
        "Each batch has 32 sentences in it, except the last batch which has only (516 % 32) = 4 test sentences in it.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pyfY1tqxU0t9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 427
        },
        "outputId": "b2cf1af0-9b64-40c4-9143-1a02dd7e8549"
      },
      "source": [
        "# Create a barplot showing the MCC score for each batch of test samples.\n",
        "ax = sns.barplot(x=list(range(len(matthews_set))), y=matthews_set, ci=None)\n",
        "\n",
        "plt.title('MCC Score per Batch')\n",
        "plt.ylabel('MCC Score (-1 to +1)')\n",
        "plt.xlabel('Batch #')\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuwAAAGaCAYAAABUsxa9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde1zUZf7//+cwzDAKCGhopmKm4hkPmaVplplZnhUPpZJZaqVt2c8W3b61u33aLLOy9VBqaYqWqYCk5iGrrVYtzdzsgK5aeYhNJxFEEIfD/P5wZcM3DIPMOAM87rdbN+O63td1vWaGw5M31/s9JqfT6RQAAAAAvxTg6wIAAAAAlI7ADgAAAPgxAjsAAADgxwjsAAAAgB8jsAMAAAB+jMAOAAAA+DECOwAAfmLs2LHq1auXr8sA4GcCfV0AAFTUl19+qbi4OEnS6NGj9cwzzxiOOXXqlHr27Km8vDx16dJFCQkJhmO+/fZbrVy5Urt375bdbldAQIAaNmyorl27atSoUWratGmx48+dO6f33ntPW7du1aFDh5Sdna2wsDC1adNGd911lwYOHKjAQNffZrOyspSQkKAtW7bol19+UUFBgSIiItSyZUvddtttGj58eAWeGVyqV69e+uWXX4o+NplMqlOnjpo0aaJ77rlH/fr1u+y5t23bptTUVD366KOeKBUAihDYAVQZQUFB2rBhg6ZPny6r1VqsLyUlRU6ns9QAPW/ePM2bN08RERHq37+/mjVrpsLCQh06dEibNm3SypUrtWvXLoWEhEiSjhw5ookTJ+rnn39Wt27dNHHiREVEROjUqVPauXOnZsyYoUOHDumPf/xjqfWePXtWsbGxOnbsmO68804NGzZMFotFx44d09dff63ly5cT2L3g6quv1hNPPCFJKiws1IkTJ5ScnKwnnnhCdrtd48aNu6x5t23bpuTkZAI7AI8jsAOoMu644w5t2LBB27Zt0913312sLykpSbfccou++OILw7i1a9dq7ty5uvHGGzV//nyFhoYW63/yySc1b968oo9zc3M1adIkHT9+XHPnzlWfPn2KHT9x4kTt27dP3377rct6V69erZ9//ll/+tOfdN999xn67XZ7mY/ZG86ePVv0i0ll4nQ6lZOTo+DgYJfHhYaGatCgQcXaRo4cqR49eigpKemyAzsAeAt72AFUGa1bt1aLFi2UlJRUrH3fvn06ePCghg0bZhjjcDg0Z84c1axZU3PmzDGEdUmy2WyaNm1aUYhds2aNfvrpJ91///2GsH5RTEyMRo8e7bLen3/+WZLUtWvXEvsjIyMNbUeOHNGMGTN0yy23qG3bturevbsefvhhfffdd8WO27Ztm0aNGqUOHTqoY8eOGjVqlLZt22aYr1evXho7dqx++OEHPfDAA7r++us1cODAYjU++eST6t69u9q2batevXrpxRdfVE5OjsvHdun833//veLi4tSxY0d16dJF8fHxOnXqlOF4h8OhN954Q/369VO7du3UuXNnPfTQQ/rhhx+KHffll18WvdYrV67U3XffrXbt2mnJkiVu1XWpsLAwWa1WWSyWYu379u3T9OnTdeedd6p9+/ZFz+WHH35Y7LixY8cqOTlZktSiRYui/37/uWi32/Xcc8/p9ttvV9u2bdW1a1fdf//92r59u6GeEydO6IknntANN9yg9u3b64EHHtBPP/10WY8NQOXHGXYAVcqwYcP0wgsv6MSJE6pXr56kC2fQ69Spo1tvvdVw/Ndffy273a5Bgwapdu3abq2xZcsWSRfOylZEVFSUpAtn/6dNm1bmfvdvv/1W48aNU35+vmJjY9W8eXNlZmZq165d2rt3r9q2bStJWrlypZ599lldd911euSRRyRJycnJmjx5sp599llD3WlpabrvvvvUt29f9enTpyiMf/fdd7rvvvtUq1YtjRw5UvXq1dP+/fuVkJCgvXv3KiEhwRBwS/Lrr79q3Lhx6tOnj+6880798MMPSkxM1Hfffae1a9eqRo0akqS8vDw98MAD2rt3rwYNGqTRo0fr7NmzWr16te655x6tWLFC7dq1Kzb3smXLlJGRoeHDhysyMlJXX311mfUUFBQoPT1d0oUtMXa7XcuXL1d2drZGjRpV7NgPP/xQP/74o/r27asGDRooIyNDycnJmjJlimbPnq0BAwZIkh566CEVFhbqq6++0qxZs4rGd+rUSZJ0/Phx3XPPPTp16pQGDRqktm3b6ty5c/rmm2+0Y8cO3XzzzUVjcnJyNGbMGLVv315Tp07V8ePHtXz5cj3yyCPasGGDzGZzmY8RQBXjBIBK7osvvnBGR0c733zzTWd6erqzTZs2ztdff93pdDqd586dc15//fXOF154wel0Op0dOnRwjhkzpmjs8uXLndHR0c4lS5a4vV6XLl2cnTp1qnDdGRkZzp49ezqjo6OdXbt2dT766KPOhQsXOnfv3u0sKCgodmxhYaGzX79+zrZt2zpTU1MNc108PiMjw9mhQwdn7969nVlZWUX9WVlZzttvv93ZoUMHZ2ZmZlH7bbfd5oyOjnauXr3aMOeAAQOcd955Z7F5nE6nc+vWrc7o6GhnYmJimY/x4vxLly4t1r506VJndHS0c+HChYa2zz77rNixWVlZzp49exZ73S6+5jfccIPzt99+K7OOS+u59L927do5V61aZTg+Ozvb0JaTk+Ps06eP86677irWHh8f74yOji5x3QcffLDEx+Z0Oou91mPGjHFGR0c7Fy1aVOyYxYsXlzoeQNXHlhgAVUpERIR69epVtD1h69atysrKKnE7jHRhv7akcu3ZPnv2bJn7pN0RFhampKQkTZgwQaGhodqyZYtefvlljR49Wr1799Y///nPomNTU1N18OBBDR06VC1btjTMFRBw4dv59u3blZOTo7FjxxZ7TCEhIRo7dqxycnK0Y8eOYmPDw8M1dOjQYm0HDhzQgQMH1L9/fzkcDqWnpxf9d/3116tmzZolbuUoSUhIiO69995ibffee69CQkKKbS15//33dd1116lNmzbF1nM4HOrWrZv27Nmj3NzcYvMMGjRIderUcauOixo0aKClS5dq6dKlWrJkiV544QW1b99ef/nLX5SYmFjs2Jo1axb9/7lz53T69GmdO3dON910kw4fPlz0+eNKRkaGPv/8c/Xo0UM9evQw9F987X7/8cW7Hl100003SbqwJQpA9cOWGABVzrBhwzRx4kR99dVXSkxMVExMjJo1a1bisRdDbXZ2ttvzh4SElOt4V2rXrq1p06Zp2rRpOn36tP71r39p06ZNev/99zVlyhSlpKSocePGRfvdW7du7XK+48ePS5KaN29u6LvYduzYsWLtjRo1MmyzOHz4sCRp7ty5mjt3bolr/fbbb2U/wP/Of+lde6xWqxo1alSslsOHDys3N7fUPf2SdPr0adWvX7/o42uvvdatGn6vZs2a6tatW7G2AQMGaMiQIXruuefUq1cvRURESLpwO9A5c+boo48+KnHP/ZkzZ8r8Ze/o0aNyOp1lvnYX1a1bV0FBQcXawsPDJV0I/wCqHwI7gCqne/fuqlevnubPn68vv/xSf/nLX0o99mKIvfSiRleaN2+u3bt369ixY2rUqFFFyy0SERGh2267Tbfddpvq16+vN954Qxs3bizah+4tF/eQl2T8+PElnhWWpFq1anm0DqfTqejoaM2YMaPUYy69zsBV7eURGBiom266ScuXL9e+ffvUs2dPOZ1OjR8/XocPH1ZcXJzatm2r0NBQmc1mJSYmasOGDSosLPTI+r/nao+60+n0+HoA/B+BHUCVYzabNXjwYC1cuFA2m039+/cv9dhOnTopMjJS27Zt0+nTp4vOrLrSp08f7d69W2vWrCm6n7entW/fXtKFu4VIUpMmTSRd2BrjysVfIA4ePGg4U33o0KFix7jSuHFjSRe2Z1x6Nrq8jh07JofDUewsu8Ph0LFjx3TdddcVW/P06dO66aabDNtEroT8/HxJ//try4EDB7R//35NnjxZf/jDH4odu2bNGsN4k8lU4rxRUVEymUxlvnYAUBr2sAOokkaNGqUpU6bor3/9q8stC1arVY8//riys7M1derUEvcknz9/Xq+88kpR3/Dhw9WkSRMtWbKkxFslShfusLJy5UqXNe7du1dnzpwpse/ivBe38rRs2VLNmzdXYmKiDh48aDj+4pnXm2++WTVr1tSKFSuKPZazZ89qxYoVqlmzZrE7kpSmdevWio6O1qpVqwxbaKQL4dbd7Rlnz57VO++8U6ztnXfe0dmzZ9W7d++itsGDB8tut2vp0qUlzuPuFpzLcf78eX3++eeS/rft6OIvDZee1f73v/9tuK2j9L/97pc+L+Hh4brlllv02WefGa4fKGl+ALgUZ9gBVEnXXHON2+84GRsbq19//VXz5s1Tnz59ir3T6eHDh7V582alp6dr4sSJki5sw1i4cKEmTpyoyZMnq3v37urWrZvCw8OVnp6uL7/8Uv/85z/14IMPulx3/fr1SkpKUs+ePRUTE6Pw8HBlZGTo008/1ZdffqlmzZoVXSxrMpn0/PPPa9y4cRo+fHjRbR3PnDmj3bt3q0ePHho7dqxq1aqladOm6dlnn9WIESM0ZMgQSRdu63jkyBE9++yzJd5r/lImk0mzZs3Sfffdp4EDB2rYsGFq1qyZcnNzdeTIEX344Yd64oknDBerliQqKkrz58/XwYMH1aZNG33//fdKTEzUddddp7FjxxYdFxcXpx07dmjWrFn64osvdNNNNykkJERpaWn64osvZLValZCQUOZ6ZcnKylJKSoqkC2H55MmTWr9+vY4dO6YRI0YU7Ytv2rSpmjdvrjfffFO5ublq0qSJfvrpJ7333nuKjo7W999/X2ze9u3ba8WKFfrrX/+qnj17ymKxKCYmRo0aNdLTTz+tH374QRMmTNDgwYPVpk0bnT9/Xt98840aNGigJ598ssKPC0DVRWAHAElTpkxRz549tWLFCm3btk3vvvuuAgICFBUVpbvvvlv33HNPsTP1jRs31rp16/Tee+9py5YteuONN5STk6OwsDC1bdtWL7zwQtE9ukszatQohYaG6ssvv9TSpUuVkZEhi8Wixo0ba8qUKbr//vuL3aUkJiZGa9eu1YIFC7Rp0yatWrVK4eHhiomJKbrftySNHj1adevW1VtvvaX58+dLunCGfv78+cXOaJelVatWSk5O1sKFC/Xxxx9r1apVCg4OVoMGDTRkyBCXF4f+3tVXX605c+boxRdf1MaNG2WxWDRgwADFx8cXe3wWi0ULFy7UO++8o5SUlKKLXevWrat27doV/fJRUb/++qv++Mc/Fn1co0YNNW3aVH/+85+L3YfdbDZr4cKFevHFF5WcnKxz586pefPmevHFF7V//35DYO/fv79SU1O1ceNGbd68WYWFhZo5c6YaNWqkRo0aKTExUfPnz9dnn32mlJQU1apVSy1btqzw/fwBVH0mJ3+LAwB4Sa9evdSgQQOPnBkHgOqKPewAAACAHyOwAwAAAH6MwA4AAAD4MfawAwAAAH6MM+wAAACAHyOwAwAAAH6M+7D/1+nT2SosZHcQAAAAvCMgwKSIiOByjyOw/1dhoZPADgAAAL/DlhgAAADAjxHYAQAAAD9GYAcAAAD8GIEdAAAA8GMEdgAAAMCPEdgBAAAAP+bTwH7y5EnNnj1bY8eOVceOHdWiRQt9+eWXbo8/fPiwHnjgAXXs2FFdunRRfHy80tPTvVgxAAAAcGX5NLD/9NNPWrx4sU6cOKEWLVqUa+yvv/6q0aNH69ixY5o6darGjx+vTz75RA888IDy8vK8VDEAAABwZfn0jZPatGmjL774QhEREdq2bZsmT57s9tg33nhD58+fV0JCgurVqydJiomJ0f3336+UlBTFxsZ6q2wAAADgivHpGfaQkBBFRERc1titW7eqV69eRWFdkrp166Zrr71WmzZt8lSJAAAAgE9VyotOT5w4oVOnTqlt27aGvpiYGKWmpvqgKgAAAMDzKmVgP3nypCQpMjLS0BcZGalTp06poKDgSpcFAAAAeJxP97BfrvPnz0uSrFaroS8oKEiSlJubq+DgYLfnrFMnxDPFAVdIfoFDgWbj10Bp7QCA6s2Z75Qp0FTuPvhepQzsF0O5w+Ew9F0M8zabrVxznjp1VoWFzooXB1whkZGhmr/iTkP75DFbZLdn+aAiAIA/i4wM1X9mHSuxr/4fG/Gz4woICDBd1kniSrklpm7dupIku91u6LPb7apTp47MZvOVLgsAAADwuEoZ2OvVq6fatWvru+++M/Tt27dPrVq18kFVAAAAgOdVisB+9OhRHT16tFhbnz599PHHH+vEiRNFbTt37tTPP/+svn37XukSAQAAAK/w+R72BQsWSJIOHz4sSUpJSdGePXtUq1YtjRkzRpI0btw4SdLHH39cNO6hhx7S5s2bFRcXpzFjxignJ0dvvfWWWrZsqUGDBl3ZBwEAAAB4ic8D+2uvvVbs48TERElSgwYNigJ7SerXr68VK1bohRde0MsvvyyLxaJbb71VM2bMKPHuMQAAAEBl5PPAfuDAgTKP+f2Z9d9r3ry53nrrLU+XBAAAAPiNSrGHHQAAAKiuCOwAAACAHyOwAwAAAH6MwA4AAAD4MQI7AAAA4McI7AAAAIAfI7ADAAAAfozADgAAAPgxAjsAAADgxwjsAAAAgB8jsAMAAAB+jMAOAAAA+DECOwAAAODHCOwAAACAHyOwAwAAAH6MwA4AAAD4MQI7AAAA4McI7AAAAIAfI7ADAAAAfozADgAAAPgxAjsAAADgxwjsAAAAgB8jsAMAAAB+jMAOAAAA+DECOwAAAODHAn1dAPxPRJhVgdYgQ3u+47xOZzp8UBEAAFdeRHiwAi3Gc5v5eYU6nZHtg4pQXRHYYRBoDVLq/EGG9laTUyQR2AEA1UOgJUCfJ9gN7T3GRvqgGlRnbIkBAAAA/BiBHQAAAPBjBHYAAADAjxHYAQAAAD9GYAcAAAD8GIEdAAAA8GPc1hEAqpnQcJtsFouhPTcvT1kZuT6oCADgCoEdAKoZm8Wi/olLDO0bho1XlgjsAOBv2BIDAAAA+DECOwAAAODHCOwAAACAH2MPOwAAXhAaXkM2i/HHbG5evrIyzvmgIgCVFYEdAAAvsFkCNTRxh6E9aVg3ZfmgHgCVF1tiAAAAAD/m08DucDj00ksvqXv37oqJidGIESO0c+dOt8bu2LFDY8eO1Y033qgbbrhBI0eO1AcffODligEAAIAry6eBffr06Vq2bJkGDhyop556SgEBAZowYYL27t3rctwnn3yi8ePHKz8/X48++qgee+wxBQQEaOrUqVqzZs0Vqh4AAADwPp/tYd+3b582btyoGTNmaNy4cZKkwYMHq3///po9e7ZWrlxZ6tiVK1cqMjJSy5Ytk9VqlSSNGDFCt99+u1JSUjR8+PAr8RAAAAAAr/PZGfbNmzfLYrEUC9dBQUGKjY3Vnj17dPLkyVLHnj17VmFhYUVhXZKsVqvCwsIUFBTk1boBAACAK8lngT01NVVNmjRRcHBwsfaYmBg5nU6lpqaWOrZLly46ePCg5syZo6NHj+ro0aOaM2eOfv75Z40fP97bpQMAAABXjM+2xNjtdtWrV8/QHhkZKUkuz7A/9NBDOnr0qN544w29/vrrkqSaNWtqwYIFuvnmm71TcCVTO8wqs9X414YCx3mlZzp8UFHFhYdZZSnhMeU5ziujgo/JW3OHh1tksdiM8+blKiMj77LnRXGh4VbZLMbXLzfvvLIyKufnO64M7pV+5YSFB8tqMZ4ndOQVKjMj22vrhocHy1LCunl5hcrw4rpVUe2wmjJbzYb2AkeB0jNzfFBR9eF2YP/pp5+0a9cuHTx4UOnp6TKZTIqIiFB0dLRuuOEGNWnSpFwL5+bmymKxGNovbmk5f/58qWOtVquuvfZa9e3bV3fccYcKCgq0evVqPf7443r77bcVExNTrlokqU6dkHKP8XfH5sYZ2ho9ulyRkZe/bSgyMrQiJVXYR2/2M7Td/uDGCj2mi9YtucvQNnj8pgrPvfztOw1tceO2KDLSGOQ9xdevky/clfKAoW3ToLdk88DnRnVSHT93Bq3damhLie0jmxefi+r4PEvSm0nGk3EPDq3r9edj86rfDG19R11VoXWr4mvozmP69eX9hrar/7+WVfL58CcuA/v58+eVmJio9957T//+97/ldDpLPM5kMik6OlqjRo3S0KFD3dpHbrPZlJdnPMN4Mai7muP//u//9O2332rt2rUKCLjwW/Ndd92l/v376/nnn9eqVavKXP9Sp06dVWFhyY+vMnL1hWO3u37LjoqM9SZv1uWtuStjzZURz0X58Hz9D1+jV46vno+q+PPwcpUVqqvb8+ELAQGmyzpJXGpgX7dunebMmaMTJ06oc+fOmjp1qjp27KioqCiFh4fL6XQqMzNTR44c0b/+9S999tlnevbZZ7Vw4UJNnTpVgwYNcrlwZGRkidte7Ha7JKlu3boljnM4HFq7dq0mTZpUFNYlyWKxqEePHnr33XeVn5+vwEDexBUAAACVX6mp9i9/+YtGjRqlsWPHqkGDBiUeY7PZVK9ePXXp0kUTJ07UL7/8omXLlunPf/5zmYG9ZcuWSkhIUHZ2drELT7/55pui/pJkZGQoPz9fBQUFhr78/Hzl5+eX+pcAV8JDg2S2GrfoFDjylJ6ZW+75UH242qcOAABQUaUG9m3btumqq64q12QNGjTQn/70J02YMKHMY/v27aslS5ZozZo1RfdhdzgcSkpKUqdOnYouSE1LS9O5c+fUtGlTSVKdOnVUq1Ytffjhh5oyZUrRPvjs7Gx98sknio6OLnFvfFnMVovsr68wtEc+PEYSwQuls1hsereEfer3jNvig2oAAEBVU2pgL29Y/72Ld3pxpX379urbt69mz54tu92uqKgoJScnKy0tTTNnziw6Lj4+Xrt27dKBAwckSWazWePHj9ecOXM0cuRIDRw4UIWFhVq7dq1+/fVXxcfHX3bdAAAAgL/x6UbvWbNmac6cOUpJSVFmZqZatGihRYsW6frrr3c57uGHH1bDhg21fPlyzZ8/Xw6HQy1atNC8efN0xx13XKHqAQAAAO/zWGD/5JNPtHXr1mJnx8sSFBSk+Ph4l2fFExISSmwfMGCABgwYUO46AQAAgMrEY+90un//fq1bt85T0wEAAACQBwM7AAAAAM9zuSUmLs74TpmlSUtLq3AxAAAAqHxqh9WU2Wo2tBc4CpSemeODiqoWl4F9165dCgwMdOs2ifn5+R4rCgAAAJWH2WrWr69+a2i/emo7H1RT9bgM7PXq1VOrVq30xhtvlDnRggULNHfuXI8VBgAAAKCMPeytW7fWd99959ZEJpPJIwUBAAAA+B+XZ9jbtGmjTz75RCdOnCh659HShIaGqn79+h4tDgAAwB9FhAcr0GI875mfV6jTGdk+qAhVmcvAPn78eA0ZMkQRERFlTjRmzBiNGTPGY4UBAAD4q0BLgL58+6Sh/cZxdX1QDao6l4G9Zs2aqlmz5pWqBQAAAMAluA87AAAA4McI7AAAAIAfu6zAfvr0abVq1Uo7d+70dD0AAAAAfueyz7A7nU5P1gEAAACgBGyJAQAAAPwYgR0AAADwYy5v63hRWlpasY8zMzMlSenp6Ya+a665xkOloTKKCLMq0BpkaM93nNfpTIcPKsKVVCvcqiCL8fU/n3deZzJ4/QF/Vyu8poIsZkP7+bwCncnIUVh4TVlL6HfkFSgzI+dKlAhUS24F9l69eslkMhnap02bZmhLTU2teFWotAKtQfpq4QBDe+dJ6yUR2Kq6IEuQnlzb19D+Uuxm8foD/i/IYtaM5F8M7TOHNJAkWS1mzUz+j6F/xhDe6RzwJrcC+/PPP18ssGdnZ+u5557T+PHj1axZM68VBwAAAFR3bgX2oUOHFvv49OnTeu6559S9e3d17drVK4UBAAAA4KJTAAAAwK+5dYYd8BQuSvV/YeEWWS02Q7sjL1eZGXk+qAgAUNnVDqsps9V4wXKBo0DpmVywXBYCO66oQGuQdi7qb2jvOnGDuCjRP1gtNv3tvTsN7U+N3CKJwA4AKD+z1awTc742tNd7vJMPqvG+2mE1ZLYaY3aBI/+y5ruswB4aGqrly5erVatWl7UoAAAAUFWZrYE6uWC1ob3uIyMua77LCuyBgYHq0qXLZS0IAAAAwH1siYFfcbXHHZ7jap96VRQaHiSbxWpoz81zKCuDzy0AgH8jsMOvBFqD9Nnifob2WyZs9EE1VZfVYtNL7xr3qT95zxYfVON9NotVd617zNC+afBryhKBHQDg3wjsAAAAQDm4uqg0PfOcx9cjsAMAAADlYLYG6uT8ZEN73clDvLIeb5wEAAAA+DECOwAAAODHLjuwp6enKz093ZO1AAAAALhEufawnzhxQq+88oo++ugjZWdnS5JCQkJ0++23a+rUqapXr55XigQAAACqK7cDe1pamkaMGKHffvtNrVq1UrNmzSRJhw8f1rp167R9+3atXr1a9evX91qxAAAAQHXjdmB/7bXXdObMGS1cuFA9e/Ys1vfpp5/q0Ucf1WuvvaYXXnjB40UCAAAA1ZXbe9i3b9+ue++91xDWJalnz56655579Pnnn3u0OAAAAKC6czuwZ2ZmqnHjxqX2N27cWGfOnPFIUQAAAAAucDuwX3311dq1a1ep/V999ZWuvvpqjxQFAAAA4AK3A3vfvn21efNmvfzyy8rKyipqP3v2rF555RVt2rRJd999t1eKBAAAAKorty86feSRR/TVV19p8eLFWrJkierWrStJOnnypAoKCtSpUyc9/PDDXiu0OqodFiSz1WpoL3A4lJ553gcVAUD1EhpeQzaL8Udlbl6+sjLO+aAi/xUWHiyrxXge0JFXqMyMbK+tGx4eLEsJ6+blFSrDi+v6q9phNWW2mg3tBY4CH1TjfbXDashsNX6NFjjylZ5Zdb5G3Q7sNWrUUEJCgpKSkrRt2zYdP35cktS9e3f17t1bQ4YMUWBguW7rjjKYrVb9Z8Ezhvb6jzwricAOAN5mswRq8NqPDe3rYnspq4TjqzOrJUB/T4T41lUAACAASURBVD5haP/DEO++R4vFEqD1q38ztA8YcZVX1/VXZqtZv8z+j6G9wbSqedttszVQJ/7+D0N7vT/cesVr8aZyJezAwECNGDFCI0aM8FY9AAAAAH7H7T3scXFx2rlzZ6n9X3zxheLi4sq1uMPh0EsvvaTu3bsrJiZGI0aMcLnGpdavX6/Y2Fh16NBBXbp00ZgxY7Rv375y1QAAAAD4M7fPsO/atUvDhw8vtT89PV27d+8u1+LTp0/X1q1bFRcXp8aNGys5OVkTJkxQQkKCOnbs6HLsq6++qjfffFMDBw7UyJEjlZOTo/3798tut5erBl+rHWaT2WoxtBc48nxQDeBdtcKtCrIEGdrP553XmQyHy7Gh4VbZShibm1d5t4eFhttksxi//nPz8pSVkeuDigDfYB864JrHNp2fOXNG1hIukCzNvn37tHHjRs2YMUPjxo2TJA0ePFj9+/fX7NmztXLlylLHfv3111q4cKHmzp2rO+64o6Kl+5TZatGJ143vDlvv4ek+qAbwriBLkB5J6mtoXzB0syTXgd1mCdJd7w8ytG8amOKp8q44m8WifklzDO0bhz6uLBHYUX1YLAFamWQ84TZ6aKQPqgH8j8vAvn//fu3fv7/o46+++koFBcarjDMyMvTuu++qadOmbi+8efNmWSyWYmftg4KCFBsbq1dffVUnT54suhPNpZYvX6527drpjjvuUGFhoc6dO6fg4GC314YUEWZVoNV4tjLfUXnPVlZGYeEWWS02Q7sjL1eZGfyVBQAAlBHYt23bpnnz5kmSTCaT3nvvPb333nslHhscHKynnnrK7YVTU1PVpEkTQ9COiYmR0+lUampqqYF9586d6tevn1555RUlJCQoJydHDRo00OOPP66BAwe6XYO7XG1bSc+snGfBAq1B+nHuYEP7dY+u80E11ZfVYtOSZX0M7ePv2yqJwA4AAMoI7EOGDFGXLl3kdDp13333adKkSbr55puLHWMymVSzZk01a9ZMQUHGM7alsdvtqlfPeKunyMgLf/46efJkieMyMzOVkZGhjRs3ymw2a9q0aQoPD9fKlSv15JNPqkaNGh7fJmO2WmR/famx1ofvV2SkMchfCZGRoVVurK/q8tXclfH58NfX3x/XrShf1eWvz4cv+PPXqKOgUFZzCfc8L6XdE/z1Z0dFxtauHSKz2WRoLyhwltjuqXX9VVX7PlyVvo+6DOwNGjRQgwYNJEkzZ87UDTfcoIYNG3pk4dzcXFlKuNjqYug/f77krRk5OTmSLmzDWb16tdq3by9JuuOOO3THHXdo/vz5V3Rfu91esTvxXu6LWta6rua127P8cl1v1lWWsuq63LGuuPN8XO663qy5IjVVxufZm3xVl78+H77gq89ZT3zPGpH4g6F99bDWFZrbm9+j/fH7jtls0scrjXvne42O9OnPpIqoSFisjN+H/fX7qKdDu9u/hg8ZMsRjYV2SbDab8vKMf/K/GNRLO1t/sb1hw4ZFYV2SrFar7rzzTu3fv1/Z2VxRDgAAgKrBO383c0NkZGSJ214u3paxtP3r4eHhslqtuuoq4zuYXXXVVXI6nTp79qxniwUAAAB8xGeBvWXLlvrpp58MZ8O/+eabov6SBAQEqFWrVjpxwvj2x7/++qvMZrPCwsI8XzAAAADgAz4L7H379lVeXp7WrFlT1OZwOJSUlKROnToVXZCalpamw4cPG8b+5z//0fbt24vazp49q02bNqljx46y2Yy3yQMAAAAqI4+9cVJ5tW/fXn379tXs2bNlt9sVFRWl5ORkpaWlaebMmUXHxcfHa9euXTpw4EBR2z333KM1a9bo0Ucf1bhx41SrVi0lJiYqKytLTzzxhC8eDgAAAOAVPgvskjRr1izNmTNHKSkpyszMVIsWLbRo0SJdf/31LsfVqFFDy5cv16xZs7RixQrl5uaqTZs2Wrp0aZljAQAAgMrEp4E9KChI8fHxio+PL/WYhISEEtsjIyP10ksveas0AAAAwC94bA97SkqK4uLiPDUdAAAAAHkwsKelpWn37t2emg4AAACAfHiXGAAAAABlc7mH/fbbb3d7It6sCEBVFBoeJJvFamjPzXMoK+O8F9e1yWaxlLBunrIycr22LgDviwgLVqDVeM4031Go05m8W/ulaofVkNlqjKwFjnwfVOMbLgP7L7/8orCwsFLfdfT3cnP5AQKg6rFZrLo7+S+G9g+G/EVZ8l5gt1ks6pc0z9C+cegUZYnvt0BlFmgN0LcLjW8A2W5SPR9U4//M1kCdeG27ob3eYzf7oBrfcBnYGzZsqMaNG+utt94qc6IFCxZo7ty5HisMAAAAQBl72Nu0aaPvv//erYlMJpNHCgIAAADwPy7PsLdu3VpbtmzR8ePH1bBhQ5cTXXPNNercubNHi4NrtcOCZLYa99YWOBw+qAYAAADe4DKwT5o0SZMmTXJrokGDBmnQoEEeKQruMVutSps/1dB+zeRXfVANAAAAvIHbOgIAAAB+7LIDe2FhodLS0uRg+wUAAADgNZcd2NPT03X77bdrz549nqwHAAAAwO+43MNeFqfT6ak6AL8VHmaVxRpkaM9znFdGpvf+whQWbpHVYjO0O/JylZmR57WxqPx40yUAqFoqFNiB6sBiDdKapX0N7cPv3yzJe4HdarFpUcKdhvaJY7dIch26rRabXnvHOPaxe8sei8rPZrGoX+JiQ/vGYRN40yUAqIS46BQAAADwY5cd2G02m4YMGaK6det6sh4AAAAAv3PZW2JCQkI0c+ZMT9YCAAAA4BJsiQEAAAD8WKmB/d5779Xu3bvLPeHOnTt1zz33VKgoAAAAABeUuiWmbt26Gjt2rFq3bq3Bgwfrlltu0bXXXlvisYcOHdKnn36qlJQUHTx4UHfffbe36gUAAACqlVID+5w5c7Rnzx4tWLBAM2fO1MyZM1WrVi01aNBA4eHhcjqdyszM1NGjR5WdnS2TyaTu3bvr2WefVYcOHa7kYwAAAACqLJcXnV5//fV66623dPToUW3evFm7d+/W4cOH9eOPP8pkMikiIkKdO3dWly5d1KdPHzVs2PBK1Q0AAABUC27dJSYqKkoTJ07UxIkTvV0PAAAAgN/hnU5RbhFhVgVagwzt+Y7zPqgGAACgaiOwo9wCrUH65vWBhvb2D7/vg2oAAACqNu7DDgAAAPgxzrADAKq00PAaslmMP+5y8/J9UA0uR3h4sCwW4znGvLxCH1QDXHkEdgBAlWazBGrQ2g8M7SmxvGdIZWGxBCgx8TdD+7BhV/mgGuDKI7ADAIqEhttks1gM7bl5ecrKyPXiuqWfBc/KOOe1dQGgMiCwAwCK2CwW9U9cZmjfMOw+Zcl7gd1mCdSAtWsN7etjY5XltVUBoHIo10WnBQUFWrdunaZNm6b7779fP/zwgyQpMzNT69at04kTJ7xSJAAAAFBduX2G/dy5cxo/frz27t2rGjVqKDc3V5mZmZKkkJAQzZ49W8OGDdPUqVO9VmxlVDvMJrPV+OflAkee0jO9d7YKAAAAVYPbgX3u3Ln67rvvNG/ePHXq1EndunUr6jObzerTp4/++c9/EtgvYbZadPKN1wztdR96TPLin5cBAJVXaHhN2SxmQ3tuXoGyMnJ8UBEAX3I7sG/evFkjR45U7969dfr0aUN/VFSUPvjAeBU+AAAoH5vFrNjEfxna1w7rwJ5+oBpyew/7yZMn1aJFi1L7a9SooezsbI8UBQAAAOACtwN7eHi4y4tKDx48qLp163qkKAAAAAAXuB3Yu3btqqSkJJ07Z7wf7rFjx5SYmKgePXp4tDgAAACgunM7sE+ZMkVnzpxRbGys3n33XZlMJn3++ed6+eWXNXToUFmtVk2aNMmbtQIAAADVjtuBvXHjxnr77bdlNpv197//XU6nU0uWLNHixYt19dVXa9myZapfv743awUAAACqnXK902nbtm31/vvv69///rcOHz4sp9Opa6+9Vq1bt/ZWfQAAAEC15tYZ9uzsbPXu3Vtvv/22JCk6Olp33XWX7r777gqFdYfDoZdeekndu3dXTEyMRowYoZ07d5Z7ngkTJqhFixb629/+dtm1AAAAAP7IrcAeHBysjIwMBQcHe3Tx6dOna9myZRo4cKCeeuopBQQEaMKECdq7d6/bc/zjH//QV1995dG6AAAAAH/h9h729u3b69tvv/XYwvv27dPGjRs1bdo0/fGPf9TIkSOL9sHPnj3brTkcDodmzpypBx54wGN1AQAAAP7E7cA+bdo0bd68WYmJiXI6nRVeePPmzbJYLBo+fHhRW1BQkGJjY7Vnzx6dPHmyzDmWL1+u3NxcAjsAAACqLLcvOp05c6Zq1aql//f//p9eeuklRUVFyWazFTvGZDJp2bJlbs2XmpqqJk2aGLbZxMTEyOl0KjU11eUbMdntdi1YsEDPPPOMatSo4e7DAAAAACoVtwP78ePHJano1o2//fZbhRa22+2qV6+eoT0yMlKSyjzD/sorr6hJkyYaNGhQheoAAAAA/Jnbgf3jjz/26MK5ubmyWCyG9qCgIEnS+fPnSx27b98+rVu3TgkJCTKZTB6tq7wiI0MZ66GxvqqrIvz1uayMY731GvJ55x9jK8Jf1/XX18FXX0uM9f3YiqqM34d9Ma8v1i3Xfdg9yWazKS8vz9B+MahfDO6Xcjqd+tvf/qY+ffqoc+fOXq3RHXZ7lst+Vy+a3Z512S9qVRxb0bkrwh+fD38d60pFXt+KzO2KN9f15efd5c7ry68zV/z1efbV5443v5a8ta4r1XGsK/74+e7O2v74tVTWut7ira+V0pQ7sJ89e1Y7duzQsWPHJEmNGjVSt27dFBISUq55IiMjS9z2YrfbJanU/esffvih9u3bp6lTpxZt0/l9bcePH9dVV11l2F8PAAAAVEblCuxr1qzRCy+8oJycnKI7xZhMJtWsWVPTp08vdseXsrRs2VIJCQnKzs4uduHpN998U9RfkrS0NBUWFuq+++4z9CUlJSkpKUmLFy/WLbfcUp6HBgAAAPgltwP7Rx99pKefflqNGjXSY489pubNm0uSDh48qBUrVuiZZ55RnTp11KtXL7fm69u3r5YsWaI1a9Zo3Lhxki7cVz0pKUmdOnUquiA1LS1N586dU9OmTSVJvXr1UsOGDQ3zTZ48WbfddptiY2PVpk0bdx8WAAAA4NfcDuxvvvmmmjZtqtWrVxc7I961a1cNHTpUI0eO1OLFi90O7O3bt1ffvn01e/Zs2e12RUVFKTk5WWlpaZo5c2bRcfHx8dq1a5cOHDggSYqKilJUVFSJczZq1Ei9e/d29yEBAAAAfs/tN07av3+/hgwZYrhvuiSFhIRo8ODB2r9/f7kWnzVrlsaOHauUlBQ999xzys/P16JFi3T99deXax4AAACgqvLYXWIu5/aKQUFBio+PV3x8fKnHJCQkuDXXxTPwAAAAQFXi9hn2Fi1aKDk5WTk5OYa+7OxsJScnl3qhKAAAAIDL4/YZ9gcffFBTpkzRkCFDFBcXV3QR6KFDh5SQkKCjR49q7ty5XisUAAAAqI7cDuy9e/fW008/rdmzZ+v//u//irbAOJ1O1ahRQ08//TQXfAIAAAAeVq497KNHj9aAAQO0ffv2ojctatSokW6++WaFhvrurXSBsoSHWWWxGt89N89xXhmZDh9UhKoiNDxINovV0J6b51BWxnkfVFQ1hYbXkM1i/JGVm5evrIxzPqio4kLDa8pmMRvac/MKfFANKouIsGAFWo07mvMdhTqdme2DinAllPui01q1aumuu+7yRi2A11isQdqwxPh523/8JkkEdlw+m8Wqu5OfN7R/MORPyhKB3VNslkANWLvO0L4+drC89+bj3mWzmDUscbehPXHYDT6oBpVFoDVAP732q6G9yWNX+6AaXCluX3T6ww8/aOXKlaX2r1y5UqmpqR4pCgAAAMAFbgf2efPm6R//+Eep/Z999pnmz5/viZoAAAAA/Jfbgf3bb7/VDTeU/me6G264Qfv27fNIUQAAAAAucDuwnz59WuHh4aX216pVS6dPn/ZIUQAAAAAucDuw16lTRwcPHiy1/9///rfCwsI8UhQAAACAC9wO7N26ddPatWtLDO2HDh1SYmKiunXr5tHiAAAAgOrO7ds6Pvzww9q6datiY2M1bNgwtWrVSpKUmpqqxMREWSwWPfLII14rFAAAAKiO3A7sUVFRevvttzVjxgy98847xfqaN2+u559/Xtdee62n6wMAAACqtXK9cVK7du20YcMGpaam6ueff5YkNWnSRC1btvRGbQAAAEC1V+53OpWkVq1aFW2JAQAAAOA9lxXYJenYsWPauHGjTpw4oWbNmmnYsGGy2WyerA0AAACo9lwG9jVr1ighIUFLly5VnTp1itq3b9+uKVOmKDc3V06nUyaTSatWrdKqVasUHBzs9aIBAACA6sLlbR3/8Y9/KDg4uFhYdzqdeuaZZ5Sbm6uJEyfq9ddf15AhQ3Tw4EG9/fbb3q4XAAAAqFZcnmHfv3+/7rrrrmJtX3/9tX755RcNHjxYU6dOlSTddttt+uWXX/TRRx9p8uTJ3qsWAAAAqGZcnmFPT09Xo0aNirV9/fXXMplMhiDfs2dPHTlyxPMVAgAAANWYy8AeGBiovLy8Ym3ffvutJKlDhw7F2sPDw+VwODxcHgAAAFC9uQzsDRo00N69e4s+Ligo0J49e9S4cWOFhYUVOzYjI0MRERHeqRIAAACoplzuYe/Tp48WLFigjh076qabblJiYqLS09M1bNgww7H79u1Tw4YNvVYoAMD3QsNtslkshvbcvDxlZeSWMbaGbBbjj53cvPwK1lT6vFkZ5yo0NwD4A5eBPS4uTikpKfrb3/4m6cIdYurXr6/777+/2HFZWVn69NNPNW7cOK8VCgDwPZvFov5rVxjaN8SOUZZcB3abJVD9175XwtiRFawpUAPXrje0vx87QFkVmhkA/IPLwB4SEqLExEStXr1aR44cUVRUlIYPH65atWoVO+7w4cMaOnSo+vXr59ViAQAAgOqmzHc6DQkJ0fjx410e06FDB8NFqAAAAAAqzuVFpwAAAAB8i8AOAAAA+DECOwAAAODHCOwAAACAHyOwAwAAAH6MwA4AAAD4MZeBvaCgQLNnz9a7777rcpJ33nlHr7zyipxOp0eLAwAAAKo7l4H9/fff11tvvaV27dq5nCQmJkaLFy/Whg0bPFocAAAAUN25DOybNm1St27d1LZtW5eTtG3bVt27d9fGjRs9WhwAAABQ3bkM7N9//726du3q1kQ33nijvvvuO48UBQAAAOACl4E9MzNTderUcWui2rVrKyMjwyNFAQAAALjAZWAPDg7W6dOn3ZooIyNDwcHBHikKAAAAwAUuA3uzZs20fft2tybavn27mjVr5pGiAAAAAFzgMrDfcccd2rFjh7Zt2+Zyko8++kg7duxQnz59PFocAAAAUN25DOyjRo1SVFSUHn/8cb366qs6fvx4sf7jx4/r1Vdf1eOPP65rr71Wo0aN8mqxAAAAQHUT6KrTZrNp0aJFmjRpkhYuXKhFixYpJCREwcHBys7O1tmzZ+V0OtWkSRMtXLhQQUFB5Vrc4XDotddeU0pKis6cOaOWLVtq6tSpZd6ZZuvWrfrggw+0b98+nTp1SvXr19dtt92mRx55RKGhoeWqAQAAAPBnLgO7JDVu3FgpKSlavXq1tmzZooMHD+q3335TcHCwOnfurD59+mj48OGy2WzlXnz69OnaunWr4uLi1LhxYyUnJ2vChAlKSEhQx44dSx339NNPq27duho0aJCuueYaHThwQAkJCfr888+VmJhY7l8cAAAAAH9VZmCXpKCgII0dO1Zjx4712ML79u3Txo0bNWPGDI0bN06SNHjwYPXv31+zZ8/WypUrSx3797//XTfeeGOxtrZt2yo+Pl4bN27U0KFDPVYnAAAA4Esu97BLUk5OjrKzs10ek52drZycnHItvHnzZlksFg0fPryoLSgoSLGxsdqzZ49OnjxZ6thLw7ok9e7dW5J0+PDhctUBAAAA+DOXgf3HH39Uly5dtHDhQpeTLFq0SF26dNHRo0fdXjg1NVVNmjQx3Ls9JiZGTqdTqampbs8lSb/99pskKSIiolzjAAAAAH/mMrCvWrVKERERmjJlistJHnnkEdWuXVvvvvuu2wvb7XbVrVvX0B4ZGSlJLs+wl2Tx4sUym83cWhKopELDrYqMDDX8Fxpu9XVpfik03FbK81X+64kAAP7N5R72nTt36s4775TV6voHZlBQkPr27ev2myxJUm5uriwWS4lzSdL58+fdnmv9+vVau3atJk2apKioKLfHeUJk5OXflYaxV25uxvr/WJslSKPX9TW0rxy8WbbIy7+Q3JefdxWZ2515+yW9bujbOPRhr9Xk68frb2OrYl3empex/j/WE+Mvd15//Jz1t3VdBvbjx49rzJgxbk3UtGlTrVmzxu2FbTab8vLyDO0Xg7q7d3r56quv9NRTT+nWW2/VY4895vb6nmK3Z7nsd/Wi2e1Zl/2iVsWxFZ3bFV4Hz411pTI+z958TBWZ25Wq+FxWZF1fjfXXr4ey6nKlMn4N++tYV/zx86qia5c1r6++d3iLN1+HkrjcElNYWKiAgDKvS70wUUCACgsL3V44MjKyxG0vdrtdkkrcLnOp/fv36+GHH1aLFi306quvymw2u70+AAAAUBm4TOORkZE6dOiQWxMdOnSoaP+5O1q2bKmffvrJcAeab775pqjflaNHj+rBBx9U7dq1tXDhQtWsWdPttQEAAIDKwmVg79y5szZs2ODWbR03bNigG264we2F+/btq7y8vGLbaBwOh5KSktSpUyfVq1dPkpSWlma4VaPdbtf48eNlMpn01ltvqXbt2m6vCwAAAFQmLvewjx49WikpKZoyZYpeffVVhYeHG47JzMzU448/rtOnT7u9312S2rdvr759+2r27Nmy2+2KiopScnKy0tLSNHPmzKLj4uPjtWvXLh04cKCo7cEHH9SxY8f04IMPas+ePdqzZ09RX1RUlMt3SQUAAAAqE5eBvV27dpo8ebLmzZun22+/XX369FGLFi0UEhKi7Oxspaamatu2bTp79qweffRRtWnTplyLz5o1S3PmzFFKSooyMzPVokULLVq0SNdff73Lcfv375ckvfnmm4a+IUOGENgBAABQZbgM7JI0ZcoUXX311ZozZ46Sk5MlSSaTSU6nU5J01VVXacaMGRo2bFi5Fw8KClJ8fLzi4+NLPSYhIcHQ9vuz7QAAAEBVVmZgl6TY2FgNGjRIX3/9tQ4ePKizZ88qJCREzZs3V6dOnUq8nzoAAACAinMrsEuSxWLRjTfeqBtvvNGb9QAAAAD4Hfdusg4AAADAJ1yeYY+LiyvXZCaTScuWLatQQQAAAAD+x2Vg37VrlwIDA93eo24ymTxSFAAAAIALXAb2wMAL3d26ddPQoUN12223KSCAXTQAAADAleIyfX/22Wd64okndPToUU2ZMkW33HKLXnrpJf34449Xqj4AAACgWnMZ2GvXrq3x48dr/fr1eu+999SrVy+tXr1a/fr108iRI7VmzRplZ2dfqVoBAACAasft/S0xMTF69tln9c9//lMvvviiatSooWeeeUbdu3dXSkqKN2sEAAAAqi2378N+UVBQkAYOHKgGDRooICBAO3bs0LFjx7xRGwAAAFDtlSuwnzx5UuvWrVNSUpKOHDmiunXratKkSRo2bJi36gMAAACqtTIDe15enj766CMlJSVp+/btCggIUK9evTRjxgz16NGDu8YAAAAAXuQysD/33HNav369zpw5o+joaMXHx2vgwIEKDw+/UvUBAAAA1ZrLwL5ixQrZbDb169dPbdq0UUFBgZKTk0s93mQyady4cZ6uEQAAAKi2ytwSk5ubqw0bNmjDhg1lTkZgBwAAADzLZWBfvnz5laoDAAAAQAlcBvYuXbpcqToAAAAAlIBbvAAAAAB+jMAOAAAA+DECOwAAAODHCOwAAACAHyOwAwAAAH6MwA4AAAD4MQI7AAAA4McI7AAAAIAfI7ADAAAAfozADgAAAPgxAjsAAADgxwjsAAAAgB8jsAMAAAB+jMAOAAAA+DECOwAAAODHCOwAAACAHyOwAwAAAH6MwA4AAAD4MQI7AAAA4McI7AAAAIAfI7ADAAAAfozADgAAAPgxAjsAAADgxwjsAAAAgB/zaWB3OBx66aWX1L17d8XExGjEiBHauXOnW2NPnDihxx57TJ07d1anTp30yCOP6NixY16uGAAAALiyfBrYp0+frmXLlmngwIF66qmnFBAQoAkTJmjv3r0ux2VnZysuLk579uzRQw89pD/84Q/64YcfFBcXp8zMzCtUPQAAAOB9gb5aeN++fdq4caNmzJihcePGSZIGDx6s/v37a/bs2Vq5cmWpY9955x0dOXJESUlJat26tSSpR48eGjBggN5++2099thjV+IhAAAAAF7nszPsmzdvlsVi0fDhw4vagoKCFBsbqz179ujkyZOljt2yZYs6dOhQFNYlqWnTpuratas2bdrk1boBAACAK8lngT01NVVNmjRRcHBwsfaYmBg5nU6lpqaWOK6wsFAHDhxQ27ZtDX3t2rXTzz//rHPnznmlZgAAAOBK81lgt9vtqlu3rqE9MjJSkko9w56RkSGHw1F03KVjnU6n7Ha7Z4sFAAAAfMTkdDqdvli4d+/eatasmd54441i7ceOHVPv3r319NNPa8yYMYZx//nPf3Trrbdq+vTpuv/++4v1rV27Vk899ZTWr1+v6OjoctXjzC+QKdBcYrvklCnQuN3fmZ9fYrs7x1xsd9V/YV1LCX15MgVaiv4tqf/CWGsJfQ6ZAq0qzHcooIT+wnyHJJXaF+DFsWX1OyWZS+gryHfIHGgt+rekfvnp2PwChwLNxv78ggtjS+sLNPtubF6BQ5YS+vP+pInDrwAAIABJREFUO7a0PksZY52SrCX0OQocspqtRf+W1C+ZZDUbvxYcBXmymi1F/15Ov6QyxubLajZ+DZfW7s4xjoL8/65b+rzeG2uS1Wz8XugoKJDVbC76t7R+V1yNvVDz5a3rq7FlP95CWc3G82IX2131X6jLO2PzCpyymE2G/ryCC3GgtD6L2aT8AqcCS+jP/+/Y0voCyxhrkmQuoa+gwCmz2VT0b0n98uLYwgKnAkroLyxwSk4pILCEvnxnie3uHHOx3VW/5HpdZ75TphL6nf8dW1Lfxf7S+v53TKFMgcbPLWd+4X/nLrnPFBjgcmxJ7e6v6yw1v5XU7s4xnhhbkblL4rOLTm02m/Ly8gzt58+fl3RhP3tJLrY7HI5Sx9pstnLXk555ToWFPvndpQy5ZbSX1i9J58toL63fV2MrNndkZKi2vHW3oefOBz6Q3Z7ltXWr29jIyFA9s7qvoefZEZt9/Dxf7tdKriIjQ3X3uhmGng8Gz/zvYyprbsB/REaGamTSYUP7e0Obym7PUuT/z96Zh1VVrY//c84BFGSWUUQF5ICoKag4m0POQ2QqXcQhM9OrXlMry6zsZmZeykpNTc15NsixzCnNeUBREgdQERSQQebhAGf//uC31z2Hg3f4Pd/vL7/fuz7P4/Pgfs/ae+13v+td717rXWu7O/BG7AML+arhTf6vvUv+aNzdHbi1PMvieNBUT/mM/gtxd3fg8bKfLY57TBvw36ZnrVZDw4b2/365/4a6/Eu4u7vXmfaiprPUlS4D4OzsjI2NTZ1pL9nZ2Wg0mjrTZSQSiUQikUgkkv+J/GEBe3BwMPfu3aOkpMTseEJCgpDXhVarRa/Xk5iYaCG7du0aTZs2xdbW9r++whKJRCKRSCQSyR/AHxawDxgwgMrKSnbt2iWOGQwGYmNjCQsLw9PTE4BHjx6RkmI+tde/f3+uXr3KjRs3xLG7d+9y7tw5BgywnKqXSCQSiUQikUj+p/KH5bC3adOGAQMGEBMTQ3Z2Nk2aNCEuLo5Hjx7x2Wefid/NmTOHCxcucOvWLXEsKiqKXbt2MWnSJF599VV0Oh3r16/H3d1dfIRJIpFIJBKJRCL538AfFrADLF68mK+++oo9e/ZQUFBAUFAQ3333He3atfuH5ezt7dm0aRMLFy7k22+/xWg00rFjR95//31cXFz+P9VeIpFIJBKJRCL57+cPDdjr1avHnDlzmDNnzlN/s2nTpjqPe3l58c033/x3VU0ikUgkEolEInkm+MNy2CUSiUQikUgkEsk/5w8dYZdI/iupNFTQ/7WDdR6XSCSS/xTKK6vZMTygzuOS/xlUGYwETfWs87jkPxMZsEv+15BfYAAsP6glkUgk/0kU5ZciP63zP5snBSX//EeS/yhkSoxEIpFIJBKJRPIMIwN2iUQikUgkEonkGUYG7BKJRCKRSCQSyTOMDNglEolEIpFIJJJnGBmwSyQSiUQikUgkzzBylxiJRCKRSCQSyX8c1YYqPKYNqPP4s4YM2CUSiUQikUgk/3HkFZT90VX4l5EBu0Qi+Y+nvNLAwYjP6jwukfxvo6KymlXDm9R5XCKRPJvIgF0ikfzHU5RfQRHyi7iS/wwK80v/6CpIJJJ/E7noVCKRSCQSiUQieYaRAbtEIpFIJBKJRPIMo1EURfmjK/EskJtbjNEoVSGR/DMcnW2oZ13P4nhFZQWF+TLnWyKRSCSSp6HVamjY0P7fLidz2CUSyb9FTVAuA3OJRCKRSP5/IVNiJBKJRCKRSCSSZxgZsEskEolEIpFIJM8wMmCXSCQSiUQikUieYWTALpFIJBKJRCKRPMPIgF0ikUgkEolEInmGkQG7RCKRSCQSiUTyDCMDdolEIpFIJBKJ5BlGBuwSiUQikUgkEskzjAzYJRKJRCKRSCSSZxgZsEskEolEIpFIJM8wMmCXSCQSiUQikUieYWTALpFIJBKJRCKRPMNY/dEVeFbQajV/dBUkEolEIpFIJP+L+X+NNzWKoij/xXWRSCQSiUQikUgk/0XIlBiJRCKRSCQSieQZRgbsEolEIpFIJBLJM4wM2CUSiUQikUgkkmcYGbBLJBKJRCKRSCTPMDJgl0gkEolEIpFInmFkwC6RSCQSiUQikTzDyIBdIpFIJBKJRCJ5hpEBu0QikUgkEolE8gwjA3aJRCKRSCQSieQZRgbsEolEIpFIJBLJM4zVH12BZwWDwcDXX3/Nnj17KCwsJDg4mJkzZ9K5c2ceP37Mxo0bSUhIIDExkdLSUjZu3EjHjh25du0acXFxnD9/nkePHuHs7ExoaChvvvkmhYWFrFy5khs3bpCbm4uDgwPBwcFMnTqVsLAwizqsXr2amJgYgoODmTt3LmPHjq2zrgcPHiQgIACAa9eusWzZMq5cuUJVVRW+vr6MHz+eCxcuEBcX99T73bJlC5s3byY+Pp7CwkIaNWpEREQE48eP58aNGyxZsoSEhASqqqpo0KABFRUVlJWVifsGzPSSkJBARUUFVlZWuLu7M3DgQDQaDdevX+fq1asYDAYaNmxIbm4uL730ErNmzeK7777j119/5eHDhxiNRmxtbQkKCuKll14iPT2dhIQE4uPjqaqqwtbWFo1Gg6+vL/3796ekpITr16+L5/Hll18yd+5cysvLGT58OOnp6Vy8eBFFUSzuvU2bNtSrV0+UbdiwIYWFhTg7O+Po6IiNjQ1JSUlP1Z2Pjw+PHz+msrISa2trGjZsSPPmzXF2diYpKYmHDx+i1Wqprq7GaDTi7+9P//79ycnJ4fz586Snp2M0GqmursbJyYlOnTrh4ODA5cuXSUtLQ6vVotVqKS0tJTw8nLfffptdu3Zx7Ngx8vLyANBoNPj5+dGrVy+Ki4u5cOECqampaDQaIW/cuDHt27cH4PLly8I+W7duzcWLF3ny5Ak9evTg4cOH3Lt3D6PRaHGvTZo0wdramkePHqHT6TAajRgMBuzt7bG3twcgPT39qbpycHCgvLwcRVFQFAUnJyf8/PzQ6XSkp6eTm5uLVqvFaDQK++3VqxdpaWncuHGD7OxsFEXBaDTi4OBA165dKSsr49atW2RnZ6PVatFoNJSXlxMeHs4777zD8uXLiY+Pp6ioCEVR0Gg0NGnShL59+3L37l2SkpLIysoy05Wvry/t27cnOzubW7duifYaGBjIjRs3KCgooE2bNmRnZ5ORkVGnXXl6eqLT6cjNzRW6qqysxMHBAWtrawCys7Ofqis7OzsMBoPQlaOjI6GhoUydOpXWrVtjMBhYvnw5W7dupbCwEGtra9555x3GjBkj7iU+Pp7Zs2eL5xUVFcXs2bOxtbXl4MGDfPfdd8K2w8PD2bRpE2VlZcTGxnLkyBESEhIoKSmhXr16vPfee4waNQqdTseSJUuIi4sTevPz82Pw4MFMmDABOzs74O/+S6fTUV1dzfLly3nhhRcYM2YMFy5csLjfQYMGsWTJEgwGA5MnT+b06dMAuLu706ZNG6ZNm0ZERMRT9TVixAhatWrFsmXLyMnJQafT0bp1a65evVrn71X/UxfOzs506NCBw4cP1ylftGgR7777bp2ygIAAunXrxoYNG+qUv/DCCxw5cuSp183Pz69T9s+wsrKiQYMGFBQU/NtlXVxcaNu2LVOnTqWyspI5c+aQlpaGoihYW1vTv39/YmJiuH79OsuWLePChQuUl5cD4OjoyNixY5k8eTJfffUVP/74Izk5OcJm33vvPYYPH87p06dZsGABqampVFdXo9Vq8fHxYfbs2QwYMIBp06Zx9uxZSkpKAKhXrx59+/Zl0aJFJCUlmfVrXl5ePHjwgKqqKn788Udef/31OtvS66+/Tr9+/Vi2bBnx8fGUlZUBoCgK7u7uNGvWjHPnzj1VL2+++Sbh4eG8//77pKamYjQasbKywsHBgSdPnjy13HPPPce1a9f+7ecAMGDAAH7++ef/p7LNmzcnOTn5/6nsfyd//etfOXv2LJcvXxZ9ltFoxNfXl+joaLp3787XX3/N+fPnyc/PR1EUbG1tiYiIIDIykpUrV3LmzBmKi4vRarVUVlbSvn17tmzZws2bN3nvvfdITk7GYDCg0Whwd3dn0qRJdOnShaVLl3Ly5ElKS0vRaDTC/w8ZMoQ+ffqwatUqEfd4eXmRlZVFaWkpH330ERcuXODIkSNUVlZa3NPzzz+PnZ2duCeNRkN1dTWurq7o9Xo0Go3wYXWh+qtt27aRlpZGgwYNaNWqlfDv/woyYP+/vPvuu/zyyy+MHTuWpk2bEhcXx+uvv86mTZswGAysXr2apk2bEhQUxJUrV0S5NWvWEB8fz4ABAwgKCiI7O5stW7YQERHB9OnTqa6uZuTIkbi7u1NUVMS+ffuIjo5m9erVdO3aVZwnOzubFStWiM5PZdy4cbRs2dLsmKenJwAnTpxg6tSphIeHM2PGDKysrLh//z4ZGRlERkbSuXNns3KKojB//nw8PT2ZMmUKDg4OREdH4+TkxKVLl/jiiy+4cOEC586dw8fHh4iICLZt20ZRUZEICEy5d+8eq1evxsPDg4qKCgDGjh1LZWUl69atQ1EUmjZtKn7v7e1NYWGhKLtp0yYAnJycKCgoYOTIkdy6dYuPPvoIqAkWbWxsqKqqYuTIkQQEBHD16lW++eYbALPnsW3bNrTamgmj2NhYmjZtirW1NQaDgTfeeEO84KSkpLBq1Sp8fX1FgNqjRw/at29PfHw8P/zwA40bN6Zp06akpqaald24cSOJiYnk5uaKBj1mzBjs7OxYtWoVVVVVDB8+nPLycjIyMqhXrx6KouDi4sI333yDg4MDPXr0IDU1FVdXV0pKSiguLubnn3/G2tqayMhIfvvtNx4+fEh1dTUAZWVlrFmzhgsXLvDkyRMCAgLw9vbm6tWr3Lt3j+TkZOrXr8/IkSPRaDQ0aNCA5ORkKisrCQkJYdeuXVhZWREZGUlwcDDZ2dl89913oiO7evUqL774ItXV1eTm5qIoCgaDgenTp/PLL7+QlpbGoEGDOHv2LBkZGWg0GrRaLU5OTmRkZDBw4EA6duxIcXExp0+fFmW3bt1KRkYGjo6OFBUVERwcTHp6OkVFRdy5c4eioiJGjx7N5cuXuXnzprABHx8f1q1bR7NmzejZsyc7d+6kYcOGaDQasrKy+Pnnn3FxcSE6Opq9e/eSnp4uguGCggLS0tIoLi6moKCAgIAAfH19uXXrFqmpqaxevRovLy8iIyM5cOAAjo6OPHz4kOzsbJo1a8YPP/yAg4MD48aNE+11zZo1IiBS2/LevXvJzc3F3t6ezMxMJkyYQGlpKUlJSXTt2pUDBw6QkZFBgwYNePLkCUFBQSQnJzNkyBBKSkooKyvjypUromx8fDzx8fF4enpy7949QkJCePLkCZmZmVy7do2oqChiY2P56quvOHr0KDqdDmtra+zs7Pj0008pLCxk2rRpJCUlMW7cOKqqqrC2tsbR0ZEdO3aQnp7OypUr2bBhA0lJSaKdqKSlpfHJJ5/Qrl07KisrsbGxwdramvnz53P9+nUWLlxIfHw8eXl54gW1c+fOrFq1ivPnz7Nx40ZycnJYsWIFVlZWwnZV1E5VLTtz5kyg5sXXYDAwduxYrly5gpWVFW5ubkRHRxMfH4+trS0ffPABixcvRlEUUfa3335j37595ObmMn/+fKysrLCxscHJyUm8PL744oskJydz48YNwsPDadSoERkZGeTm5tK2bVt69uzJsmXL8PT0JDw8HBcXFzZu3Cj865MnT2jcuDGZmZl4eXnh4uICwLBhw9i7dy+BgYEEBgZSv359njx5IoL1Ro0a0axZM9zc3LC2tiYjI4Pjx48Dlr58//79XLx4Ufzf09OTiooKBgwYII7Z2tqybt06xo0bR1BQEGvXriUjI4OOHTvSrVs3bt26xc6dOwkPD8ff399M7ykpKVy8eJHnn38eRVE4efIkoaGh+Pv74+3tTVxcHK+88grV1dUoikJoaCg+Pj7cvHmT/fv3o9Fo+Pnnn2nevDllZWX4+/vj5eVFSUkJy5cvJzExkePHj2NlZYWfnx/379/Hw8ODjIwMTpw4wZQpUzAajQQGBuLv709+fj7p6em8+eabDB06lKNHj+Lt7U337t2xsrIS183JyeHy5ctm/dr3338v+qDLly+TnZ1NgwYNeOGFF9DpdGRnZ2Nra4uXlxdRUVGEhYVRv359NBoN/v7+NGzYkNatW3PgwAF0Oh3+/v6EhISIsqmpqTx48AA7OztGjx4NQFhYGD4+Pjx48IAbN25gZ2fHrFmzWL9+PRkZGXTp0oVz585hY2PDtWvXiIyMxNvb28yu9u3bR1VVFS1atODVV19l5cqVpKen07hxY+7evYutrS3jx48nJCSEL7/80syufvzxR4xGIx06dGDkyJGsWbMGNzc33Nzc+Omnn7Czs+P+/fv4+/szefJkM7s6efKkmc2tXLmSvLw8+vfvT2xsLK6urowYMUIE0apdhYeHc/r0aWxtbWncuDFDhgzhwYMHQE38oJZt3rw5p0+fpkWLFiQlJREaGoqfnx979+5FURQ+/PBDPDw8cHR0JDs7myZNmogXoE8//ZR69erh6OhIQUEBnp6euLu7c+3aNbZt28bOnTvx9PTEwcGBsrIyHBwcyM3N5d69e2RlZREVFUVJSQlNmjQhODhY+MkFCxZgbW2Nh4cHbm5uwlYTExNRFIWVK1fy7bff4unpKeKeDRs2UFpaCsDixYtxdXXF09OToqIiAgICiI+PJywsjIEDB7JkyRKcnZ2pV68e1tbW+Pj4kJycjIuLCxcuXKBhw4YMHDgQW1tb7t+/L8r6+PiY+athw4YxevRoCgoK2L59u/DvgYGB/FMUiZKQkKDo9Xpl3bp14lh5ebnywgsvKFFRUUpRUZGSl5enKIqiHD58WNHr9cq5c+cURVGUy5cvKxUVFWbnu3fvntKqVStlzpw5FtcqLS1VunTpokyaNMns+Jw5c5QxY8Yo0dHRyrBhw5Rz584per1eOXz4cJ11LiwsVDp37qx88skn//J9Xrx4UdHr9cqECRMUvV6v3L5920w+ffp0JSgoSAkPD1fy8/PFfWdlZSmtWrUyu29FUYR80KBBSp8+fczkixYtUoKCgpR79+4p27dvF7J27dopc+bMUYqKipTr168r6enpZjo1Go3K6NGjldatWytlZWUW+lYURfnwww8VvV6v5ObmCnmLFi2UL7/80uy3nTp1emqdP/jgA6Vz585m8n/0nBVFUfr06aP07NlT0ev1ypQpU8zkq1evVvR6vahbXFycmR1ER0crnTp1UiZMmKB0795dKS4uFvLBgwcrer1eOXPmjPLo0SOlqqpKuXfvnhIUFKR06dJFuXz5spKZmWn2vNSy3bt3V/R6vfLgwQML2Zw5c5S//vWvil6vV65du6YoiqLcvXtXCQkJUVq0aKHo9Xrlp59+UhRFEXZnWla17ZUrVyrt27dXHjx4IOQTJ078h3Y/fPhwpW3btkqbNm2U6dOnm8knT56s6PV65b333lP0er2ydOlSs3axaNEipVWrVsrYsWOFrlT50KFDLXRVWlqqtGjRQunQoYOiKIqSm5trpiu1bI8ePSx0ZXrd77//vk5dtW3b1qwtqrqqqy2b6uppbb32dceNG6eEhoaa6UqVR0VFmelq2LBhZn5C1dXjx4+ViRMnKs8995wSFRUl5Dt37hT6mj59uijbunVrJTo62kxftX3Qu+++K/RVW6Yoipm+5syZo4wcOVLYrKm+unTporRt29asrKm+WrZsqYwaNapOeV3XNdVXz549zeS7d+9W9Hq9MnbsWGFbKqpPDQkJMbMtlUWLFil6vV5ZuHChUlVVpSiKogwbNkyJjo4WZWNjYy38pqIoymuvvabo9Xpl27ZtFrKPPvpI0ev1ypo1a8Sxu3fvKi1btlSWLl2q6PV6Ra/XK4MHD7a4f9N+wNS26pLXJiIiQtHr9cr+/fuVtm3bCttSiY+PF9c21ZOiKMJvzJ07Vxk0aJDy0ksvCZ0oiqJ89tlnil6vV6ZNm6YYjUZFURTh29W+ac6cOUp6errZeY1Go7Dp+fPnW9T5gw8+UPR6vTJv3jyze2zZsqXw7x06dFBCQ0OVKVOmmJU17RM/+OADpXfv3sKf15bXpm/fvsoLL7ygdOzYUdHr9cqiRYvM5MeOHbPw72p/umLFCmX69OlK586dzfy7Kv/zn/9s4bNUWe/evRVFsfRZqlx9hqbP3PS6dfks1bfX5bNMy6qY2lVd8rquW5fPUuWRkZGKXq9Xpk6damZb06dPV0JCQpQ//elPil6vV0aMGGHWBtX4Q6/XK7t27RK6UpQa2woKClJWrFhRp71Pnz5d3PPJkyctZCEhIcrEiRMVvV6vHDhwQOiqZcuWysCBA0XZ27dvm/katax63QULFpi1wenTpyvBwcFPjadCQkKUMWPGWOhK5datW4per1e+/vprC33XhcxhBzHCOXLkSHGsXr16jBgxgsuXL1NaWipGWGoTFhaGjY2N2bFmzZoRGBhISkqKxe9tbW1xdXUVI81Qk9ayd+9e3nvvvTqvUVxcTFVVldmxffv2UVhYyIwZM8RvlDqm6U1RR02aNGkC1EwRm+Lm5oaiKHTt2hUnJyfs7e1xcXHBw8Ojzrc/e3t7cnNzSU5Oplu3bmay8ePHoygKv/zyi8V11LKtWrXCx8fH7LhGo6F///5UVFTw8OHDOu9DHbUvKioSo3l9+/YVxx0dHc1+X1VVJUaU7e3t0el0xMXF8cILLwBQWVkp0jye9pyvXbtGWloaffr0qfMaappQeno6Go2GgQMHmtnB6NGjycvL4+zZs0RERNCgQQMhr1+/PnZ2dvz00094e3uj0+lo1qwZNjY2lJWVERYWhqenp9kzUMva2toCcPfuXQtZSkoK4eHhQlcAn332Gb1798bX1xeomVY3pXHjxvj7+5OSkkJYWBhWVlZs2rSJUaNG4evrS6NGjQgICCA/P/+pdn/z5k1u3LhBv379KC8vx83NzUyemZkJQE5ODgCDBw82axeDBg3CYDBw4cIFoStVbmdnZ6ErW1tbs5FdV1dXM12pZRs0aGChK9PrNmrUqE5dubq61mkT1tbWODs7i7ZsNBrNdKXT6czkpqjXzc3N5fz58/Tt29dMV6rcYDCY6er27dtmfkLV1YEDBzh9+jQVFRXMmzdPyF988UXs7OzYvHkzR44cqdPHuLq6UlZWZuGD+vbtC8DRo0fr9E+qvhISEti7dy9WVlY4OzubzRJeu3aNnJwcPDw8RGqTitFo5Pvvv6eqqor58+djNBrNfFhdfvHx48dCX2VlZWRkZJjJnZycgL8/Q9PRapWqqirOnz/PsGHDhE0AwofduXOHsrIyC58LNSle3t7eFjI19SwzM9PCX6vPtKysTMg+++wzevXqRYcOHczOU11dTXl5uUgTMdXVhg0bGDFiBL6+vhgMBjGrqWJ63cePH4vUJzW9rrYdX758Wfw9ePBgsz5E9SvOzs4kJycTERFhNjOj+kk1FcC0rNo3vf/++/j4+JjJ1PQFgJdeeknUW5WraSd/+tOfACgsLGTBggVER0cL/15SUiL875MnT8QIqXrdCRMmEBcXR3R0NM7OzlRUVGAwGJ7aZ167do3U1FT8/f0pLi4Wz8y0XuozNPXvan86ZMgQBg0aRG5urpl/V+VvvfWWhc/av38/8Pc+uLbPUstGRUUB5j7L9Lp1+Sxvb+86Z8QB9u7dK8qCpc+qLTdFvW7nzp3r9FmqXJ3pUW148ODBQodWVlYi/rh+/brQlSpXbezixYtCVwA6nQ6NRiOede2UYtOytVPE1OvWq1dPnEvVVa9evfDz8xO/NY1VqqqqcHJywsrKSlz3wIEDQlcGgwEXFxeh67riKZ1Ox4ULFyx0ZfobgPr161vouy5kSgyQlJSEn5+fmfOGmrw0RVFISkrCw8PjXz6foijk5OQQHBwM1DgGg8FAfn4+P/74I7dv32bq1Knit5988gkRERG0aNHC4lxvv/02paWlWFlZ0bFjR+bMmUNQUBBnz57F39+fEydO8Le//Y3MzEwcHR2JjIxk5syZwihVKisr+emnnwgNDaVPnz5s3bqV999/nxkzZuDk5MTFixeJi4tDq9XWaTxqcFY73/LGjRsAohGqeHp64uXlxY0bNyymav8ZanDi4uLCvXv3gBqHlJGRQWJiIt9//z2+vr40btyYNWvWABAREfHUHMOJEyeKqb/XX38dd3d3DAYDDg4OALz22mvodDo6derE/PnzLe4FahwdQFRUFEeOHOHo0aMA5OXlcfXqVRYtWkRAQACenp5YWVlhbW1tZgdqB1hdXU2rVq0AczupnTevKArV1dVPbchqWTVAUTvQ6upq8vPzefz4MV5eXnz11Vc4ODjQqlUrTpw4wZkzZzhw4ACRkZEW50xJSaFNmzZUVVVhY2PDypUr6dmzJ9nZ2TRt2pS//OUvHD58WOSTJyYminsxrZO9vT1Go5EXX3yR1NRU4uLiaNu2Le3btxfBuru7O35+fpw4cYLHjx8TFxcn2oWqK6PRSLNmzbh7965Zu9FqtSQmJpKXlyfaVEVFhZnDrKvN9ezZk5SUFOrXr09eXh55eXns2LGD27dvM3ToUKErPz8/9u/fz+nTpxkxYoRZjn5xcTGVlZUkJyfTunVrjEYjdnZ2Zrpyc3Pjtdde4/Tp0yiKgqenJ4mJiTRr1syiTl27dhW6unv3LrGxsTRq1IhHjx5x+/ZtmjVrhru7O3q9nhMnTjBgwAAzP6Hq6ty5c1RXV9OpUyczuY2NDcHBwZw+ffqpPuZpPkhth7t27SIiIgK9Xk9lZSVVVVWcOnWKr776Cnt7e2JjY+nUqRMXLlygRYsWwkeo53Vzc+Phw4ekpqaiKArdunUjOjqa559/nvz8fMLCwlixYgWXLl0CYNSoUXzwwQd11ungwYOpQLjIAAAgAElEQVQYjUaGDRvGL7/8QkVFBXfu3KGiooKKigrhD27fvg3UpLB06tSJOXPmWNzz2rVruXHjhvCp6svr2bNnadeunUi1MfVfT/PHapCwbt06VqxYgU6nIzQ0lKFDh7J7924A1q5dy9KlS9FqtSiKwqpVqyzan9FopE2bNkBNm1YHCN555x3KyspYv349e/bs4cmTJxiNRpGuV7tewcHBIticO3cuANu2bePSpUu89957NGzYkPXr14v1I2PGjCE7O1v0IYmJiQAinefTTz9l6dKlon9JTEzEysqK3377jd27d4t2nZiYSHFx8T/sm27evAnU9B3Tpk0jKysLe3t7OnfuzIkTJ7C2tiY5OZkpU6aQmZmJRqOhU6dO4mWvcePGFBQUcPToUeGHHRwcxGCDmsa6aNEi/va3v4n8eRcXF3x9fS3qpQ5eVFVV4efnx+PHj4mJiWHx4sXY29vTp08fUlNTzfw7IPrTxo0bi8E51b+b9rd+fn4idQT+3hfb2tqKINIU07LqM1T9e3l5OQcPHqRVq1bcv3/fwr+fPn0aOzs7mjdvzp07d8zOm5ycLHT/yiuvEB0dbebfp02bJtZwzJo1iw8//FD4d9M6XblyxcK/q+lGwcHBIqdeHXArLCxk7969Is3Yw8ODuLg4FEXB2dmZjIwMEX8MGTKEPXv2cOzYMZEqefHiRQoKCmjUqBHh4eGsWrWqztildevWJCQksG3bNpo3b469vT2//fYbsbGx9OvXT6yj2b17N9nZ2Zw+fZq33nqLJUuWCB29//77lJaWkpyczHPPPUd1dTV2dnZivURubi42Nja8/vrr/Pbbb+Ie8vPz66xThw4dOHXqlEVf2KFDBwoKCvj6669xd3f/h2t1TJEBOzX542peuCnqSMDjx4//rfPt3buXrKwskas5d+5cDh06BNSMyr3yyisi5+zHH38kOTmZ5cuXm51DXfjTo0cPXFxcuHXrFt9//z1RUVHs3r2b1NRUMjMzeffdd5k4cSIhISEcP36c1atXU1FRwfvvv292vlOnTpGfn8/QoUPp1q0bM2bMYNWqVRw7dkz85i9/+Qs///wzV69exWg0ijdWg8HA/fv3ASyCYtWQa484q/r7d3WXn5/Prl27CA8PNxsRUl9wAFq1asVnn31GUVGRGKmws7OzqFujRo3Iy8tj+vTpuLu7s3PnTj788EN69uwJwNatW8W5HR0dWbZsGePGjWPfvn1m56muruann37iueeew9/fn2+++YYpU6YANYuUANq2bcvmzZvZs2cPlZWVYtRGtQM1GFH1AuZ28uuvv5otltu7dy9VVVVPHd1VyyqKQpMmTYRjTUlJYejQoUDNs/Hz8+Pbb7/F1taWhQsXMmbMGK5evWqhK19fXzp27EhBQQEbN26kadOmLFmyRDi5L774Al9fX0aNGsX27dvRaDSMGzeOvXv3ilkStU7qwuNOnTrh4+PDzJkzefvtt8W13Nzc2LZtG2+99RZQs+7BtF3ExsaK36qjp6bylJQUjh8/LtZoqCPd6mgTWLa5UaNGcfnyZZo0acLmzZv55ZdfzO5/3759Qleffvqp6Lh++OEHevfuLdrJ3LlzxRoWKysrOnToIBZlqrpatGgRUDOSExYWRmpqKuPGjSMsLEzklqr3k5CQIHTl6OhIWVkZf/vb30S9FEVh27ZtrF+/HkDMmKiodqUGqOPGjbOwlcrKSsrLy4Wt1qYuH6QoChs2bMDV1ZVHjx6xbt06UlJSxL2/9tpr+Pn5ERUVxebNm3F1dWXMmDEi0AM4f/48ycnJ9OzZk4CAAH766SdKSkoICAhgyZIlou2qay78/f0pLS0lMzOT0aNHo9VqLfzi3r17cXd3F7YfEBBgZlteXl706NGDxo0bs3XrVvr168epU6eIiorir3/9KyEhIWKQYcSIEfz000/Cp1pbW+Pp6UlZWRmLFi3i1q1bLF26lKSkJHJycp7qj7dt28ahQ4ewtbXl/fffp7y8nAULFnDp0iUuXbqEu7s7HTp0ICIiAkdHRz744ANKS0uZNWsWH3zwAQChoaFotVq8vLzQaDQcOXKEwsJCdu/eTePGjXn++efZsmULNjY2PHnyBBsbGyZNmsSmTZvQ6XRMnToVPz8/Ua+zZ8/i4OBAly5d6NGjB1VVVSxfvpw7d+4wYcIEYYM2NjaUl5fTtm1bhg0bJvoQdZZEDUQXLFhAfHy86F/Ul69r164xefJkQkJCmDlzJnfu3CEnJwej0Vhn31RYWMiDBw/Q6XQsWLCAvn37cvDgQYqLizl8+LCYMZs7dy7R0dHs2LGDwMBANm7cSJcuXYCaF8mSkhI6d+5Mq1atuHz5MvHx8SQlJVG/fn0RMNrZ2WFvb8/jx48JCwsT6zBM63Xs2DHi4uJwc3MjOzubrKwsSkpKsLe3p7CwkOLiYvbs2YObmxv79u0T/n3z5s2iPzVth1Dj3037W/WY6t9VWV0xh6l80KBBbNiwwcy/7969m4KCAq5fvy7aoKl/79mzJ0eOHOGll14yC9h9fX3x8PBg//79REREkJOTY+Hf1cGfIUOGcPHiRTP/bno/u3fvtvDv6oLswsJCmjVrxvjx49myZQtQ8wIONfHF1KlT2b59u6jX559/zueff24mv3HjBnfu3OHFF18Uv3Nzc8PHx+epscvUqVM5dOgQTk5OJCYmmpVV/Yafnx8vv/wyBw4cEH540aJFDB8+nNjYWIYMGcKRI0fE4mpAzOTFxsbSpEkTHjx4wLfffivkvXv35vfff8fGxobTp09bxFNHjx79h31hs2bN2LZt2788ICwDdmreWtWFa6aob7+1px7/ESkpKfz1r3+lXbt2wmimTp1KZGQkmZmZ7NmzB4PBINIwvvjiCyZNmmTxwMLCwsymffr06UPv3r15+eWXWbZsGaWlpRQUFDB79mwmTZoEQL9+/SgtLWXbtm1MmTLFLNjbv38/1tbWDBw4EKgZpQgPD6dv3744Ozvz66+/snTpUgYNGsSBAweYN28eEyZMwGg0smLFCjGtr07Rm+oOeKr+1FSUfwVFUXjrrbcoKioym9YHmDNnDj4+Ppw7d46kpCRKS0vFQk51uqo2U6ZMYerUqbRr146OHTsyfPhwoqKiOHXqFFAzfV5aWkrHjh3p2LEjfn5+TJo0iR9++MEsVefs2bPk5OTwxhtvADUvJ40bNyYnJ4cZM2ZQv359Vq1axYwZM/j8889Zvnw5s2bNIi8vj1atWlFeXi5eDqBm1LO2nZw9e1boUpXZ2tpaTKGZyt3c3Hj8+DFr1qwRL1eVlZXY2trSqFEjevfuzZkzZygpKWHjxo0UFBQwcOBAXn31VfR6vQjyABYuXEhKSgqjRo2iXbt2bN68mZkzZ4rdCzQaDfPnz2fcuHG0a9eODz/8kIiICDZs2MDcuXNFnVq2bMnvv//O+PHj0Wq12NvbExgYSEBAAL/88gseHh5UVlYyefJk3n//fd566y0MBgPe3t7k5+dz4MABlixZIkb+PvzwQ+rXr2/WburVq4eNjQ0rVqwQbermzZtm6RS129yFCxe4f/8+a9aswcPDg1deeYUHDx6wfft2nJ2dadGiBRcuXKCkpITGjRtjb2/PzJkz+eWXX8x2DKh9Xh8fH+bNm8fcuXOFruzt7Zk9ezY///wzPj4+vP3220RGRuLs7Mz3338vyubl5ZGUlCR09cYbb4i0kNzcXOzs7MjMzGTSpEnk5+fj6OjI8uXLcXd3p6KigsLCQpYsWYJOpxMjnLWDgOLiYm7fvk29evXq7BSKi4vr9EFZWVnk5+fj5OQkZPb29gQFBVFcXMzgwYM5efIk27dvp127diQmJor2prJnzx4mTZokXm7Pnz+PlZUVa9euZerUqWLnFJ1Ox4YNG5g8eTKFhYV8/PHHREZGEhoaalYng8HA3bt3GT16NF9++SVjxozh8ePHdOzYkbNnz1JUVIRWq+XRo0csWLCAEydOcOnSJaZNm8bixYvZvHkzjx49Erb1yiuvMHr0aOFTv/jiCzp16sSJEyfo06cPffr0Yc+ePaSmpnLs2DGx2B3M/fHUqVN59OgRq1evpnv37pSWlhIQEMDt27dZtGgRBoOB1157jV69erF27Vo0Go1YSPrjjz8CNbOAaooewM2bN3n55Zfx9vbm4cOHeHt7AzUzKuvXryc6Opq7d++yceNGhg0bxuPHj5k4cSJ9+vQhKCiIP//5z7i7u4s65+bmcuXKFTFS2rJlS27cuCF2Mbp69SoRERFMmzaNlJQUEhISgL9vCNC3b19Gjhwp+hcPDw+qq6vx8PAQL4K2trZiITdQZ9+0c+dO0U7ffPNNRowYwciRIykqKiImJoYHDx5QXFzM7NmzxYLfrVu3Mnv2bPESrcrVc0NNcHTo0CHKy8vp2LEj58+fx8fHh7179zJz5kyOHj2Ki4sLubm5dOvWTdTZzs6OuLg48vLyqFevHgUFBUyYMIGHDx/SrFkznnvuOb755htu3brFn//8Z7788kuWL1/O8uXL0el0tGnThh07dlj499r9bb169YR/V2V1DXKZyq9fv05KSgqrV682SxXR6XTCX58+fdrMv6sL0sPDw812LVq4cCGzZ8/G2tqad999FxcXF2bMmGHm31u0aMGjR4+YN28e2dnZDBs2TPh3tU4tW7bk448/tvDvJSUlpKWlMWvWLLZs2cJPP/1EWFgYt27dwmg00rJlS5YuXcqjR484cuQIGo0GRVGYMmUKISEhIv5wdXXFwcEBKysrPvroIxGb/PDDD2JXsrpiF/XFfuLEidy+fZu+fftia2vLnj17OHXqFF27duXJkyc4OjrSqFEjMjMzxa4w6gBRw4YNLc4bGxvLRx99hL29vdCVnZ0dH3/8MefOnSM2NpYpU6bw7bff4uPjw+TJk83iKUVRLHQVFhZGx44dyc7OZvXq1UyePJktW7bg7Oxcpz2YInPYqckfqmsbHzVQr2vaqi6ys7N54403cHJy4uuvvxaNLCgoiK5du/Lyyy+zdu1afv/9d9577z1WrFiBtbU1r7766r90/uDgYDp37sy5c+dEqkTtXLOhQ4dSWVnJ9evXxbGSkhKOHj1Kt27dcHFx4cCBA3z00UcsWLCAUaNG0a9fPxYuXMhLL73E8ePHGT9+PHv37mXw4MEMHTqUBw8eiM6kdoqG+v+n6e9fzc2Cml1YTp06xWeffUZQUJCZrGXLlvTv35+PPvqIPn36MG7cOLZv387w4cP/5fPrdDqxiwbUjGqZ8vzzz+Pk5ER8fLzZ8X379qHT6Rg0aJDY3USdim7Xrh0TJkxg6dKlXLhwQdQ/MzOTsrIyEhMTiYmJESNpUDNjU9tOVF2Z2pCPj49FLqIq12q15OTkMHv2bLp37y5kM2bMwNXVlQ0bNvDWW28xceJEpkyZwrJly3j11VeZNWsWTk5OTJs2rc7zmtZJHYkD6Ny5M2+++aaQBwcHExwcTHx8vFlZNSd36NChVFVVMX78eGxsbLhy5QoNGzZk69atrF+/ntTUVM6fP8+GDRvw9PTk9u3b/Pzzz7z33ntMnTpVjPCFhIRYtJuKigoaNGhg1qbKy8vN8jxN21zXrl25f/8+gYGBdO/eXcj+9Kc/sXv3bp48ecKjR4+Ernbs2MGsWbOIjo5m7dq1Ynap9nlN62Sqq379+hEVFSXka9euJTg4mHv37pmVVXOIVV198sknBAYGsn79emJjYykrK6N58+bcv38fg8HAunXrcHJyYurUqSQkJJCRkcHUqVOxtrY2mw0zZcWKFWg0GjF6Vpu6fFBGRgb5+fl06NCBBg0aCJmdnR1OTk44ODgwe/ZsGjVqRGFhIZcuXWL69OkWAYhOp3uqbzPNc+/Vq5dZOuLhw4exsrKyuBc17aSkpAQrKyuOHTuGk5MT8+bNw9XVFRcXF9atW0dqaipbt25l1apVODk58dlnn1FdXU18fLyZbRkMBjOfCpY+q379+jg6Ota5FWBwcDC+vr48ePCAWbNmiXZoZ2dHly5dGD9+PN26dcNoNPLnP/+Zc+fO8e233zJ9+nTat29P586dxWh/Xefu3LkzpaWlKIpCamqq0FVoaKios16vF+1Q5ffffwcQ2+mp7dDJyYkvv/ySrl27kpGRQePGjYGaXbJUu1JHDFXUIEJ9Fmr/ogbdtV8Q1XOCZd+k3oua3jFkyBBcXV3p0qUL/fv3N0tb8vHxYfv27bz77rtYWVkxdOhQs92Hap972LBh4m81NW7AgAFotVpRZ7UfN11HsW/fPvECp95TXFwc7dq1Y9asWbzwwgv85S9/AeDKlSucOnWKJUuWUFpaSnV1NcOHD2fx4sVm/r2wsNCsv4W/25VpX1w7bRX+3lc3adKEPXv2mNlVSUkJJ06coEePHkRERDB79mwz//7GG29w8uRJunXrJtI9a5/XtE6mPqt79+6cOHFCyE3tyrTsiRMnAHP/bmdnR1ZWFj169GDixImMGzeOrKws9Ho9mzdvxsPDg7Nnz6IoCrt376ZLly5C188//7xZ/LFw4UISEhJwcXExi02cnJx48OABu3btsohd1JQ1nU7Hli1bhGzo0KGsWbOGl156ifj4eFq1asW6det49OgRb731FhERESxcuFCsXdm2bVudMdHixYvNdrsbMGAAw4YNE/I1a9ag0WhwcHAwK6vGML169TJrg/PmzaNv375ERUUJf7Vu3ToLW6gLGbDz9NQNNd3jX5muKCoq4vXXXxdbwalpD7WxtramT58+HDp0iA0bNhAVFUVOTg7p6emkp6dTUVFBZWUl6enpde6v6+3tTUFBgTj/0xYxmJY9cuQIZWVlYnpu69attGzZ0sLZ9u7dm9LSUnr37s3p06fZsmULe/fu5YcffhANrLYu1HrUtbAuOzv738r9P3r0KG+//XadC15MGTBgAOXl5Xh5eeHl5SWupaZ5PH78mIyMjDrLqr+HutN4ai8INhgMHD58mM6dO+Pm5sahQ4fIycnhueeeMysXHh6Ovb0958+fZ/ny5Tg4OIg9s0+ePCnyUqFmerm2nWRnZ9OwYUMzG6q9IFS1sdzcXAoLCxk9erQYZXqa/ZmO2u3Zs4eCggI+/fRT8dKSm5vLzZs36yxrqqvz589byBs2bEh+fr5Z2ePHj+Pn50erVq24ePEit2/f5ty5c2ZlmzVrhr+/P/Hx8QQGBrJ//372798vFqENGzZMLFhS26Dabn755ReysrLM7Mra2hp7e3uePHliNp0JNVt8LlmyhFatWpGcnGwhNz2v6rg1Gg3dunUjPT2drKwssRYlMzOT9PR00RZMy5ouVlbboKm89uJTa2tr8QyaN28udNW7d2+zsqdOnRJBjrOzMytXrmTt2rUEBATQtGlTWrRoQXl5uXiBvHnzpvAjZWVlbNiwARcXF1xcXMx8jKIoFBcXW/igtWvXkpaWhr29PVevXmX48OF1+qeEhAR+++03cT/Nmzfn0qVLFBUViRmvtm3bkpiYSFpamkXZgwcPirL16tWzqHOjRo3ENoCqrLCwEG9vbw4cOED37t3FOgLTc1tZWdG0aVML2+rVqxdWVlaMGjXKwrZUn6oeq+2zbGxs6vTFsbGx3Lt3D61Wazbaa4q3tzdlZWVotVoWL16Mg4ODsC1HR0exyDE3N9fMttSyal3VF3fVtkzrrH5HQmX//v04ODiIsrVtSy2rziJmZWUJPW3ZsoWVK1eKc6nreVRdqddXX7DUkX8V05cd075p2bJlIg1BPf60vgtqXiRDQkIICAgQ360wpfYAkWlZNVitfR21zupAXHl5OYcPHxbpJqr8yZMnQlem5evVqyfSahRFYfbs2XX692PHjpn1t/B3u6rdF9dGlasbFZjaVV1lVf+u0WgwGo2UlZXRpUsXsf5Etau6ypr696KiIgu5alemZffv32/h3+3t7c3KHj58GFtbW27dumXWBtVZDdNUHdO99Hv37o3BYMDGxsbCrtR1UZs2bTKLXWJjY4mJiaFHjx5UV1fTpEmTp8Y1ajqfTqcTbTA9PV34ThcXF7OXUdOypt9RMLU1tc62trYW2QTqNx80Go1FG1Qx7Qv/FWRKDDWjGZs2baKkpMRspEedFlQ77KdRUVHB5MmTuX//PuvXr/+niyzVoKGyspKYmBhiYmIsftOnTx9ef/11keerkpaWhouLCy1btuTMmTNkZWWJRTOAmBo3TYfZt28fdnZ2wlhycnLqzI1WnaD6QR915wNALFZRF7mpqAvC1L1aVbKyssjMzKxzkVtt1Lf2/v3789prr/3T36v6e/jwodizffbs2UI+adIk3Nzc+Pjjjy3KpqWlib9rL6A1Go1kZ2eb7ZV85coVSkpKhDNSG2HtDkRdJHrmzBmxOKxt27ZCfubMGfF3ZmYmW7ZsEXZiMBi4ceMG9vb2FBcX12lDqo2lpKRQVVVF//79RdrQP7I/1QGVlpaKfNTx48cL+YcffgjUdLQbNmwwK1tbV1u3bjWTZ2RkkJeXR05ODuvXr6eoqIjU1FQxIqXaYmZmJps2bTIrW1VVJQJWjUZDYGAgLi4uKIoiFvPodDoSExPp168fgPgA082bN81G1FT9Q81Ikho0HDlyhHnz5tGvXz98fHxITEw0k6uo5y0oKEBRFEpKSsQ1Tfnkk0/45JNPuHbtmhitU8uqnZBOpyMrK8vi3JmZmWZtLiEhwWzEuC67Uu1cURSzHYpMiY6OBv7ePufPn2/xm4yMDDIyMizKq6O7dfkgNZBctmwZy5YtszinmpcKNbuf1PWRt+PHj4tFi08rCzUjW9u2bTM7pvqT2nVWX8R37twJYOEf+/TpIz6ABn+3rfLyclxdXTl//ryFbak+1WAwkJSUZBFMVVRUWOwepdqWq6urxYu1Keq5c3JyyM/PJyMjo07bUtuhqW2lpaWJD/YEBgZibW0tbEs9L9T4WtW2EhISSE1NxdfXVwSmtW2rdh9SVlYm9ASYfcQnNDSU3377jcTERFq2bCnadOPGjblz547ZiDpgFrSofdOWLVtYunQpPXv25NdffxULIp/Wd6l1zsvLq9PmASIjI81mPUzLqnVSdaXKAgICuHPnjujDjh07RklJCa1bt+batWtmCzVN26HpuauqqkR/OmbMGHEu1b/rdDqOHz9u1t+a2pVpX/zdd99Z3Je6VuWFF16wSAut3Y/D3/17SUkJixcvBmoWCKuodtW1a1eLsqp/1+l0/P777xZy1a7U67q7u5v5d9Wuzp49axFfKIpi4d9V3aovE7X9uxp/lJeXW8QNqn8vKCgQMZqpf+/fvz8nT56s8+N/6nnV/v5p/j0rK4s+ffqYtUG1rDoYWNu/q/La680SEhLEy0h1dbXZx6NqY9oX/jPkCDs1I7aVlZXs2rVLHDMYDMTGxoot9Z5GdXU1b775JlevXuXrr782C9LUh2RKcXExhw4dwtPTU+TCmf4LDAzEx8eHRYsWWawcvnTpEufPn6dbt25iuzJ1BwKoMepdu3ZhZ2cn6qFuJajmdAH4+fmRmJhoEWSrH5WonY5y8OBBi9+qqB/FUPPCVdQPGdXVMGqfW9W7uoWVSn5+vsVHWADx++nTp4sRiBkzZjBmzBigZqHivHnzLBpHRUUFa9eupUGDBmJUs3ZdiouLzT44debMGWxtbcUWd82aNQPMt0SDmlGFsrIy8vPz67SD1atX4+rqKqbOTLfwiouLo6ysjLy8PIuy8HcbU1fnd+jQgZiYGPE1VVUWExNjUVYNbDQaDVOmTBF2pm5t1rRpU7RarUXZ6upqMdKmfvTGdPecy5cvk5KSQllZmaizulhXnb7+4YcfAOjQoYPFy4v6gSAVtV14eXmxevVqmjdvTufOndmzZ4/4wNShQ4fExzRMt+srLi6msLAQGxsbMRJ58eJFZs2aRfv27Zk/fz6HDh3Cw8ODhg0bih1kTK/r7e0t6h8VFSX0FBMTI0bsJk2aRExMjOg8TOu8ceNGtFotHTp04OjRo+Tl5Qm5utuRqV2punF3d0ej0Qi7OnDggNm5VZt5/vnnLfxEo0aN8PHxwc3NjW+++YYWLVrg7OzMF198QWBgoPgoDdS0FdOy1tbW+Pv7M2/ePJYvX86bb76JlZUVgYGBNG/eHG9vbyZNmsS8efOIiYnhq6++MvNPMTExok10796dSZMmMWnSJLy9vYW+1LSQJUuWmJX9/PPPCQgIQKPRoNfrqV+/PosWLRJ1Vl/GBgwYIOqspvS8++67zJs3j3feeQeoSdUyPff06dMpKCgwsy3Vb4aFhfH111+b2dapU6eET926dSulpaVmtlVaWkphYaHZ9ouqbQUFBZGfn0+3bt3E8zZN41Gv6+7ujqIoDBkyRNzPzJkz0Wg0op7jx4/nq6++EmuB1LJWVlZotVp69epFt27dOHr0KMePHxd1PnHiBHfu3BG2pdrww4cPRZ1Nbcu0D9Hr9QBmaXeKorBjxw5xrLy8HH9/f3bs2EFVVZXoX9QXItP1Q4qicP/+ffFCvHv3bg4ePMiCBQsYMmQIVVVVZh8mUn2TWnbXrl0iTSQoKEjoatmyZbRo0cLsxahTp04WdVZ58uQJer2effv2UV5eLuqsDhio/cK+ffuoX78+d+/eNauXqivTetnY2FBRUUGzZs0s+tO8vDzWrFlDt27daN++PWlpafTq1UvI9+zZQ2lpKV26dLEoa8rRo0e5ceMGbm5uYh2PSlpaWp1l1f5/+PDhaLVawsPDzfz7pEmTmD9/vtiG0XS3slWrVqHVamnbti2PHz+mR48eQn7lyhXu3LlD27ZtxXXVhfrqC61qV4mJiWbndnNzo7y83OxlTNWpTqcTH1Jr27at8O+1dV5720Z1BikgIIDExET2798v/HtMTIyYsUtNTSUpKcmsHarXVQP9oUOHmvlSdYRdq9Xy7rvvmo2wq2VVm2ndurXw76Z1NhgMZj5HbYdarZagoCAL/67y+++/W11j1YgAABNBSURBVPSF/wiN8s827/4PYcaMGRw9epRx48bRpEkT4uLiSExMZMOGDbRr106sDE5JSWH//v28/PLLNG7cmLNnz3LhwgV69eolFpiorFmzBi8vL0JDQ3F3dycjI4PY2FgyMzP58ssvGTRokEU9xowZQ2FhIU5OTtja2hIaGoqLiwt37txhx44dODg4sHv3bho1asScOXPYs2cPI0aMICQkhBMnTvDrr7/y9ttvM3HiRAA2b97MJ598wpo1a0Tnra4Ad3FxYfTo0Tg5OfHrr79y8uRJevXqRXl5OV27dsXZ2Zlt27Zx48YNGjVqxMOHD8V9Ozo6Eh0dzbfffsu9e/fEtodhYWFUV1dz7do1WrVqJRY+Xrx4kZCQEG7duoW7uzuRkZHCQan5qh06dMDV1ZX69etTXl5OWVkZly5dorS0lNDQUOzt7Xn48CF3796lWbNmvPjii2bPo7CwkMOHDxMVFSW2bVLL2tracvPmTfLy8ujVqxfOzs78+OOPKIpCmzZtsLa25urVq7i4uBAZGcn9+/fFvrKBgYFERkYSHR3N0qVL2bZtmxhdUB3LlStXUBSF7t278+DBA/z8/HBzc6OoqIjz58+L6f727dtz5coVvLy8CA8Pp6CggOPHj6MoCr169SIoKEhsZXnmzBkRKF+9ehWdTodWq2Xw4MGiU0xISCApKYmQkBDS0tJo06YNbm5uGI1G0tLSxDRgz549zexN/eIr1IyiJScnExoaSsOGDamoqDB7oQsPD+fKlSs4OTnRqVMnDAYDx44do6qqiu7duzN06FCMRiMLFizA1dWVOXPmiC9guri48OTJE1q3bk3z5s0pKiri2LFjaDQaoqOjuXjxIlZWVqSmpoqv3mVnZ9O2bVuCgoLYuXMnrq6uIiVCo9Hg7OzM2LFjKS4u5vr169y4cYPi4mJcXFxo0KABiqLw+PFjFEWhS5cuXLt2jYKCAjw8PHB2dsbPz49Tp07h7+/P/fv3KSwspE2bNiQkJGBvb8/48ePx8vIya6+KohASEkKLFi04dOgQzZs35+7duxQWFuLr60taWhq+vr48//zz7NixAzs7OxRFobCwEK1Wi7W1NePHj8fX15eHDx+yatUqjEYjS5YsYfv27dSrV4/79+/z4MEDAgICyM7OprCwEFtbW7RaLXFxccTExODl5UXz5s1ZvXo1WVlZWFtbi2ni33//nVdeeYXAwECKi4vFv44dO7J69WouXrzIxYsXxTZ8Xl5ejBgxgsLCQnbu3El1dTXvvPMOmzdvpqysTOxypdVq+fzzzxk4cCAnTpwQ3wY4dOgQISEhbN++XWz7OmbMGDIzM3nw4IFIDZs9ezZDhgzh2LFjlJaWim0uX3/9dYYNG8aoUaPw9PSkurqasrIyKisrsbOz48cff8TZ2Znq6mqee+45rKysxKwn1OTgnj59mv79+3Pz5k2Ki4vFKNrYsWM5f/48RqOR5ORkbGxsqK6upqqqisjISBwdHfnuu+9QFAU7OzsiIiLYunWr2EoxNzeX3NxcEhIS0Gq1eHh4UL9+fdq2bcv/ae/eY6K43gaOfxdYQOQmIhh3BesFVo2CUosCrZrVBrVoLa0QUolSNU2rEVdt1dq00WhJ5BcvqPVarSJRNEEL0TZKjfHeiEWqIPcaLEFQ5B5ZBN4/fPfEFa3an31F3+eTkDDDXs5MmDnPnHnmOceOHVMjYl26dGHevHmkpqbS0tJCVVUVAQEBtLW1kZWVpQIUZ2dn4uLi8PT0tDqXf/XVV5hMJvr06UNlZSWBgYHcv39fDQi0trbSu3dvPvzwQ8xmM1u2bFGl5qKiotizZw92dnbExcXh7e1NQkICzc3NeHh44OPjg5ubG8OGDeOnn36iuLgYGxsbunTpQmRkpAqW7t+/T//+/TEYDOTm5lJSUoJWqyU0NJRTp04REhLC2bNnVfm64cOHk52drWavHDRoEF5eXpw6dYq2tjbCwsJUqp1lH1lmHJ44cSJjxoxh06ZN3LhxAz8/PzV3g+UOoLe3N5WVlR36tQkTJnDs2DEMBgPXr1/HYDDQr18/cnNz1TkzICCAnJwc3n77bU6fPo2zszP19fUEBweTnZ2No6MjdXV1TJ48mfT0dPXgvqXPXLx4serLAgMDaWpqoqCgAFtbWzw9PZk+fTqJiYmEh4cTFhZGVVUVBw4coK2tjf3795OSksKOHTvw9fVl5syZVFRUsGvXLoKDgxk9erQqVerr60tKSgr29vbqGNy3bx9ms5mYmJgOAy/nzp3j8OHDGI1GQkNDaW1tJSsrSx2DkydP5rvvvlP9/MWLF4mNjWXTpk1cuHCBvXv38u6776rnIo4dO6aOQRsbG7Zu3Yq3tzczZ86kqamJH3/8UV3EJCYmsm3bNpYtW4Zer7e6OJowYQIlJSUEBQURERFBVVUVu3fvprGxkW7dutG9e3c8PDy4e/cuhYWFuLq6YjabMZvNahZsLy8vNUO35WFUT09PxowZw+3btyktLeXGjRt07dqVCRMmqMEOW1tbxo8fT0lJCfn5+QwaNIjr16/TtWtXVV6zurqaoqIidY6GB6k+lplOLXGP5Tzn4uJCS0sLQ4cO5fbt2xQVFeHh4UF1dbV6FqihoQFXV1fc3d0pKSnBzs5OjbBPnz4dV1dXEhISaGlpITo6Wt3pf/h8NWrUKKqqqkhOTub+/fukpaVZ5ck/iQTs/6u5uZl169aRnp5ObW0t/v7+mEwmVUrq0VFnC3t7+w4PR1m4u7vj5+dHUVERdXV1uLi4EBgYSFxcXIfybBaWgD0yMpL09HT11LyHhwdhYWHMmzdPla8zm81s3ryZw4cPc/v2bfR6PTNmzCA6Olp9XlRUFGVlZZw+fdrqIZecnBxVsqympgadTkdkZCRGo5FVq1aRm5tLY2PjE7dNp9Px66+/PnG/uLi4qMkcXiStVovJZFKloJ6Hvb09GzZssJrG+WFOTk5PrDjztO21PLj0pM/18/OzKtv4rO/9b9jZ2eHl5UV5eflzv1er1aLX61VH+CjLifVxdDodOp1OlQx7XLsMBgPl5eVWaUlOTk6EhYVhMBg4d+4cRUVF1NbWqvxMZ2dnBg8ejNlsprS0lJqamn+035ycnFQaADzY/3369EGv11NfX09paanV8Tpq1ChWr17NgAEDqKqqUm16eFsGDBhAWVlZhzY7ODjQt29f7OzsKCsro66uDkdHRxoaGpg+fTrLly/n0KFDHDlyhMLCQqtcaUdHR4KDg4mPj8dgMLB161bS0tIoLy9Xo5WpqalWqUaXLl0iMTGR7OxsbGxsiImJwWQy4eTkRFJS0mPTW55myZIlFBYWcunSJcrKylQ5xfHjxzN79myrNMJHA3Z/f3/WrFnD1atX1f/hkCFDiImJUc8s5OTksGbNGlUab/z48aoqFMDp06eZNWsW3t7eqnOFB6O/O3fu5OjRoxQXF6PRaNRomeWCx/J/6uzsTK9evWhvb+fWrVs0NDTg4uKiJv2x1N+/d++equv+OA4ODn9bNczW1lbdFbSxsVHzRdTW1nLz5s0O5/KysjJiY2PR6XRUVlaqW+y2trZqmvobN26ofsDFxQVbW1vq6+vVMQoP0jYsE8kFBASwbt06Tpw4YdWHODg4YGNjg9lsxsHBgaCgID7//HO17y31yg0GA//5z3/Q6/Wqf6msrFTb1qNHDyIjI5k1axbR0dHk5+c/dl9YanY/iU6no6KigtbWVjQaDT179iQ2Npbo6Gi2bdvWoV+zt7dn6dKlJCQk8P3331NWVkZbWxsajQZfX1+WLFlCaGioVZstE6q5uLhgNBqZP38+KSkppKSkqAGCzz77TPWZZrOZtWvXcuDAATXya0n3MJlMmEwmrl+/Tvfu3amsrMTNzY3Ro0czf/58vL29iYqKoqSkhL59+5KXl4ezszMTJ07EZDIxc+ZM8vPzn6tymkWvXr24c+eOGtCw5GtbjsG4uDirfv7hgH3jxo0UFxfj6enJnTt3sLGxYcCAAeoYjIqKorS0FD8/P65evYqtrS2hoaF8+eWXmEwmysrK+O6779QdN8udbICPPvqIwsJCNaBnb29PUFAQEydOJD09naysLBobG9FoNDg4ODB69Gji4+NpaGggKSlJDabAgwu7qVOnEh4ezvbt29Vg3fPq27cv5eXl3Lt3D41Go87vlguWHTt2WMU9I0aM4ODBgyxevJhTp06RnZ2t7gLY2toycOBAVWji0TZrtVrGjBnDlClTOHDgAHl5eVRXV9Pa2srYsWNVNSGwPl/dvHlT7SvL+f1ZSMAuhBBCCCFEJyY57EIIIYQQQnRiErALIYQQQgjRiUnALoQQQgghRCcmAbsQQgghhBCdmATsQgghhBBCdGISsAshhBBCCNGJScAuhBBCCCFEJyYBuxBCiBfm5s2b+Pv7k5SU9LKbIoQQrw0J2IUQ4hVy8eJF/P39rX6GDBmC0Whk6dKlaor3fyopKYkTJ068oNa+OMePH8ff359bt24BcPToUQwGA3V1dS+5ZUII8e+ze9kNEEII8fzee+893nnnHQCam5vJz8/n4MGD/PLLL6Snp6PT6f7R527cuJGpU6cybty4F9nc/9rly5fR6/V4e3sDkJWVRf/+/XF1dX3JLRNCiH+fBOxCCPEKGjRoEFOmTLFa5+vry6pVqzh+/DgzZsx4OQ37l/z+++8MHz5cLWdlZTFs2LCX2CIhhPi/IwG7EEK8Jry8vADQarVW6/ft20dmZiaFhYXcvXsXd3d3Ro4cSXx8PHq9HniQe240GgFIS0sjLS1NvT8/P1/9fuHCBX744QeuXLlCU1MTXl5eBAcHs2jRIjw8PKy+9+TJk2zcuJGCggLc3NyIiIhg4cKF2Nk9vetpaWmhvr4egNbWVq5du4bRaKS6upp79+5RUFDABx98QHV1NQDu7u7Y2EiWpxDi9aRpb29vf9mNEEII8WwuXrxIbGws8+bNIyYmBniQElNQUMDq1aupra0lPT2dHj16qPcYjUYCAwPx9/fH3d2dgoICDh06hLOzM+np6XTr1o2mpiaOHz/OF198wZtvvsm0adPU+y0j+fv37+fbb7/F29ub999/H51OR3l5OSdPniQhIYGBAweqwH/IkCH89ddfREdH06NHDzIzMzlz5gwLFizg008/febtfFaZmZnq4kMIIV43ErALIcQr5O8C2f79+7Nhwwb69etntb6pqQknJyerdefPn2fGjBksWrSI2bNnq/X+/v5MnTqVhIQEq9dXVFQwbtw4fHx82L9/f4fc8ba2NmxsbFTA3qVLFzIyMlQQ3d7eTkREBDU1NZw5c+ap21lbW8u1a9cASE1N5bfffiMxMRGAlJQUrl27xqpVq9Trg4KCcHBweOrnCiHEq0hSYoQQ4hUUFRVFeHg48GCEvaioiF27djFnzhz27Nlj9dCpJVhva2ujsbGRlpYW/P39cXFxIScn55m+7+eff6alpYW5c+c+9kHPR9NRjEaj1Yi3RqMhODiY5ORkGhsb6dq1699+n5ubGyEhIQCsX7+ekJAQtbxmzRrCwsLUshBCvO4kYBdCiFeQr6+vVcA6duxY3nrrLaZNm0ZiYiJr165Vfzt//jybN2/mypUrNDc3W31ObW3tM33fn3/+CcDAgQOf6fW9e/fusM7d3R2Ampqavw3YH85fb2xs5I8//iAiIoLq6mrq6+vJy8sjJiZG5a8/mjsvhBCvGwnYhRDiNREQEICLiwsXLlxQ63Jycvjkk0/w8fFh4cKF6PV6HB0d0Wg0LFiwgH8rK9LW1vaJf3vad16+fLlD2s/KlStZuXKlWl6+fDnLly8HrB+KFUKI15EE7EII8RppbW3FbDar5YyMDFpbW9m+fbvVqHdTU9NzTTrUp08fAPLy8njjjTdeWHsfx2AwsGvXLgCSk5MpKChgxYoVAOzcuZPy8nK+/vrrf7UNQgjRmUgNLCGEeE2cPXuWpqYmBg8erNY9aaR769attLW1dVjv5ORETU1Nh/Xh4eFotVo2bdpEQ0NDh7+/yJF6S/56SEgIlZWVjBw5Ui1XVFSo3x/OaxdCiNeZjLALIcQrKDc3lyNHjgBgNpspKioiNTUVrVZLfHy8et24cePYvXs3s2fPJioqCq1Wy9mzZ8nPz6dbt24dPjcwMJDz58+zbds2evXqhUajYdKkSfTs2ZNly5axYsUKIiIimDJlCjqdjlu3bpGZmcnq1aufOb/9WTU0NJCbm8vHH38MQHV1NcXFxcydO/eFfo8QQnR2ErALIcQrKCMjg4yMDOBBhRZ3d3dCQ0OZM2cOQ4cOVa8LCgoiKSmJzZs3s379ehwcHAgJCSE5OVkFwg/75ptvWLFiBVu2bKGxsRGASZMmARATE4OPjw87d+5k7969mM1mvLy8GDVqFD179nzh23j58mVaW1sZMWIE8GB20/b2drUshBD/X0gddiGEEEIIIToxyWEXQgghhBCiE5OAXQghhBBCiE5MAnYhhBBCCCE6MQnYhRBCCCGE6MQkYBdCCCGEEKITk4BdCCGEEEKITkwCdiGEEEIIIToxCdiFEEIIIYToxCRgF0IIIYQQohOTgF0IIYQQQohO7H8AHtgJt90tV/8AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 864x432 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1YrjAPX2V-l4"
      },
      "source": [
        "Now we'll combine the results for all of the batches and calculate our final MCC score."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oCYZa1lQ8Jn8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d46672a4-9ddc-4572-8247-16e1cbc83ec9"
      },
      "source": [
        "# Combine the results across all batches. \n",
        "flat_predictions = np.concatenate(predictions, axis=0)\n",
        "\n",
        "# For each sample, pick the label (0 or 1) with the higher score.\n",
        "flat_predictions = np.argmax(flat_predictions, axis=1).flatten()\n",
        "\n",
        "# Combine the correct labels for each batch into a single list.\n",
        "flat_true_labels = np.concatenate(true_labels, axis=0)\n",
        "\n",
        "# Calculate the MCC\n",
        "mcc = matthews_corrcoef(flat_true_labels, flat_predictions)\n",
        "\n",
        "print('Total MCC: %.3f' % mcc)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Total MCC: 0.569\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XcTQ3s_qqHk0",
        "outputId": "4f86df63-6e85-43f9-d1dc-56af02c2d50e"
      },
      "source": [
        "from sklearn.metrics import classification_report\r\n",
        "print(classification_report(flat_true_labels,flat_predictions ))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.63      0.60      0.61       272\n",
            "           1       0.96      0.96      0.96      2569\n",
            "           2       0.00      0.00      0.00         2\n",
            "\n",
            "    accuracy                           0.93      2843\n",
            "   macro avg       0.53      0.52      0.52      2843\n",
            "weighted avg       0.92      0.93      0.93      2843\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:1272: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
            "  _warn_prf(average, modifier, msg_start, len(result))\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jXx0jPc4HUfZ"
      },
      "source": [
        "Cool! In about half an hour and without doing any hyperparameter tuning (adjusting the learning rate, epochs, batch size, ADAM properties, etc.) we are able to get a good score. \n",
        "\n",
        "> *Note: To maximize the score, we should remove the \"validation set\" (which we used to help determine how many epochs to train for) and train on the entire training set.*\n",
        "\n",
        "The library documents the expected accuracy for this benchmark [here](https://huggingface.co/transformers/examples.html#glue) as `49.23`.\n",
        "\n",
        "You can also look at the official leaderboard [here](https://gluebenchmark.com/leaderboard/submission/zlssuBTm5XRs0aSKbFYGVIVdvbj1/-LhijX9VVmvJcvzKymxy). \n",
        "\n",
        "Note that (due to the small dataset size?) the accuracy can vary significantly between runs.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GfjYoa6WmkN6"
      },
      "source": [
        "# Conclusion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xlQG7qgkmf4n"
      },
      "source": [
        "This post demonstrates that with a pre-trained BERT model you can quickly and effectively create a high quality model with minimal effort and training time using the pytorch interface, regardless of the specific NLP task you are interested in."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YUmsUOIv8EUO"
      },
      "source": [
        "# Appendix\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q2079Qyn8Mt8"
      },
      "source": [
        "## A1. Saving & Loading Fine-Tuned Model\n",
        "\n",
        "This first cell (taken from `run_glue.py` [here](https://github.com/huggingface/transformers/blob/35ff345fc9df9e777b27903f11fa213e4052595b/examples/run_glue.py#L495)) writes the model and tokenizer out to disk."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6ulTWaOr8QNY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c8f768fe-d47e-458c-d9fe-4aee9e3299f0"
      },
      "source": [
        "import os\n",
        "\n",
        "# Saving best-practices: if you use defaults names for the model, you can reload it using from_pretrained()\n",
        "\n",
        "output_dir = './model_save/'\n",
        "\n",
        "# Create output directory if needed\n",
        "if not os.path.exists(output_dir):\n",
        "    os.makedirs(output_dir)\n",
        "\n",
        "print(\"Saving model to %s\" % output_dir)\n",
        "\n",
        "# Save a trained model, configuration and tokenizer using `save_pretrained()`.\n",
        "# They can then be reloaded using `from_pretrained()`\n",
        "model_to_save = model.module if hasattr(model, 'module') else model  # Take care of distributed/parallel training\n",
        "model_to_save.save_pretrained(output_dir)\n",
        "tokenizer.save_pretrained(output_dir)\n",
        "\n",
        "# Good practice: save your training arguments together with the trained model\n",
        "# torch.save(args, os.path.join(output_dir, 'training_args.bin'))\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saving model to ./model_save/\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "('./model_save/tokenizer_config.json',\n",
              " './model_save/special_tokens_map.json',\n",
              " './model_save/vocab.txt',\n",
              " './model_save/added_tokens.json')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z-tjHkR7lc1I"
      },
      "source": [
        "Let's check out the file sizes, out of curiosity."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mqMzI3VTCZo5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6a97a793-15f7-4023-a72b-57baed93b39e"
      },
      "source": [
        "!ls -l --block-size=K ./model_save/"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "total 428004K\n",
            "-rw-r--r-- 1 root root      1K Dec 25 06:50 config.json\n",
            "-rw-r--r-- 1 root root 427760K Dec 25 06:50 pytorch_model.bin\n",
            "-rw-r--r-- 1 root root      1K Dec 25 06:50 special_tokens_map.json\n",
            "-rw-r--r-- 1 root root      1K Dec 25 06:50 tokenizer_config.json\n",
            "-rw-r--r-- 1 root root    227K Dec 25 06:50 vocab.txt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fr_bt2rFlgDn"
      },
      "source": [
        "The largest file is the model weights, at around 418 megabytes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-WUFUIQ8Cu8D",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bc00ad1f-4135-4338-b8de-c9c09d93c74a"
      },
      "source": [
        "!ls -l --block-size=M ./model_save/pytorch_model.bin"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "-rw-r--r-- 1 root root 418M Dec 25 06:50 ./model_save/pytorch_model.bin\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dzGKvOFAll_e"
      },
      "source": [
        "To save your model across Colab Notebook sessions, download it to your local machine, or ideally copy it to your Google Drive."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Trr-A-POC18_"
      },
      "source": [
        "# Mount Google Drive to this Notebook instance.\n",
        "from google.colab import drive\n",
        "    drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NxlZsafTC-V5"
      },
      "source": [
        "# Copy the model files to a directory in your Google Drive.\n",
        "!cp -r ./model_save/ \"./drive/MyDrive/Codalab/HopeSpeech Detection For Equality,DiversityAnd Inclusion/Models/BERT_English\""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W0vstijw85SZ"
      },
      "source": [
        "The following functions will load the model back from disk."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nskPzUM084zL"
      },
      "source": [
        "# Load a trained model and vocabulary that you have fine-tuned\n",
        "model = model_class.from_pretrained(output_dir)\n",
        "tokenizer = tokenizer_class.from_pretrained(output_dir)\n",
        "\n",
        "# Copy the model to the GPU.\n",
        "model.to(device)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NIWouvDrGVAi"
      },
      "source": [
        "## A.2. Weight Decay\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f123ZAlF1OyW"
      },
      "source": [
        "The huggingface example includes the following code block for enabling weight decay, but the default decay rate is \"0.0\", so I moved this to the appendix.\n",
        "\n",
        "This block essentially tells the optimizer to not apply weight decay to the bias terms (e.g., $ b $ in the equation $ y = Wx + b $ ). Weight decay is a form of regularization--after calculating the gradients, we multiply them by, e.g., 0.99."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QxSMw0FrptiL"
      },
      "source": [
        "# This code is taken from:\n",
        "# https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/examples/run_glue.py#L102\n",
        "\n",
        "# Don't apply weight decay to any parameters whose names include these tokens.\n",
        "# (Here, the BERT doesn't have `gamma` or `beta` parameters, only `bias` terms)\n",
        "no_decay = ['bias', 'LayerNorm.weight']\n",
        "\n",
        "# Separate the `weight` parameters from the `bias` parameters. \n",
        "# - For the `weight` parameters, this specifies a 'weight_decay_rate' of 0.01. \n",
        "# - For the `bias` parameters, the 'weight_decay_rate' is 0.0. \n",
        "optimizer_grouped_parameters = [\n",
        "    # Filter for all parameters which *don't* include 'bias', 'gamma', 'beta'.\n",
        "    {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n",
        "     'weight_decay_rate': 0.1},\n",
        "    \n",
        "    # Filter for parameters which *do* include those.\n",
        "    {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n",
        "     'weight_decay_rate': 0.0}\n",
        "]\n",
        "\n",
        "# Note - `optimizer_grouped_parameters` only includes the parameter values, not \n",
        "# the names."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IKzLS9ohzGVu"
      },
      "source": [
        "# Revision History"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SZqpiHEnGqYR"
      },
      "source": [
        "**Version 3** - *Mar 18th, 2020* - (current)\n",
        "* Simplified the tokenization and input formatting (for both training and test) by leveraging the `tokenizer.encode_plus` function. \n",
        "`encode_plus` handles padding *and* creates the attention masks for us.\n",
        "* Improved explanation of attention masks.\n",
        "* Switched to using `torch.utils.data.random_split` for creating the training-validation split.\n",
        "* Added a summary table of the training statistics (validation loss, time per epoch, etc.).\n",
        "* Added validation loss to the learning curve plot, so we can see if we're overfitting. \n",
        "    * Thank you to [Stas Bekman](https://ca.linkedin.com/in/stasbekman) for contributing this!\n",
        "* Displayed the per-batch MCC as a bar plot.\n",
        "\n",
        "**Version 2** - *Dec 20th, 2019* - [link](https://colab.research.google.com/drive/1Y4o3jh3ZH70tl6mCd76vz_IxX23biCPP)\n",
        "* huggingface renamed their library to `transformers`. \n",
        "* Updated the notebook to use the `transformers` library.\n",
        "\n",
        "**Version 1** - *July 22nd, 2019*\n",
        "* Initial version."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FL_NnDGxRpEI"
      },
      "source": [
        "## Further Work\n",
        "\n",
        "* It might make more sense to use the MCC score for “validation accuracy”, but I’ve left it out so as not to have to explain it earlier in the Notebook.\n",
        "* Seeding -- I’m not convinced that setting the seed values at the beginning of the training loop is actually creating reproducible results…\n",
        "* The MCC score seems to vary substantially across different runs. It would be interesting to run this example a number of times and show the variance.\n"
      ]
    }
  ]
}