08.model.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "from keras.utils.vis_utils import plot_model\n",
    "from keras.models import load_model\n",
    "import modules.deep_model as DeepModel\n",
    "import modules.model as Model \n",
    "import tensorflow as tf\n",
    "import warnings\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "warnings.filterwarnings('ignore')"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Công việc 2.2 (Tiếp theo):\n",
    "* Ở phần này, chúng ta sẽ xây dựng một **Comment sentiment model** bằng LSTM.\n",
    "* Chúng ta biết rằng hạn chế của RNN là nó không thể lưu trữ các thông tin trong dài hạn - RNN tiến hành lưu trữ các thông tin mà nó học được từ chuổi vào các **hidden state** nhưng khi chuổi đầu vào quá dài - nó sẽ không thể giữ lại các thông tin này do vấn đề **gradient-vanishing** [tiêu biến gradient] trong quá trình backpropagation:<br>\n",
    "  ![](./images/14.png)\n",
    "* LSTM ra đời để khắc phục gradient-vanishing bằng một cấu trúc đặc biệt gọi là **gate**. Gate sẽ lưu các thông tin của chuổi lại đồng thời nó quản lí được thông tin nào nên xóa khỏi bộ nhớ và thông tin nào cần giữ.\n",
    "* Giả sử chúng ta có câu: **Giao hàng nhanh**. Với RNN nó dễ dàng dự đoán được comment này thuộc class positive. Nhưng giả sử ta có câu dài hơn: **Giao hàng chậm \"nhưng\" hàng đẹp - cảm ơn shop**. Thì câu hỏi đặt ra là làm sao RNN có thể lưu trữ các thông tin trước từ **nhưng** là negative word và phía sau là positive word. RNN sẽ không thể lưu trữ các thông tin này do vấn đề vanishing-gradient. Nhưng với LSTM nó sẽ giảm bớt hiện tượng này nhờ sử dụng một cấu trúc là LSTM cell. <br>\n",
    "  ![](./images/15.png)\n",
    "* Lúc này, mỗi LSTM cell sẽ chứa ba gate lần lượt là:\n",
    "  * **Input gate**: quyết định thông tin nào cần được thêm vào từ chuổi đầu vào.\n",
    "  * **Output gate**: quyết định thông tin nào cần được lưu trữ\n",
    "  * **Forget gate**: thông tin nào cần được loại bỏ.\n",
    "  \n",
    "  ![](./images/16.png)\n",
    "\n",
    "* Ở RNN, một hidden state được sử dụng cho hai mục đích là lưu trữ thông tin và đưa ra dự đoán. Tuy nhiên ở LSTM nó tách hiddent state thành hai phần là:\n",
    "  * **Cell state**: đây là nơi lưu giữ thông tin.\n",
    "  * **Hidden state**: dùng cho việc dự đoán.\n",
    "* Cell state và hidden state chia sẻ thông tin lẫn nhau. Như hình trên ta thấy đầu vào sau khi đi qua hidden state $\\rightarrow$ qua các gate $\\rightarrow$ đến cell state.\n",
    "\n",
    "Bây giờ chúng ta sẽ tiến hành xây dựng một **Comment sentiment model** sử dụng LSTM của TensorFlow."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Load dữ liệu training."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "X_train, y_train = Model.loadData(\"./data/data/train\")\n",
    "X_train = X_train.fillna(\"\")\n",
    "\n",
    "X_test, y_test = Model.loadData(\"./data/data/test\")\n",
    "X_test = X_test.fillna(\"\")\n",
    "\n",
    "display(X_train, y_train)\n",
    "display(X_test, y_test)\n"
   ],
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>raw_comment</th>\n",
       "      <th>normalize_comment</th>\n",
       "      <th>emoji_decode</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>form k đẹp lắm</td>\n",
       "      <td>form không đẹp lắm</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Áo Rộng thật sự\\nGl nhanh\\nChật lươ...</td>\n",
       "      <td>áo rộng thật sự nhanh chật lượng v...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Màu túi hơi tối do với ảnh chụp</td>\n",
       "      <td>màu túi hơi tối do ảnh chụp</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Chất liệu vải k ổn lắm</td>\n",
       "      <td>chất liệu vải không ổn lắm</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Mã \"màu đen\" hiện lên áo màu đen, còn ...</td>\n",
       "      <td>mã màu đen hiện lên áo màu đen còn mã...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11359</th>\n",
       "      <td>Khi bán combo ghi 3 đôi, nhưng nhận chỉ ...</td>\n",
       "      <td>bán combo ghi đôi nhưng nhận chỉ đôi ta...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11360</th>\n",
       "      <td>Áo đẹp, from chuẩn. Sẽ ủng hộ tiếp ạ !</td>\n",
       "      <td>áo đẹp from chuẩn sẽ ủng hộ tiếp</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11361</th>\n",
       "      <td>Đặt màu trắng kem thì giao màu xanh đen...</td>\n",
       "      <td>đặt màu trắng kem giao màu xanh đen giao...</td>\n",
       "      <td>expressionless</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11362</th>\n",
       "      <td>Đồ chơi tí hon.\\nShop nên dừng bán sp ...</td>\n",
       "      <td>đồ chơi tí hon nên dừng bán sản phẩ...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11363</th>\n",
       "      <td>Son đẹp lắm nha mọi ng, đáng đồng tiề...</td>\n",
       "      <td>son đẹp lắm mọi đáng đồng tiền bát m...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>11364 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             raw_comment  \\\n",
       "0                                      form k đẹp lắm   \n",
       "1      Áo Rộng thật sự\\nGl nhanh\\nChật lươ...   \n",
       "2               Màu túi hơi tối do với ảnh chụp   \n",
       "3                        Chất liệu vải k ổn lắm   \n",
       "4      Mã \"màu đen\" hiện lên áo màu đen, còn ...   \n",
       "...                                                  ...   \n",
       "11359  Khi bán combo ghi 3 đôi, nhưng nhận chỉ ...   \n",
       "11360  Áo đẹp, from chuẩn. Sẽ ủng hộ tiếp ạ !   \n",
       "11361  Đặt màu trắng kem thì giao màu xanh đen...   \n",
       "11362  Đồ chơi tí hon.\\nShop nên dừng bán sp ...   \n",
       "11363  Son đẹp lắm nha mọi ng, đáng đồng tiề...   \n",
       "\n",
       "                                       normalize_comment    emoji_decode  \n",
       "0                                 form không đẹp lắm                  \n",
       "1      áo rộng thật sự nhanh chật lượng v...                  \n",
       "2                     màu túi hơi tối do ảnh chụp                  \n",
       "3                   chất liệu vải không ổn lắm                  \n",
       "4      mã màu đen hiện lên áo màu đen còn mã...                  \n",
       "...                                                  ...             ...  \n",
       "11359  bán combo ghi đôi nhưng nhận chỉ đôi ta...                  \n",
       "11360         áo đẹp from chuẩn sẽ ủng hộ tiếp                  \n",
       "11361  đặt màu trắng kem giao màu xanh đen giao...  expressionless  \n",
       "11362  đồ chơi tí hon nên dừng bán sản phẩ...                  \n",
       "11363  son đẹp lắm mọi đáng đồng tiền bát m...                  \n",
       "\n",
       "[11364 rows x 3 columns]"
      ]
     },
     "metadata": {}
    },
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11359</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11360</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11361</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11362</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11363</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>11364 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       label\n",
       "0          0\n",
       "1          1\n",
       "2          0\n",
       "3          0\n",
       "4          0\n",
       "...      ...\n",
       "11359      0\n",
       "11360      1\n",
       "11361      0\n",
       "11362      0\n",
       "11363      1\n",
       "\n",
       "[11364 rows x 1 columns]"
      ]
     },
     "metadata": {}
    },
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>raw_comment</th>\n",
       "      <th>normalize_comment</th>\n",
       "      <th>emoji_decode</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Shop bán hàng kỉ quấn đầy đủ. Chất l...</td>\n",
       "      <td>bán hàng kỉ quấn đầy đủ chất lượn...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Đặt xl, áo ghi xl nhưng lại bé tí tẹo ...</td>\n",
       "      <td>đặt xl áo ghi xl nhưng bé tí tẹo ngang ...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Kẹp rất rất đẹp săn được giá sale h...</td>\n",
       "      <td>kẹp rất rất đẹp săn được giá sale h...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>K nghĩ áo đẹp vậy đâu giao hàng nhanh k...</td>\n",
       "      <td>không nghĩ áo đẹp giao hàng nhanh không ...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Hôm nay mới nhận đc nhưng nhìn qua thâ...</td>\n",
       "      <td>hôm nay mới nhận được nhưng nhìn qua...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2837</th>\n",
       "      <td>Mình đặt 2 dây buộc tóc nhưng lại chi...</td>\n",
       "      <td>đặt dây buộc tóc nhưng chỉ nhận đưo...</td>\n",
       "      <td>star slightly_smiling_face slightly_smiling_fa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2838</th>\n",
       "      <td>Chất vải đẹp, có vẻ mát, giao hàng nha...</td>\n",
       "      <td>chất vải đẹp có vẻ mát giao hàng nhanh...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2839</th>\n",
       "      <td>Dây nhìn tạm. K đẹp. Mắc sai dây mình ...</td>\n",
       "      <td>dây nhìn tạm không đẹp mắc sai dây pha...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2840</th>\n",
       "      <td>Quần đẹp chất lượng ok nhưng chữ ru...</td>\n",
       "      <td>quần đẹp chất lượng ok nhưng chữ cu...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2841</th>\n",
       "      <td>Ok hơi chật tí</td>\n",
       "      <td>ok hơi chật tí</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2842 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            raw_comment  \\\n",
       "0     Shop bán hàng kỉ quấn đầy đủ. Chất l...   \n",
       "1     Đặt xl, áo ghi xl nhưng lại bé tí tẹo ...   \n",
       "2     Kẹp rất rất đẹp săn được giá sale h...   \n",
       "3     K nghĩ áo đẹp vậy đâu giao hàng nhanh k...   \n",
       "4     Hôm nay mới nhận đc nhưng nhìn qua thâ...   \n",
       "...                                                 ...   \n",
       "2837  Mình đặt 2 dây buộc tóc nhưng lại chi...   \n",
       "2838  Chất vải đẹp, có vẻ mát, giao hàng nha...   \n",
       "2839  Dây nhìn tạm. K đẹp. Mắc sai dây mình ...   \n",
       "2840  Quần đẹp chất lượng ok nhưng chữ ru...   \n",
       "2841                                 Ok hơi chật tí   \n",
       "\n",
       "                                      normalize_comment  \\\n",
       "0     bán hàng kỉ quấn đầy đủ chất lượn...   \n",
       "1     đặt xl áo ghi xl nhưng bé tí tẹo ngang ...   \n",
       "2     kẹp rất rất đẹp săn được giá sale h...   \n",
       "3     không nghĩ áo đẹp giao hàng nhanh không ...   \n",
       "4     hôm nay mới nhận được nhưng nhìn qua...   \n",
       "...                                                 ...   \n",
       "2837  đặt dây buộc tóc nhưng chỉ nhận đưo...   \n",
       "2838  chất vải đẹp có vẻ mát giao hàng nhanh...   \n",
       "2839  dây nhìn tạm không đẹp mắc sai dây pha...   \n",
       "2840  quần đẹp chất lượng ok nhưng chữ cu...   \n",
       "2841                                 ok hơi chật tí   \n",
       "\n",
       "                                           emoji_decode  \n",
       "0                                                        \n",
       "1                                                        \n",
       "2                                                        \n",
       "3                                                        \n",
       "4                                                        \n",
       "...                                                 ...  \n",
       "2837  star slightly_smiling_face slightly_smiling_fa...  \n",
       "2838                                                     \n",
       "2839                                                     \n",
       "2840                                                     \n",
       "2841                                                     \n",
       "\n",
       "[2842 rows x 3 columns]"
      ]
     },
     "metadata": {}
    },
    {
     "output_type": "display_data",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2837</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2838</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2839</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2840</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2841</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2842 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      label\n",
       "0         1\n",
       "1         0\n",
       "2         1\n",
       "3         1\n",
       "4         1\n",
       "...     ...\n",
       "2837      0\n",
       "2838      1\n",
       "2839      0\n",
       "2840      1\n",
       "2841      1\n",
       "\n",
       "[2842 rows x 1 columns]"
      ]
     },
     "metadata": {}
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Ở đây, nhóm đã viết gọn tất cả các tính năng của một LSTM model vào một object là `SentimentLSTM` nằm trong file `deep_model.py`, về ý tưởng của nhóm như sau:\n",
    "  * **Bước 1**: Dữ liệu training ở đây là các comment đã qua bước tiền xử lí ở feature `normalize_comment` - ta bỏ qua không cần xử lí N-Grams cho feature này.\n",
    "  * **Bước 2**: Ta sử dụng cơ chế Tokenizer do TensorFlow cung cấp, object `Tokenizer` của TensorFlow yêu cầu một hyper-pamrameter là `num_words` - tức số lượng từ tối đa ta muốn tokenize, ở đây nếu để bằng `None`, `Tokenizer` sẽ được tự động tính. Cuối cùng `Tokenizer` sẽ lưu các comment dưới dạng mảng chứa các index của từ, với index của từ phổ biến nhất là 1.\n",
    "  * **Bước 3**: Ta sử dụng phương thức `pad_sequences()` của TensorFlow - phương thức này sẽ dựa vào thuộc tính `maxlen` để đưa toàn bộ các comment về cùng một độ dài, các comment nào có chiều dài lớn sẽ được cắt bớt. Các comment ngắn hơn sẽ được thay thế bằng giá trị 0.\n",
    "  * **Bước 4**: Ta sẽ định nghĩa kiến trúc của model trong `SentimentLSTM._defineModel()`. Ở đây, để tiện cho quá trình HyperParams Optimization, ta sẽ để thuộc tính `optimizer` linh động dựa trên tham vào truyền vào của hàm. Tiếp theo, ở output ta sử dụng 2 neuron để lưu giá trị của xác suất để dự đoán comment đó ở lần lượt hai class là negative và positive, ta sử dụng loss function là **binary cross-entropy** và action function **softmax** cho output layer này.\n",
    "  * **Bước 5**: Để ta có thể kiểm soát được liệu model có bị overfitting trong quá trính training không, nếu có ta có thể dừng quá trình training lại. Tại đây ta sử dụng object `CheckPoint` do TensorFlow cung cấp, object này sẽ theo dõi qua trình training và dựa vào thông số trên loss function mà ta chỉ định là trên train hay validation data để tiến hành lưu lại **weights set** của model tại thời điểm loss value này thấp nhất, ở đây nhóm sẽ sử dụng 10% dữ liệu training làm validation và chỉ định validation loss để tránh overfitting.\n",
    "\n",
    "Code dưới đây ta sẽ định nghĩa một hyperparams set cho model. Ở đây `num_words` và `maxlen` là `None` để cho TensorFlow tự tính toán, ta sử dụng 10 neuron cho LSTM layer và dropout regularixation có xác suất để một neuron bị tắt ở layer này là 20%.\n",
    "\n",
    "Sau đó ta sẽ cho in `summary` để xem tổng số weight của model và trực quan hóa kiến trúc của model."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "hyperparameters_1 = {\n",
    "    'pX': X_train['normalize_comment'], \n",
    "    'py': y_train, \n",
    "    'pnum_words': None, \n",
    "    'pseq_length': None,\n",
    "    'pno_units': 10, \n",
    "    'pdropout': 0.2, \n",
    "    'pbatch_size': 32, \n",
    "    'pepochs': 10, \n",
    "    'psave_weights': './models/lstm_model_comments_1.h5'\n",
    "}\n",
    "\n",
    "model_1 = DeepModel.SentimentLSTM()\n",
    "model_1.define(**hyperparameters_1)\n",
    "model_1.model.summary()\n",
    "plot_model(model_1.model, to_file='./images/lstm_model_comments_1.png', show_shapes=True, show_layer_names=True)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (None, 100, 79)           283215    \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, 10)                3600      \n",
      "_________________________________________________________________\n",
      "output (Dense)               (None, 2)                 22        \n",
      "=================================================================\n",
      "Total params: 286,837\n",
      "Trainable params: 286,837\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgkAAAGVCAIAAAD7TPUKAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzde1gTV/o48JNACCE3EBRCpBVpwRXbqOBXsVLk0lAKglIQL+iuLS5rVaTqVrBefk8VXZW2WgtKwbrbegFLH7sLXqqLWquiRS1YrQoFVASC3AmUO/P747Sz0wmEAUImwPv5i5w5mXkzIXkzZ868wyEIAgEAAAAUXLYDAAAAYHAgNwAAAKCD3AAAAIAOcgMAAAA6Y7YD+J/Q0FC2QwAAADatXbvWzc2N7SgQMqjjhvT09KdPn7IdBRhirl+/fv36dbajMCDwORq60tPTS0pK2I7iNwZ03IAQevfdd+fPn892FGAowYebX331FduBGAoOhwOfoyGKw+GwHcL/GNBxAwAAAAMBuQEAAAAd5AYAAAB0kBsAAADQQW4AYMQ5cuQI53cikYi29PHjx4GBgQ0NDVVVVWS3KVOmtLS0ULtRl3I4HFdXVz2+gj44ffq0o6OjsXGP825yc3P9/f3Nzc3FYrGPj8/Vq1f72icmJiYtLY32lJiYGHLnzJgxQyevRZ8gN4ARqrGx8cUXXwwICGA7ENYcOHCAIIjGxkZqY25urqurq1KplEgkVlZWBEHk5OTg9ujoaGpPvDQ7O9vS0pIgiJs3b+o1egYKCwsDAwNjY2MrKip66nPjxo2ZM2eKxeL79+8XFxePHz9+9uzZ586d61Of5cuXx8bGbt68mfqsf/zjHwRBEARhZGSk85emB5AbwAhFEERXV1dXVxdbAYhEolmzZrG19W41NDTMmTPnzTffXLVqFbWdz+dbWlomJSUdP36crdj6YfPmzTNnzrx165ZYLO62Q1dX19tvv21ubn748GGZTGZlZXXgwAEHB4eIiIjW1lbmfRwcHE6ePBkXF3fixAk9vbbBB7kBjFBisbiwsPD06dNsB2JAdu/erVKptmzZQms3NTU9evQol8uNjIzMz89nJbZ+OHToUExMjJbRpMuXL9+7dy8kJEQgEOAWIyOjhQsXlpSUZGZmMu+DEFIoFCEhIevWrevo6Bi0F6RXkBsAAAghRBBESkrK9OnTbW1tNZf6+vpu2rRJrVaHhobSTjwYLPLbvCcXLlxACNHOlOCHWVlZzPtg8+bNe/r06alTpwYat2GA3ABGom+++YY8T4i/6agtjx49CgsLMzc3t7S0DAgIKCwsxM+Kj4/HHcaOHZuTk+Pt7S0Wi83MzDw9PcmTk9u3b8d9yPGis2fP4hYrKyvqepqamq5evYoXafltqzd5eXkVFRUKhaKnDlu3blUqlXfu3Fm9erWW9VRXV69du9bBwcHExMTCwsLPz+/ixYt4EZOdjFVWVkZFRY0bN87ExGT06NHBwcG5ubk6eZlUDx48QAiNHTuW2iiXyxFC5OERkz7Y5MmTEULffvutzuNkBeQGMBLNnTuXIIigoKBuW6Kjo6Ojo0tLS9PS0i5cuLBw4ULcZ/369QRBKBSKurq6NWvWbN++XaVSXb58uaamxsvL67vvvkMIbdq0iSAIoVBIrvn1118nCMLFxYVswesRCoWvvPIKPl1JHYjw8vKytLTUf5Gou3fvIo0vQSoul3v06FE7O7uUlJSjR49220elUk2bNu3YsWP79u2rqqq6ceOGmZmZt7d3SkoKYraTEULl5eXTpk07ceJEYmJiTU3NpUuXampq3NzcsrOzdfuS6+rqEELUNwshhCdu1dbWMu+D4YSBd+MwALkBALqIiAg3NzehUOjj4+Pv75+Tk1NVVUXt0NTUlJiYiPu4uroeOXKkra1tzZo1Otl6V1cXThg6WRtz5eXlCCGpVKqlj5WV1YkTJ3g8XmRkJP5BTRMbG1tcXLx3796AgACJROLo6Hjs2DGZTBYVFUWbLKRlJ8fGxj5+/Pijjz564403RCKRs7NzamoqQRDaj1d0Be957aWNuu0jkUg4HA7ejcMA5AYA6KZNm0b+bWdnhxAqKyujdhAKhXgAAXvppZdsbW3z8vJ08r1A/kwe+Kr6BI+t8Xg87d1mzJgRHx/f1NQUGhra3NxMW3ry5EmEkL+/P9nC5/O9vb2bm5tpgy1advI333zD5XKp04ttbGycnZ1v3bql2xKz5ubmCKGmpiZqI36IFzHsQzI2NtbcJ0MU5AYA6Ki/nU1MTBBCtKmuml8KY8aMQQg9e/Zs8KMbLKampgih9vb2XntGRUWFhYXdvXuXNtW1tbW1vr7e1NSUNmfU2toaIaRSqaiNPe1kvJKuri6pVEq9tu727dsIoYKCgn6/QE0TJkxACNHyTWlpKULI0dGReR9SR0dHryfAhwrIDQD0WXV1NW3MB2cFnCEQQlwut62tjdoBD1tTGVRBZoSQTCZDCNXX1zPpnJKS4uTk9Pnnn3/55ZdkI5/Pl0qlLS0tarWa2hmPJtnY2DBZM5/PNzc3NzY2bm9vJzR4enr24SX1Bq/t1q1b1Eb80Nvbm3kfrKGhgSAIvBuHAcgNAPRZS0sLvloY++mnn8rKyhQKBfm9IJPJ8E9LTKVSPXnyhLYSMzMzMn84OTl99tlngxx1LyZNmoQ0fiD3RCQSff3110KhMDExkdo+b948hBB1Hmdra2tWVpZAIPD19WUYSXBwcEdHB60uxa5du5577jndXj3g4eExceLE9PR0clZuZ2dnamqqnZ0dOSzGpA+G33G8G4cByA0A9JlUKt24cWN2dnZTU9PNmzfDw8NNTEz27dtHdlAqlWVlZZ9++mljY2NhYeGaNWvIQwrS1KlT8/PzS0pKsrOzi4qK3N3dcTtb85QUCsWYMWPy8vIY9nd2dk5KSqI17ty5097ePjo6OjMzU61W5+fnL1q0qLy8fN++fXhkiYmdO3c6ODi89dZbZ86cqa+vr6mpSUpK+uCDD+Lj48nJvuHh4RwOp7i4mOE6u8Xlcg8dOlRTU7Ns2TKVSlVdXb1y5cqCgoLk5GQ8wsawD4Zn2SqVyoGEZEA0j9rYghBKS0tjOwowxISEhISEhPT1WfiUKWnx4sW0+ZHvv/8+8cdRI39/f/xchUIhl8t//vlnX19fsVgsEAg8PDyuXLlCXX9dXV1ERIRMJhMIBLNmzcrJySHnsG7YsAH3efDggbu7u1AotLOzS0hIIJ/r7u5uYWFx7dq1/u0QJp8jPBCE6ylRbdy40djYuLS0FD+srKyk7gEXFxfNVa1YsQLXUyJVVVVFR0fb29vzeDypVOrr65uVlYUXMd/J+CKJ8ePH83i80aNHK5XK8+fPU7fi5eUlEok6Ojq0vMyMjAzNb7zk5GRat9u3b/v5+UkkEpFI5OXlRXsrmfcJDQ2Vy+VtbW20diMjo+nTp2uJk2RQ34GQG8DQ1r/cMBA4N+hzi30ykNxQV1cnl8sjIyMHLTrdqK2tFQgEERERbAfyP7m5uRwO5/jx45qLhmhugDElAMBvpFJpRkZGenp6QkIC27H0iCCIqKgoiUSybds2tmP5TVFRUXBwcGxs7IIFC9iORWcgN+iJSCSiTsiLj49nO6LfGGxgYLCtWLFC8/4NU6ZMuXnz5pkzZxoaGtgKTLuKioqioqKsrCyGE5/0ICkpKS4uLi4ujtpI3r+hs7OTrcAGhO0Dl/9BDI6n1Gr1Cy+8QA5KDi0//vgjQigoKIjtQOgMNjAm9DmmtGfPHupnBw+XGxomnyNgmAzqvRtixw0E1NwfgCEdvCHAdZBI27dvZzsiAAYL+9Uf+wTX3Gc7CgAAGOaG2HEDAAAAPRhKuWGY1dwfQsF3dHSkpaW99tprNjY2AoHgpZde2rdvHx7Zq6uro57KxsMsHR0dZEtISAheiZaK/NRd8fDhw/nz51taWuKHtAKoAAA9Yek8RzcQs/MwuPh7c3MzrSUoKOjatWuNjY3nz58XCATTpk2jPkuhUAiFQjc3N9wnJyfn5ZdfNjExuXTpEtmHWk8fc3FxoV3Xo9kH8/T0HDVqVHZ2tpbIuz3lawjB93ouGl9DtGPHjpqamsrKyk8++YTL5VIH3319fblc7i+//EJ9lpub29GjR/HfZWVlzz//vLW19alTp9Rq9d27dz08PExNTalXeOFd4eHhcfHixaampuvXrxsZGVVWVvYUFab/6xsMHMPPETBABvXeDaXjBu2GdM19doNnYvbs2bGxsRYWFlZWVqtXr160aNG+ffvIaY5r167t6ur66KOPyP5Xr1598uRJaGgofsi8Iv+GDRtmz55tZmY2ffr0jo4O8tAHAKBPwyc3DOma++wG36uAgADyto6YQqFob2+/d+8efqhUKl966aV//vOf1dXVuGXPnj2rV68mbwbAvCL///3f//U1vPT0dA74HUIoLCyM7ShAf/T1P39QDbF5Slr0r+Z+WVnZs2fPWC+ra+DB19fXf/jhhydPnnz69Cm11vSvv/5K/h0dHf32228nJiZu3rw5Pz//woULhw8fxotwRX7Uwz3FCgoKqPehpN18kYkZM2a8++67fX3WcBUWFhYdHa3/WwOBgQsLC2M7hP8ZPrmhV7jmPjU5D6Ga++wGP2fOnO+//37fvn0LFy60srLicDh79+599913qWNoixcv3rhx46effvree+99+OGHf/7zny0sLPAiXJG/sbGxubl5gCfwuzV27Nj58+frfLVDVFhYmJubG+yQocigcsPwGVPq1ZCuuc9W8MbGxvfu3bt69aqNjU1UVNTo0aNxgtG88SGfz3/nnXeePXv24YcfHj16lHYuRG8V+QEAOjGCcsOQrrk/qMFrZ2RkNHv2bJVKtWfPnqqqqubm5osXLx48eFCz5zvvvCMQCDZt2uTj4/PCCy9QFzGpyA8AMCDsTZGiQ73N3xrSNfdpw+h79uwxkOB7Hd+/f/9+ZWVlZGSknZ0dj8eztrb+y1/+EhMTg5fSavovX74cIfTdd99p7gEtFflpu6JP/5Ywh5Wm188RMFgG9d5xiP5Ou9Q5DoeTlpY2SOOkkydPrqqqYni/Q0MzhII/fPhwQkLCzZs39bZFPE32q6++0tsWDdygfo7AoDKo924EjSkBPTh48ODatWvZjgL04siRI+S8SVqNboTQ48ePAwMDGxoaqqqqyG5Tpkwhb5iMUZdyOBxXV1c9voI+OH36tKOjo5ahy9zcXH9/f3Nzc7FY7OPjQzsrxqRPTExMWloa7SlkjW4OhzNjxgydvBZ9gtwABiolJWXevHmNjY0HDx6sra01kF89oFf4vm+NjY3UxtzcXFdXV6VSKZFIrKysCILAkyByc3Ojo6OpPfHS7OxsfPm9Pg8WGSosLAwMDIyNja2oqOipz40bN2bOnCkWi+/fv19cXDx+/PjZs2efO3euT32WL18eGxu7efNm6rP+8Y9/4MEZIyMjnb80fWBtNEsDGpyxtiFRc78nQyL45ORkhJCxsfHLL79869YtPW9dz+cbeqo7YjjrZ/I56umeoPX19WPHjqXdEzQnJ4fP51taWiKEjh07RnsKmRsM0MKFC3fu3Nne3i6Xy42MjDQ7dHZ2Ojs7y2SyX3/9Fbd0dHQ4OTnZ2dm1tLQw70P8fk/Qbvc83BPUQA3pmvtDInh829729va8vLypU6eyHQ7ov927d6tUqi1bttDaTU1Njx49yuVyIyMj8/PzWYmtHw4dOhQTE6NlNOny5cv37t0LCQkRCAS4xcjIaOHChSUlJZmZmcz7IIQUCkVISMi6deuGzZzs4Z8bAABMEASRkpIyffp0W1tbzaW+vr6bNm1Sq9WhoaG0Ew8Gi/w278mFCxcQQrQzJfhhVlYW8z7YvHnznj59eurUqYHGbRggN4CRAk+idXBwMDExsbCw8PPzI4tEDaTOuYHUUR+4vLy8iooKhULRU4etW7cqlco7d+5oVkik0rKfmdSlx7RUdNehBw8eIISoJVsQQnK5HCFEHh4x6YPhimfffvutzuNkBeQGMCKoVKpp06YdO3Zs3759VVVVN27cMDMz8/b2TklJQQht2rSJ+OOlHq+//jpBEOQ1Iuj38T3q+QA8eoDbFQpFXV3dmjVrtm/frlKpLl++XFNT4+Xl9d133w1w/dhgX1yJELp79y7S+BKk4nK5R48etbOzS0lJOXr0aLd9tO/nuXPnEr8XY4+Ojo6Oji4tLU1LS7tw4cLChQvJlZSXl0+bNu3EiROJiYk1NTVkIUvN62AGCNeVoV3igydu1dbWMu+D4YSBd+MwALkBjAixsbHFxcV79+4NCAiQSCSOjo7Hjh2TyWRRUVFaJrH0iSEXgWcC1/Ttth4iycrK6sSJEzweLzIyEv+gpmG+n7XUpWde0X0w4J2svfhYt30kEgmHw9FPaWQ9gNwARgR8Ub2/vz/Zwufzvb29m5ubdTUIYMhF4JnAZxHIsuo9mTFjRnx8fFNTU2hoqGZZLeb7WUtdeuYV3QcIlzduamqiNuKHZOVjJn1IxsbGmvtkiILcAIY/XCTc1NRULBZT262trRFCKpVKJ1vpto46+r1iruEzNTVFCLW3t/faMyoqKiws7O7du6tWraK292k/91SXHq+kq6tLKpVSr627ffs2QqigoKDfL1DThAkTEEK0fINrVjo6OjLvQ+ro6Oj1BPhQAbkBDH98Pl8qlba0tKjVamo7HuWwsbHBDwdY5xzXUae2DKEi8AghXNMX32mjVykpKU5OTp9//jm+VAJjuJ+1wxXdjY2N29vbNSfde3p69uEl9Qav7datW9RG/NDb25t5H6yhoYEgCNZvBqMrkBvAiDBv3jyEEHV+YWtra1ZWlkAg8PX1xS0DrHM+pIvAI4QmTZqENH4g90QkEn399ddCoTAxMZHazmQ/90pvFd09PDwmTpyYnp5Ozsrt7OxMTU21s7Mjh8WY9MHwm4t34zAAuQGMCDt37rS3t4+Ojs7MzFSr1fn5+YsWLSovL9+3bx8e8UADrnM+pIvAI4QUCsWYMWPy8vIY9nd2dk5KSqI1MtnPvWJS0T08PJzD4RQXFzNcZ7e4XO6hQ4dqamqWLVumUqmqq6tXrlxZUFCQnJyMR9gY9sHwLFulUjmQkAyIDq6t1hFkSNeLg6GCec2Mqqqq6Ohoe3t7Ho8nlUp9fX2zsrKoHQZSpJ31IvAkJp+jnmpmbNy40djYuLS0FD+srKykflfQ6rFjK1asoNXM0LKfmdel11LRHfPy8hKJRB0dHVpeZkZGhuY3XnJyMq3b7du3/fz8JBKJSCTy8vKivWvM+4SGhsrl8ra2Nlr7EK2ZAbkBDG0Gcv8GnBvYjoIgBpYb6urq5HI5rZ6SAaqtrRUIBLhYi4HA9ZSOHz+uuWiI5gYYUwIA/EYqlWZkZKSnpyckJLAdS48IgoiKipJIJNu2bWM7lt8UFRUFBwfHxsYuWLCA7Vh0BnIDACPUihUrNO/fMGXKlJs3b545c6ahoYGtwLSrqKgoKirKyspiOPFJD5KSkuLi4uLi4qiN5P0bOjs72QpsICA3ADAguA5SXl5eaWkph8PZtGkT2xH1Ljw8nBw6oN2/ASE0bty4zMxMiUTCSmy9srGxuXLlirOzM9uB/M+uXbs0jxjI+zcQBDGokwgGCdzGHYABWb9+/fr169mOAgAdg+MGAAAAdJAbAAAA0EFuAAAAQAe5AQAAAJ1hnYvW+b07wLCH6/+cOHGC7UAMCHyOwMBxiMG8W0ifsFiBEgAADEFaWtr8+fPZjgIhg8oNALCLw+EYzicTAHbB+QYAAAB0kBsAAADQQW4AAABAB7kBAAAAHeQGAAAAdJAbAAAA0EFuAAAAQAe5AQAAAB3kBgAAAHSQGwAAANBBbgAAAEAHuQEAAAAd5AYAAAB0kBsAAADQQW4AAABAB7kBAAAAHeQGAAAAdJAbAAAA0EFuAAAAQAe5AQAAAB3kBgAAAHSQGwAAANBBbgAAAEAHuQEAAAAd5AYAAAB0kBsAAADQQW4AAABAB7kBAAAAHeQGAAAAdJAbAAAA0EFuAAAAQAe5AQAAAB3kBgAAAHTGbAcAAGuSk5NramqoLf/+97+Li4vJh8uWLRszZoze4wKAfRyCINiOAQB2/O1vf0tKSuLz+ZqL2tvbLSwsVCqVsTH8fgIjEYwpgZFr4cKFCKHW7hgZGS1atAgSAxix4LgBjFwEQcjl8vLy8m6XXrt2zc3NTc8hAWAg4LgBjFwcDmfx4sUmJiaai2xtbWfMmKH/kAAwEJAbwIi2cOHCtrY2WqOJicmf//xnDofDSkgAGAIYUwIj3YsvvvjLL7/QGu/cufPSSy+xEg8AhgCOG8BIFx4ezuPxqC0vvPACJAYwwkFuACNdeHh4R0cH+ZDH4y1btozFeAAwBDCmBACaPHnynTt38GeBw+EUFhba29uzHRQAbILjBgDQ0qVLjYyMEEIcDsfFxQUSAwCQGwBACxcu7OrqQggZGRktXbqU7XAAYB/kBgCQTCZ75ZVXOBxOV1dXaGgo2+EAwD7IDQAghNCSJUsIgpg9e7aNjQ3bsQBgAAiKtLQ0tsMBAADAgpCQEGo66KaUGGQIMBKEhYVFR0dTKyZ9/PHHf/3rX4VCIYtRsSU7O3vv3r3w2R+xPv74Y1pLN7lh/vz5egkGADaFhYW5ublR/9tnzZpla2vLYkjs2rt3L3z2R6yvvvqK1gLnGwD4zUhODADQQG4AAABAB7kBAAAAHeQGAAAAdJAbAAB68vjx48DAwIaGhqqqKs7vpkyZ0tLSQu1GXcrhcFxdXdkKWLvTp087OjpquXFsbm6uv7+/ubm5WCz28fG5evVqX/vExMSwNXkMcgMAfdDY2Pjiiy8GBASwHcjQk5ub6+rqqlQqJRKJlZUVQRA5OTm4PTo6mtoTL83Ozra0tCQI4ubNmyyF3KPCwsLAwMDY2NiKioqe+ty4cWPmzJlisfj+/fvFxcXjx4+fPXv2uXPn+tRn+fLlsbGxmzdvHsQX0wPIDQD0AUEQXV1duPgSK0Qi0axZs9jaer81NDTMmTPnzTffXLVqFbWdz+dbWlomJSUdP36crdj6YfPmzTNnzrx165ZYLO62Q1dX19tvv21ubn748GGZTGZlZXXgwAEHB4eIiIjW1lbmfRwcHE6ePBkXF3fixAk9vbbfQW4AoA/EYnFhYeHp06fZDmSI2b17t0ql2rJlC63d1NT06NGjXC43MjIyPz+fldj64dChQzExMVpGky5fvnzv3r2QkBCBQIBbjIyMFi5cWFJSkpmZybwPQkihUISEhKxbt456lxE9gNwAABhcBEGkpKRMnz692ytIfH19N23apFarQ0NDaSceDBb5bd6TCxcuIIRoZ0rww6ysLOZ9sHnz5j19+vTUqVMDjbsvIDcAwNQ333xDniDF32LUlkePHoWFhZmbm1taWgYEBBQWFuJnxcfH4w5jx47Nycnx9vYWi8VmZmaenp7kicft27fjPuR40dmzZ3GLlZUVdT1NTU1Xr17Fi7T8bjUoeXl5FRUVCoWipw5bt25VKpV37txZvXq1lvVUV1evXbvWwcHBxMTEwsLCz8/v4sWLeBGTNwKrrKyMiooaN26ciYnJ6NGjg4ODc3NzdfIyqR48eIAQGjt2LLVRLpcjhMjDIyZ9sMmTJyOEvv32W53HqY1mrT0CgBEAIZSWltaPJwYFBSGEmpubaS1BQUHXrl1rbGw8f/68QCCYNm0a9VkKhUIoFLq5ueE+OTk5L7/8somJyaVLl8g+QqHwlVdeoT7LxcUFn4/V0gfz9PQcNWpUdnZ2P14RMcif/S+//BIhtGPHDlp7Tk6OVCrFf1dWVtrZ2SGEjhw5glvIc9FYeXm5vb29tbV1RkZGfX39w4cPg4ODORxOcnIy2afXN6KsrOz555+3trY+deqUWq2+e/euh4eHqanptWvX+vfS5HK5kZGRZvtrr72GELp+/Tq1saCgACE0depU5n2w+vp6hJC7u3v/gmQiJCSEVmsPjhsA0I2IiAg3NzehUOjj4+Pv75+Tk1NVVUXt0NTUlJiYiPu4uroeOXKkra1tzZo1Otl6V1cX/kjrZG26VV5ejhCSSqVa+lhZWZ04cYLH40VGRuIf1DSxsbHFxcV79+4NCAiQSCSOjo7Hjh2TyWRRUVG0yUJa3ojY2NjHjx9/9NFHb7zxhkgkcnZ2Tk1NJQhC+/GKrhC/33S2r30kEgmHw8G7UW8gNwCgG9OmTSP/xj+By8rKqB2EQiEeHMBeeuklW1vbvLw8nXzmL126VFNTQy0razjw+BuPx9PebcaMGfHx8U1NTaGhoc3NzbSlJ0+eRAj5+/uTLXw+39vbu7m5mTbYouWN+Oabb7hcLnUKso2NjbOz861bt54+fdqPl9YTc3NzhFBTUxO1ET/Eixj2IRkbG2vuk0EFuQEA3aD+LjYxMUEI0aa6an7gx4wZgxB69uzZ4EfHJlNTU4RQe3t7rz2joqLCwsLu3r1Lm+ra2tpaX19vampKmzNqbW2NEFKpVNTGnt4IvJKuri6pVEq9tu727dsIITyYoysTJkxACNHyTWlpKULI0dGReR9SR0dHryfAdQtyAwB6Ul1dTRvzwVkBZwiEEJfLbWtro3aoq6ujrUT7iIRhkslkCCE8aN6rlJQUJyenzz//HJ+lwPh8vlQqbWlpUavV1M54NInhrfr4fL65ubmxsXF7e7vmgLunp2cfXlJv8Npu3bpFbcQPvb29mffBGhoaCILAu1FvIDcAoCctLS34SmDsp59+KisrUygU5GdeJpPhn42YSqV68uQJbSVmZmZk/nBycvrss88GOWodmDRpEtL4gdwTkUj09ddfC4XCxMREavu8efMQQtR5nK2trVlZWQKBwNfXl2EkwcHBHR0dtLoUu3bteu6553R79YCHh8fEiRPT09PJWbmdnZ2pqal2dnbksBiTPhj+r8C7UW8gNwCgJ1KpdOPGjdnZ2U1NTTdv3gwPDzcxMdm3bx/ZQalUlpWVffrpp42NjYWFhWvWrCEPKUhTp07Nz88vKSnJzs4uKipyd3fH7V5eXpaWltevX9ff62FMoVCMGTMmLy+PYX9nZ+ekpCRa486dO+3t7aOjozMzM9VqdX5+/pr3fVMAACAASURBVKJFi8rLy/ft24dHlpjYuXOng4PDW2+9debMmfr6+pqamqSkpA8++CA+Pp6cEBweHs7hcIqLixmus1tcLvfQoUM1NTXLli1TqVTV1dUrV64sKChITk7GI2wM+2B4lq1SqRxISH1GPaqCOaxg5EB9n8OKT4eSFi9enJ2dTW15//33iT+OGvn7++PnKhQKuVz+888/+/r6isVigUDg4eFx5coV6vrr6uoiIiJkMplAIJg1a1ZOTo6Liwtez4YNG3CfBw8euLu7C4VCOzu7hIQE8rnu7u4WFhb9nos52J/9jRs3Ghsbl5aW4oeVlZXUveTi4qL5lBUrVtDm71ZVVUVHR9vb2/N4PKlU6uvrm5WVhRcxfyPwRRLjx4/n8XijR49WKpXnz5+nbsXLy0skEnV0dGh5ORkZGZrfpdTZtNjt27f9/PwkEolIJPLy8qK93cz7hIaGyuXytrY2LSENkOYcVsgNYITqR24YCJwb9La5vhrsz35dXZ1cLo+MjBy8TehEbW2tQCCIiIhgO5D/yc3N5XA4x48fH9StGO71DdRrR3W42tTUVLxa2jFaX/vomUgk4mg1kMqUsKuB/kml0oyMjPT09ISEBLZj6RFBEFFRURKJZNu2bWzH8puioqLg4ODY2NgFCxboedOGkhvWr19PEISWq+r7Z8GCBQRB0E76M+nDbinmxsbGH3/8ESEUFBSkmeG1X0PUK9jVgBVTpky5efPmmTNnGhoa2I6lexUVFUVFRVlZWQwnPulBUlJSXFxcXFyc/jdtKLnB0BBsl2IeOYb9rsYHanl5eaWlpRwOZ9OmTWxHxJpx48ZlZmZKJBK2A+mejY3NlStXnJ2d2Q7kf3bt2qX/IwZsaNTq0j9cipntKLqnOed9SDPkXa0T69evX79+PdtRANA3cNwwlMyaNeuf//wn21EAAIa/fuYGLXVuqcVyHz9+HBYWJhaLLS0tlyxZUltb++jRozlz5ojFYplMtnz5ctpVjtiDBw/8/f2lUimtjnGvmyafPnfuXKlUKhQK3d3dr1y50u0mtPTpXylm2prNzMz+7//+LzMz08fHBz8xIiKiXzu7R7Cr9barARhxqCc5Gc5jY1LnFhfLDQ4OvnnzZmNj4xdffIEQ8vPzCwoK+vHHH9Vq9cGDBxFC7777LnXNCoVCKpV6enpeuXJFrVZr1jHuddMFBQXm5uZyufzcuXNqtfrOnTtKpXLcuHF8Pp/cCpM+RL9KMdPWfPfuXR8fn9GjR9PWzKScMj4Xrenw4cO0nrCrtexqLZB+57AaOJi/PsLp5vqGP//5zwiho0ePki3l5eV8Pp96AQv+bJ86dYpswWd4vvvuO7LF3t7eycmJumY8eYb6pXnnzh2EkEKhYLjp0NBQhFB6ejrZobS0lM/nU78ymPQhev7CysjIIFtCQkIQQpWVlT2t+dmzZ2ZmZrQ1e3h49HqZUrfzlF555ZWecgPs6m53tRaQG6ggN4xwmrmhP+eitde5pc6ap97uztbW9t69e9QWuVyueRm9qanp9OnTyYfUOsYymazXTZ89exYhRK2vYmtr6+joSL2PEpM+WnRbARjfnEtzzaNHj54wYcK9e/eoa7h06RKTDfUJ7Opud7V2tItpRzK8K/R/w3pgIGhf3agf85RwnVvUw506CgoKqBugTlbjcrlGRkZmZmZki5GRkebMRUtLS1qlyTFjxpSVlT179mzUqFHaNz169Gi1Wm1qaioSiWhrIL+MWltbe+2jnZYKwN2u2cLCgslqmeh2QB+DXY36vqv37t27d+/ePj1leAsLC2M7BMAafGhO6nNuwHVuGxsbm5ubB+N2tZqFfMk6xkw2LRaL1Wp1Y2Mj9VujpqaGGn+vffqnpzUbbHV+2NVpaWnz588fYDDDw4kTJ8LCwgiDvG0c0AM8SEvVn3lKg1rntrGxkTr6Qatj3Oum/fz80O8DDlhVVdXDhw+p/Zn06R/NNatUKoa/kZlzdXVNTU0d+HpgVwMAetKf3MCkzm2/CYXCVatW3bhxo9s6xr1ueseOHaNGjYqOjj5//nxjY+PPP/8cHh5OG3lg0qd/aGu+e/fusmXLNK+/N5ByyiNhVwMA+ol6Ypr5XAUtdW41i+VS72eCENq5c+f3339Pbdm6deuePXvw33K5/IcffvD09BSJRN3WMe61xO7Dhw/nzp0rkUjwrMfMzEyygM/bb7/NpM9ASjGTazYzM5s5c+Z33303e/ZsMzMzaoS9llMWCoXa3zVclBF2da+7WgsE85QoYJ7SCKc5T4lDUD54MOY4GCZMmNDc3Pz48WO2Axn++rSrORwOnG8gwWd/hMPnG7766iuyBWpm6JJKpRo1ahT1numPHj0qLCz08vJiMaphCXY1AIMKcoOO1dbWRkZGlpSU/Prrrz/88ENYWJhEItm8eTPbcQ1DsKtZ9Pjx48DAwIaGhqqqKrLGyZQpU8hbH2PUpRwOh3rNjUE5ffq0o6OjltOlubm5/v7+5ubmYrHYx8eHNkeDeR/tDh482NMtW/DkC6yzs3Pv3r2TJ082MzOTSqVeXl7//e9/qeuJiYnBg4QDAblBl2xsbP773//W1dW9+uqrFhYWgYGBL7744g8//DB+/Hi2QxtuYFezKDc319XVValUSiQSKysrgiDwia7c3Nzo6GhqT7w0Ozsb391zIPekGiSFhYWBgYGxsbEVFRU99blx48bMmTPFYvH9+/eLi4vHjx8/e/bsc+fO9bXPQMycORP/0dnZOXfu3Pfeey8iIqKkpCQ3N3fcuHFKpZI6d3H58uWxsbED/Z1EPfkA56PAyIH0eC5aKBS+8sorhrx+5p/9+vr6sWPH0u7umZOTw+fzLS0tEULHjh2jPYXMDQZo4cKFO3fubG9vl8vlRkZGmh06OzudnZ1lMtmvv/6KWzo6OpycnOzs7FpaWpj3YeLAgQOat/PKz8/n8/nl5eX4Ia7EvHr1arJDV1fXhAkTLCwsamtryUZ8J1Hm/+GGe09QAMCQsHv3bpVKtWXLFlq7qanp0aNHuVxuZGTkELrQ5NChQzExMVpGky5fvnzv3r2QkBCBQIBbjIyMFi5cWFJSkpmZybwPEy+88IK7uzutcf/+/XPnziXnZ+OpfXPmzCE7cDicoKCg2tra9PR0slGhUISEhKxbt67f15xBbgAAMEUQREpKyvTp021tbTWX+vr6btq0Sa1Wh4aG0k48GCzy27wnFy5cQH+sV0Y+zMrKYt6HCR8fn3Xr1lFb1Gr1v/71r3feeYdswWNfY8aMoXbD16vSaurMmzfv6dOnp06dYh4AFeQGALTBF3k4ODiYmJhYWFj4+fldvHgRL9q+fTs+Tzhr1izccvbsWdyCCwKi328I2tTUdPXqVbwI/0TF7RwOZ+zYsTk5Od7e3mKxmHYXjYGsf5Dk5eVVVFRoudn41q1blUrlnTt3Vq9erWU9WvYq8xt49Hp/EZ148OABQohWh04ulyOEyMMjJn365/Dhw88999yrr75KtuC3nnZ2pLKyEiH06NEjauPkyZMRQt9++20/t00dYILzDWDkQAzON5SXl9vb21tbW2dkZNTX1z98+DA4OJjD4SQnJ5N9NMf6XVxcaMPrPZ0PUCgUQqHQzc0N36ZC8y4aA1w/kzuFYAw/+19++SVCaMeOHbT2nJwcqVSK/66srMRFc48cOYJbaOcbmOzVXm/gweQuMn3S0/mG1157DSF0/fp1amNBQQFCaOrUqcz79ENXV5ejo2NiYiK1cf/+/eiP5xsIgnBxcUEIubq6UhtxwTR3d3cm24LzDQD0QWxsbHFx8d69ewMCAiQSiaOj47Fjx2QyWVRUlJZpLX3S1NSUmJjo5uYmFApdXV2PHDnS1ta2Zs0anay8q6sLf851sjaEUHl5OeqhOi/JysrqxIkTPB4vMjIS/6CmYb5XIyIi8J7x8fHx9/fPycmpqqoiV/L48eOPPvrojTfeEIlEzs7OqampBEFoP17RFbxLaWWM+9FHuzNnzpSXly9ZsoTaGBER4eLicvDgwYSEhOrq6idPnqxataq0tBRpjI9JJBIOh4Pfsn6A3ABAj/B5P39/f7KFz+d7e3s3Nzf3/1D9j4RCIT72x6h30Rj4yi9dulRTU+Pm5jbwVWH4LAKPx9PebcaMGfHx8U1NTaGhoc3NzbSlzPdqtzfwwA+131+kHy+tJ+bm5gihpqYmaiN+iBcx7NMPn3zyydKlS2n1x0xNTS9evLhmzZr4+HiZTDZ9+nSCIPD1zJr1xIyNjTX3P0OQGwDoHr5ViampqVgsprZbW1sjhFQqlU62ovndgU8zGmZpd1NTU4QQ9XL0nkRFRYWFhd29e3fVqlXU9j7tVS038Kivr+/q6pJKpdQLxG7fvo0QwoM5ujJhwgSEEC3f4N/pjo6OzPv0VX5+/rlz56hnoUlisXjPnj3FxcVtbW3l5eUJCQk4D02dOpXWs6Ojo9eT7T2B3ABA9/h8vlQqbWlpUavV1HY87kH+RuNyuW1tbdQOdXV1tFVpGViorq6mjfmQd9HQyfp1C8+H0bzzR7dSUlKcnJw+//xzfJYCY7hXtcP3FzE2Nm5vb9ccOvf09OzDS+oNXtutW7eojfghWTWSSZ+++uSTT1599dWJEycy6YxnKAUHB1MbGxoaCILAb1k/QG4AoEfz5s1DCFFnAba2tmZlZQkEAvJ2pDKZDP9CxFQq1ZMnT2jrMTMzI7/fnZycPvvsM3JRS0sLtXou7S4aA1+/bk2aNAlp/EDuiUgk+vrrr4VCYWJiIrWdyV7t1aDeRYbKw8Nj4sSJ6enp5Kzczs7O1NRUOzs7cliMSZ8+aWho+OKLL1auXKm5qKqqisvlkmNruHNKSsqCBQtoxyj43wa/Zf1BzbcwTwmMHKiP85QaGhrIGTWfffYZ2QePmezfv1+tVv/yyy/z58+Xy+W0eUSvv/66VCp98uTJtWvXjI2Nf/75Z9yuUCikUqm3t7eWeUoDWb/O5yl1dXWNGTNGc04UdZ4SzZEjRxBCPc1T6mmv4nlKzc3NZMuGDRsQQj/++CN+WFFR4eDgMH78+NOnT9fV1VVXVx88eNDMzIz6ni5evBghVFRU1OvrInqep0QQRHZ2tqmp6YIFC8rLy6uqqiIjI42Njc+ePdvXPszj+fjjj2UyWbdHRXi6qlKpLCgoaGlpuXHjhpubm0KhwAegVMeOHUMInTx5stfNEd3NU4LcAEYoJrmBIIiqqqro6Gh7e3sejyeVSn19fbOysqgd6urqIiIiZDKZQCCYNWtWTk4OnlCIENqwYQPu8+DBA3d3d6FQaGdnl5CQQD5XoVDI5fKff/7Z19dXLBZ3exeNgay/1zuFkJh/9jdu3GhsbFxaWoof4q8qkouLi+ZTVqxYQUtmWvYq8xt49Hp/ES8vL5FI1NHRoeXlZGRkaP5ips6mxW7fvu3n5yeRSEQikZeXF+09YtiHSTwEQXR1db3wwgtbtmzpqcP58+cDAwNtbGwEAsGkSZO2bdtG1uqgCg0NlcvlbW1t2jeHQW4A4DcMc8OgwrmB3Rgw5p/9uro6uVxOq6dkgGprawUCQUREBNuB/EbP8eB6SvgmYEzA9Q0AgAGRSqUZGRnp6ekJCQlsx9IjgiCioqIkEsm2bdvYjgUhvcdTVFQUHBwcGxu7YMGCfq8EcgMAoG+mTJly8+bNM2fONDQ0sB1L9yoqKoqKirKysgzkFuJ6jicpKSkuLi4uLm4gKxnE0isAgJ7Ex8f//e9/x39zOJz3339/+/bt7IbUJ+PGjetThVE9s7GxoRWeY5ee49m1a9fAVwK5AQAWrF+/fv369WxHAUCPYEwJAAAAHeQGAAAAdJAbAAAA0EFuAAAAQNfNuejQ0FD9xwGA/n388ce4uDHAJZLgsz9iXb9+fcaMGdQWDkG5Hj07O/ujjz7Se1QAGISsrKxJkybhYtEAjDRubm5r164lH/4hNwAwknE4nLS0tPnz57MdCADsg/MNAAAA6CA3AAAAoIPcAAAAgA5yAwAAADrIDQAAAOggNwAAAKCD3AAAAIAOcgMAAAA6yA0AAADoIDcAAACgg9wAAACADnIDAAAAOsgNAAAA6CA3AAAAoIPcAAAAgA5yAwAAADrIDQAAAOggNwAAAKCD3AAAAIAOcgMAAAA6yA0AAADoIDcAAACgg9wAAACADnIDAAAAOsgNAAAA6CA3AAAAoIPcAAAAgA5yAwAAADrIDQAAAOggNwAAAKCD3AAAAIAOcgMAAAA6yA0AAADoOARBsB0DAOxYunTpjz/+SD4sKSmxtLQ0MzPDD3k8XmZmpq2tLUvRAcAmY7YDAIA1Tk5OX375JbWlvr6e/HvixImQGMCIBWNKYOQKDw/ncDjdLuLxeH/5y1/0Gw4ABgTGlMCI5urqevv2bc1PAYfDKSoqGjduHBtBAcA+OG4AI9rSpUuNjIxojVwud8aMGZAYwEgGuQGMaAsWLOjq6qI1crncpUuXshIPAAYCcgMY0caMGePh4UE7dCAIIjg4mK2QADAEkBvASLdkyRLq+QYjIyMfH58xY8awGBIArIPcAEa6N99809j4f5O5CYIIDw9nMR4ADAHkBjDSSSQSPz8/Mj0YGxsHBgayGxIArIPcAAAKDw/v7OxECBkbGwcFBUkkErYjAoBlkBsAQAEBAbhURmdn5+LFi9kOBwD2QW4AAJmamr755psIIaFQ+Prrr7MdDgDsG9r1lLKzs0tKStiOAgwHY8eORQhNmzbt3//+N9uxgGFi/vz5bIfQf0O7ZkZoaGh6ejrbUQAAQDeG9Lfr0D5uQAiFhIR89dVXbEcBUGhoKEJoSL8XcXFxMTExmiU0+ofD4aSlpQ3pX46g306cOBEWFsZ2FAMC5xsA+M2GDRt0lRgAGOogNwDwG+oVcACMcJAbAAAA0EFuAAAAQAe5AQAAAB3kBgDY9/jx48DAwIaGhqqqKs7vpkyZ0tLSQu1GXcrhcFxdXdkKWLvTp087OjpqOX+Tm5vr7+9vbm4uFot9fHyuXr3avz7aHTx4kNMDPz8/sltnZ+fevXsnT55sZmYmlUq9vLz++9//UtcTExOTlpbW160PdZAbAMsaGxtffPHFgIAAtgNhTW5urqurq1KplEgkVlZWBEHk5OTg9ujoaGpPvDQ7O9vS0pIgiJs3b7IUco8KCwsDAwNjY2MrKip66nPjxo2ZM2eKxeL79+8XFxePHz9+9uzZ586d62ufgZg5cyb+o7Ozc+7cue+9915ERERJSUlubu64ceOUSmVqairZefny5bGxsZs3b9bV1ocGYigLCQkJCQlhOwpAEAN4LxoaGsaPH+/n56fzkBgSCoWvvPKKzleLEEpLS+u1W319/dixYyMjI6mNOTk5fD7f0tISIXTs2DHaU8jcYIAWLly4c+fO9vZ2uVxuZGSk2aGzs9PZ2Vkmk/3666+4paOjw8nJyc7OrqWlhXkfJg4cOBAUFERrzM/P5/P55eXl+OE///lPhNDq1avJDl1dXRMmTLCwsKitrSUbc3Nz8QUrDDeNjzOYh2qA4LgBsEwsFhcWFp4+fZrtQNixe/dulUq1ZcsWWrupqenRo0e5XG5kZGR+fj4rsfXDoUOHYmJitIwmXb58+d69eyEhIQKBALcYGRktXLiwpKQkMzOTeR8mXnjhBXd3d1rj/v37586da2Njgx+ePHkSITRnzhyyA4fDCQoKqq2tpdZcUCgUISEh69at6+joYB7AkAa5AQDWEASRkpIyffp0W1tbzaW+vr6bNm1Sq9WhoaG0Ew8Gi/w278mFCxcQQrQzJfhhVlYW8z5M+Pj4rFu3jtqiVqv/9a9/vfPOO2QLHvui3eZPJpMhhK5cuUJtnDdv3tOnT0+dOsU8gCENcgNg0zfffEOeHsRff9SWR48ehYWFmZubW1paBgQEFBYW4mfFx8fjDmPHjs3JyfH29haLxWZmZp6enuQZy+3bt+M+s2bNwi1nz57FLVZWVtT1NDU1Xb16FS/S8+VveXl5FRUVCoWipw5bt25VKpV37txZvXq1lvVUV1evXbvWwcHBxMTEwsLCz8/v4sWLeBGT/YlVVlZGRUWNGzfOxMRk9OjRwcHBubm5OnmZVA8ePEC/VzYkyeVyhBB5eMSkT/8cPnz4ueeee/XVV8kW/M9AOztSWVmJEHr06BG1cfLkyQihb7/9diABDCVsD2oNCJxvMBwDeS+CgoIQQs3NzbSWoKCga9euNTY2nj9/XiAQTJs2jfoshUIhFArd3Nxwn5ycnJdfftnExOTSpUtkH81zCS4uLrTB+p7ON3h6eo4aNSo7O7t/LwoxON/w5ZdfIoR27NhBa8/JyZFKpfjvyspKOzs7hNCRI0dwC+18Q3l5ub29vbW1dUZGRn19/cOHD4ODgzkcTnJyMtmn1/1ZVlb2/PPPW1tbnzp1Sq1W371718PDw9TU9Nq1a/17+T2db3jttdcQQtevX6c2FhQUIISmTp3KvE8/dHV1OTo6JiYmUhv379+P/ni+gSAIFxcXhJCrqyu1sb6+HiHk7u7OZFtwvgGAQRQREeHm5iYUCn18fPz9/XNycqqqqqgdmpqaEhMTcR9XV9cjR460tbWtWbNGJ1vv6urCHxKdrK1b5eXlCCGpVKqlj5WV1YkTJ3g8XmRkJP5BTRMbG1tcXLx3796AgACJROLo6Hjs2DGZTBYVFUX7Oaxlf8bGxj5+/Pijjz564403RCKRs7NzamoqQRDaj1d0Be9kDoczwD7anTlzpry8fMmSJdTGiIgIFxeXgwcPJiQkVFdXP3nyZNWqVaWlpUhjfEwikXA4HPyWjQSQG4DhmjZtGvk3/u1cVlZG7SAUCvGRPvbSSy/Z2trm5eXp5AN86dKlmpoaNze3ga+qJ3gYjcfjae82Y8aM+Pj4pqam0NDQ5uZm2lJ8NtXf359s4fP53t7ezc3NtAEQLfvzm2++4XK51JnENjY2zs7Ot27devr0aT9eWk/Mzc0RQk1NTdRG/BAvYtinHz755JOlS5eKRCJqo6mp6cWLF9esWRMfHy+TyaZPn04QBC4nTJ6vJhkbG2vu/+EKcgMwXNQf1CYmJgihrq4uagfNbwp8UvHZs2eDH50OmJqaIoTa29t77RkVFRUWFnb37t1Vq1ZR21tbW+vr601NTcViMbXd2toaIaRSqaiNPe1PvJKuri6pVEq9QOz27dsIITyYoysTJkxACNHyDf6d7ujoyLxPX+Xn5587d456FpokFov37NlTXFzc1tZWXl6ekJCA89DUqVNpPTs6Ono92T5sQG4AQ1h1dTVtzAdnBXLaCZfLbWtro3aoq6ujrWQgwxQDhOfD4IHsXqWkpDg5OX3++ef4LAXG5/OlUmlLS4taraZ2xqNJmr98u8Xn883NzY2Njdvb2zXHnT09PfvwknqD13br1i1qI37o7e3NvE9fffLJJ6+++urEiROZdMYzlIKDg6mNDQ0NBEHgt2wkgNwAhrCWlhZ8CTH2008/lZWVKRQK8gMsk8nw701MpVI9efKEthIzMzMyfzg5OX322WeDHPX/TJo0CWn8QO6JSCT6+uuvhUJhYmIitX3evHkIIercytbW1qysLIFA4OvryzCS4ODgjo4OWl2KXbt2Pffcc7qd0e/h4TFx4sT09HRyVm5nZ2dqaqqdnR05LMakT580NDR88cUXK1eu1FxUVVXF5XKpY5UNDQ0pKSkLFiygHaPgfyT8lo0EkBvAECaVSjdu3Jidnd3U1HTz5s3w8HATE5N9+/aRHZRKZVlZ2aefftrY2FhYWLhmzRraTHaE0NSpU/Pz80tKSrKzs4uKisirpby8vCwtLa9fvz548SsUijFjxuTl5THs7+zsnJSURGvcuXOnvb19dHR0ZmamWq3Oz89ftGhReXn5vn378MgSEzt37nRwcHjrrbfOnDlTX19fU1OTlJT0wQcfxMfHk/N6w8PDORxOcXExw3V2i8vlHjp0qKamZtmyZSqVqrq6euXKlQUFBcnJyXiEjWGfPsXz+eefi0QinEQ1EQSxbNmyX375pbW19Ycffnj99detra0TEhJo3fCMXqVS2c9XPuToaT7U4IA5rIajf+8FPo9KWrx4cXZ2NrXl/fffJ/44auTv74+fq1Ao5HL5zz//7OvrKxaLBQKBh4fHlStXqOuvq6uLiIiQyWQCgWDWrFk5OTl4eiJCaMOGDbjPgwcP3N3dhUKhnZ1dQkIC+Vx3d3cLC4t+T+JEzGpmbNy40djYuLS0FD/EM+tJLi4umk9ZsWIFbRpuVVVVdHS0vb09j8eTSqW+vr5ZWVl4EfP9iS+SGD9+PI/HGz16tFKpPH/+PHUrXl5eIpGoo6NDy8vJyMjQ/JKhzqbFbt++7efnJ5FIRCKRl5cX7V1j2IdJPARBdHV1vfDCC1u2bOmpw/nz5wMDA21sbAQCwaRJk7Zt20bW6qAKDQ2Vy+VtbW3aN4cNgzmsQzt6yA2GQ//vBc4N+txinzDMDXV1dXK5nFZPyQDV1tYKBIKIiAi2A/mNnuPB9ZSOHz/OsP8wyA3Df0yJeg0t27H0SCQSUeeHxMfHd9uts7Pz4MGDM2fOlEqlPB7P1tb2jTfe+PTTT8kLOCdPntxTUWJSTEwM9SHtdyXV3//+d7Lb9u3bB+OFA6lUmpGRkZ6erjmIYTgIgoiKipJIJNu2bWM7FoT0Hk9RUVFwcHBsbOyCBQv0sDkDMfxzw/r16wmC0FKWwBA0Njb++OOPCCFcNnL9+vXddluyZMnKlSvnzp177949tVr9/fffT5kyJSoqilp55quvviIzf2RkJELozJkzZEtYWJhIJCIIAm8OIdTTp6u6uvrgwYMIocWLFxMEsWnTJt2+ZECaMmXKzZs3z5w509DQwHYs3auoqCgqKsrKymI48Wmw6TmepKSkuLi4uLg4PWzLcAz/3NA/IpGIrMNjIHJyco4fP/7222+/9957Y8eONTU1dXBwiIuLW7FioDraEQAAIABJREFURf9WKBAInn/++TNnznR7G4CPP/4YXx5lgPCxYF5eXmlpKYfDGQZ5a9y4cZmZmRKJhO1AumdjY3PlyhVnZ2e2A/mNnuPZtWvXiDpiwCA3DBn37t1DCDk5OdHa58+fT/6dm5sbEhKiZSWpqankNymXy42JiUEIaY4X1dXVHThwYMOGDQMPezDgY0ESjHcBoHOQG4YMPB/x/PnztHYPDw9alSHmli1bJpfL//Of/9y5c4fa/sknn7zxxhsODg79Wy0AYKgbobmhtbV1y5YtEyZMMDMzGzVq1Jw5c/7zn/90dnainus2U2sdP378OCwsTCwWW1paLlmypLa29tGjR3PmzBGLxTKZbPny5bSLVHXC3d3dxsbm22+/9fPzu3TpEq16RP/w+fy///3vBEFQx1IbGxv379+/cePGga8fADBEjdDcsGrVqk8++WT//v3V1dX379+fMGFCUFDQ999/j34fr6DWbcbXhc6dO5f4vdbx2rVr33vvPZVKtXfv3iNHjixevDg6Onrbtm3l5eX/7//9v5SUlK1bt1I3p5OrqEQi0VdffWVnZ3f27FlPT0+ZTBYeHn78+PFff/11IKv961//am1tnZ6efv/+fdySkJDg5eX1pz/9aSCrBQAMaSM0N2RlZTk7O7/22msCgcDa2nrPnj19quH19ttvu7i4CIXCJUuWODs7nzlzZu3atZMnTxaJRJGRkfb29rQ7XOqq2vOsWbMKCgr+9a9/BQUFNTc3Hz16dNGiRc899xz1vud9JRAI1q5d29XVtWPHDoTQr7/++vHHH7///vsDDBUAMKTp9S5XhuP1118/cODAX//617feemvatGlGRkYPHz5k/nTqnFFbW9t79+5RW+RyOa0KwqVLlwYc8m/4fP7SpUuXLl3a0dFx+fLl5OTk1NTU8PBwJyenKVOm9G+d77zzzu7du48fP75169aMjIwZM2a8/PLL/VjP9evXQ0ND+xfDsPTxxx/jas9gpNFtYXNWjNDjhoSEhC+++KKoqMjb21sikbz++uu04g3aUecacrlcIyMjMzMzssXIyEgnJwO0MzY29vLyOn78+IYNGzo7O6n3Pe8rkUgUHR3d2dm5devW+Pj4YTAlFAAwQCP0uIHD4SxZsmTJkiXt7e2XLl2Kj48PDg7+8MMP165dS3ZgN0JNV69eDQ4Opt3JCyHk6em5a9eu2tragax89erV8fHxx44d8/Pzo93DnbkZM2bAz2QSh8N59913qTOMwchx4sSJsLAwtqMYkBF63GBubo5vr8jj8V577TU8B4la5ZjFus2ajI2NHzx4QBDEs2fPNE9o4yvX+j2ghEml0rVr10qlUjhoAACgEZsbEEJ/+9vf7ty509ra+uzZs927dxME4eXlRS7tqW5z/+iw2vP8+fOPHTtWVlbW2tr66NGj+Pj4Dz74wMXFZenSpQNc85YtW+rq6mbOnDnwIAEAQ94g1fDTDya1P/fs2UN9vbhGcW5ubmRk5J/+9Cd8fcOMGTOSk5PJ2UREd3WbNWsdU+8qgxDauXMnngVL2rp1K15br9WehUKh9rfp/v37nZ2dV65cWb9+/fTp021tbY2NjcVisaur644dO5qammgrPHz4MG0NarW62835+vp2GxLt6fv379e+n6EmLg1iVocVDEvDoA4rhxjwxEoW4VkxMMZtCOC9oOFwOGlpaXC+YWTC5xuG9LfryB1TAmD4efz4cWBgYENDQ1VVFXkZ/5QpU8iba2LUpRwOp9+zDwbb6dOnHR0dyRvPacrNzfX39zc3NxeLxT4+PrR7msbExODf76AfIDcAMEzk5ua6uroqlUqJRGJlZUUQBB72zM3NjY6OpvbES7Ozs/H947otxMuuwsLCwMDA2NhYzYl5pBs3bsycOVMsFt+/f7+4uHj8+PGzZ88+d+4c2WH58uWxsbGbN2/WS8jDDeQGMCQNdhF1AyzSrl1DQ8OcOXPefPPNVatWUdv5fL6lpWVSUtLx48fZiq0fNm/ePHPmzFu3bonF4m47dHV1vf322+bm5ocPH5bJZFZWVgcOHHBwcIiIiGhtbcV9HBwcTp48GRcXd+LECT3GPkxAbgBgONi9e7dKpdqyZQut3dTU9OjRo1wuNzIyMj8/n5XY+uHQoUMxMTFaRpMuX7587969kJAQgUCAW4yMjBYuXFhSUpKZmUl2UygUISEh69atw1XRAHOQGwAY8giCSElJwRPYNJf6+vpu2rRJrVaHhobSTjwYLPIbvycXLlxAf6xeQz7MysqiNs6bN+/p06fUq5cAE5AbgL5VV1evXbvWwcHBxMTEwsLCz8/v4sWLeNH27dvx2VFyPOfs2bO4xcrKCrf0VESdemPwnJwcb29vsVhsZmbm6elJnqIcyPoNWV5eXkVFhZYb327dulWpVN65c2f16tVa1qPlraHWqH/06FFYWJi5ubmlpWVAQEBhYSF1JZWVlVFRUePGjTMxMRk9enRwcHBubq5OXiYVvnaVdhN4uVyOEKIdHk2ePBkh9O233+o8hmGOzQm0AwZz6g0Hw/eivLzc3t7e2to6IyOjvr7+4cOHwcHBHA4nOTmZ7EMtkI65uLjgs6Za+mAKhUIoFLq5uV27dq2xsTEnJ+fll182MTG5dOmSTtbv6ek5atSo7OzsXl8p0uP1DV9++SVCaMeOHbT2nJwcqVSK/66srMQ3eT1y5AhuIc9FY0zeGlyjPigoCO/e8+fPCwSCadOmkR3Kysqef/55a2vrU6dOqdXqu3fvenh4mJqaarm4Rzu5XG5kZKTZ/tprryGErl+/Tm0sKChACE2dOpXaWF9fjxByd3fvXwD9Mwyub4DjBqBXsbGxxcXFe/fuDQgIkEgkjo6Ox44dk8lkUVFRWmak9ElTU1NiYqKbm5tQKHR1dT1y5EhbW9uaNWt0snJdlVvXrfLycoSQVCrV0sfKyurEiRM8Hi8yMhL/6KZh/tZERETg3evj4+Pv75+Tk0PeeTA2Nvbx48cfffTRG2+8IRKJnJ2dU1NTCYLQfryiK/h9oRVDk0gkHA4H7yLAHOQGoFe43q2/vz/Zwufzvb29m5ubdXXULxQK8TAC9tJLL9na2ubl5enk2+HSpUs1NTVubm4DX5UO4bMIPB5Pe7cZM2bEx8c3NTWFhoY2NzfTljJ/a6ZNm0b+jY9FysrK8MNvvvmGy+UGBASQHWxsbJydnW/duqXbstXm5uYIoaamJmojfogXURkbG2u+XqAd5AagP62trfX19aamprSJifhW2CqVSidb0fxqGDNmDELo2bNnOlm/ATI1NUUItbe399ozKioqLCzs7t27tKmufXprqAcoJiYmCCFclB6vpKurSyqVUq+tu337NkIID/joyoQJE5DGbRJKS0sRQpr36ero6Oj15DaggdwA9IfP50ul0paWFtr9tPGQhY2NDX7I5XLJIrhYXV0dbVVaiqhXV1fTxnxwVsAZYuDrN0AymQwhhAfWe5WSkuLk5PT555/jsxQYw7dGOz6fb25ubmxs3N7erjl+7enp2YeX1Bu8tlu3blEb8UNvb29qY0NDA0EQeBcB5iA3AL2aN28eQog6obC1tTUrK0sgEPj6+uIWmUyGfwBiKpXqyZMntPVoKaLe0tJCLYP4008/lZWVKRQK8tthgOs3QJMmTUKM7zUmEom+/vproVCYmJhIbWfy1vQqODi4o6ODVrti165dzz33nG6vMPDw8Jg4cWJ6ejo5K7ezszM1NdXOzo46LIZ+P5jAuwgwB7kB6NXOnTvt7e2jo6MzMzPVanV+fv6iRYvKy8v37duHhy8QQkqlsqys7NNPP21sbCwsLFyzZg35k5+kpYi6VCrduHFjdnZ2U1PTzZs3w8PDTUxM9u3bR3YYyPp1WG5dhxQKxZgxY2h3otXC2dk5KSmJ1sjkrenVzp07HRwc3nrrrTNnztTX19fU1CQlJX3wwQfx8fHkVODw8HAOh1NcXMxwnd3icrmHDh2qqalZtmyZSqWqrq5euXJlQUFBcnIyHmEj4Rm0SqVyIJsbifQ3JWoQwBxWw8H8vaiqqoqOjra3t+fxeFKp1NfXNysri9qhrq4uIiJCJpMJBIJZs2bl5OS4uLjgf9cNGzbgPppF1DGFQiGXy3/++WdfX1+xWCwQCDw8PK5cuaKr9fdabp2E9Fuje+PGjcbGxqWlpfhhZWUl9WPu4uKi+ZQVK1bQZu5qeWs0a9QTfxy48/f3xz3xRRLjx4/n8XijR49WKpXnz5+nbsXLy0skEnV0dGh5ORkZGZpfVtTZtNjt27f9/PwkEolIJPLy8qK90VhoaKhcLm9ra9OyOZ0bBnNYh3b0kBsMh4G8Fzg3sB0FQeg9N9TV1cnl8sjISL1tsX9qa2sFAkFERIR+Npebm8vhcI4fP66fzZGGQW6AMSUAhgOpVJqRkZGenp6QkMB2LD0iCCIqKkoikWzbtk0PmysqKgoODo6NjV2wYIEeNjfMQG4AYJiYMmXKzZs3z5w509DQwHYs3auoqCgqKsrKymI48WmAkpKS4uLi4uLi9LCt4QdyAxgmcB2kvLy80tJSDoezadMmtiNiwbhx4zIzMyUSCduBdM/GxubKlSvOzs762dyuXbvgiKHfDL2IGAAMrV+/fv369WxHAcAwAccNAAAA6CA3AAAAoIPcAAAAgA5yAwAAADrIDQAAAOg4hIHdpaRPQkND09PT2Y4CAAC6MaS/XYd2bsjOzi4pKWE7CjBMhIWFRUdHG9p9e8DQNX/+fLZD6L+hnRsA0CEOh5OWljakP88A6AqcbwAAAEAHuQEAAAAd5AYAAAB0kBsAAADQQW4AAABAB7kBAAAAHeQGAAAAdJAbAAAA0EFuAAAAQAe5AQAAAB3kBgAAAHSQGwAAANBBbgAAAEAHuQEAAAAd5AYAAAB0kBsAAADQQW4AAABAB7kBAAAAHeQGAAAAdJAbAAAA0EFuAAAAQAe5AQAAAB3kBgAAAHSQGwAAANBBbgAAAEAHuQEAAAAd5AYAAAB0kBsAAADQQW4AAABAB7kBAAAAHeQGAAAAdJAbAAAA0BmzHQAArHn8+HFnZye1paKioqioiHxoa2tramqq97gAYB+HIAi2YwCAHf7+/qdPn+5pKY/Hq6iosLCw0GdIABgIGFMCI9eCBQt6WsTlcpVKJSQGMGJBbgAjV3BwcE9DRgRBLFmyRM/xAGA4IDeAkUsoFAYEBPB4PM1FfD4/ICBA/yEBYCAgN4ARbfHixR0dHbRGHo8XHBwsFAr/f3v3HtTEtQYA/GwkQExIUFBApEWpjxptVHAEK1VBQxWVAYmIaKdanNRaER+1xmdbRauljs5UKxWtjg8UcUBBrVqUdtBoARusWqsD1ooY5CEEIvIwe/84t3v3LhAWCFlIvt9f5OzJ7rebIV929+x3OAkJgK4AcgOwatOmTROJRIzGhoaGqKgoTuIBoIuA3ACsmq2trUKhsLW1pTeKxeLJkydzFRIAXQHkBmDt5s6dW19fT73k8/mRkZGMbAGAtYHnG4C1MxgMrq6upaWlVMsvv/zy3nvvcRgSAJyD8wZg7Xg8XlRUFDVaqU+fPuPHj+c2JAA4B7kBABQZGdnQ0IAQsrW1/fDDD3k8+L8A1g6uKQGASJL09PT8559/EEK5ubne3t5cRwQAx+D3EQCIIIgPPvgAITRw4EBIDAAga6jDunPnTrVazXUUoKvT6XQIIXt7e4VCwXUsoBs4deoU1yF0Lss/b1Cr1Tdu3OA6CmBKN27cMPlnKhaLHR0dPTw8TLta80hJSSkqKuI6CmtRVFSUkpLCdRSdzvLPGxBCvr6+Fp/krQr+aW/yz/Tnn3/upo+8EQSxfPny2bNncx2IVUhOTo6IiOA6ik5n+ecNALDUTRMDAJ0BcgMAAAAmyA0AAACYIDcAAABggtwAgHV5/PjxzJkzdTpdWVkZ8a9Ro0a9evWK3o2+lCAIHx8frgI27vz584MHD7axaXFYjUajCQ4OdnR0dHBwmDx58rVr1+hL16xZc/Lkyc4Ps/uB3ACsSE1NzaBBg6x5QjeNRuPj4yOXy8VisbOzM0mSOTk5uD02NpbeEy9Vq9VOTk4kSebm5nIUcosKCgpmzpypUqlKSkpa6nPz5s1x48Y5ODj8+eefjx49Gjhw4MSJEy9dukR1WLRokUql2rBhg1lC7k4gNwArQpKkwWAwGAxcBSASiTgs5KfT6WbMmDFr1qxPP/2U3m5nZ+fk5JSQkJCUlMRVbO2wYcOGcePG5eXlOTg4NNvBYDB89NFHjo6OP/74o5ubm7Oz8/fff+/l5RUdHV1XV4f7eHl5paamxsXFJScnmzH2bgByA7AiDg4OBQUF58+f5zoQbuzYsUOr1W7cuJHRbm9vf+zYMR6Pp1QqHzx4wEls7XDgwIE1a9YYuZr066+/3r17Nzw8XCAQ4JYePXpERkY+efIkIyOD6iaTycLDw1euXNl0dlhrBrkBAKtAkmRiYuLYsWP79evXdGlQUND69eurq6sVCgXjxkOXRX3jt+TKlSsIIcadEvwyMzOT3hgaGlpUVHTu3DlTx9iNQW4A1iItLY26s4q//ugtf//9d0REhKOjo5OT0/Tp0wsKCvC74uPjcYf+/fvn5OQEBgY6ODj07Nlz0qRJ1F3NLVu24D7U9aKffvoJtzg7O9PXo9frr127hhcZ+cHbGfLz80tKSmQyWUsdNm3aJJfLb9++vXTpUiPrKS8vX7FihZeXl62tba9evaZOnXr16lW8iM3xxEpLS2NiYjw9PW1tbfv06RMWFqbRaEyym3T3799HCPXv35/e6O7ujhBinB6NHDkSIXTx4kWTx9CNkZYuPDw8PDyc6yiAKXXkMw0JCUEI1dbWMlpCQkKuX79eU1Nz+fJlgUAwZswY+rtkMplQKPTz88N9cnJy3nnnHVtb26ysLKqPUCh899136e/y9vbGN3KN9MEmTZrUu3dvtVrdvp1CCJ08edJ4nyNHjiCEtm7dymjPycmRSCT479LSUlxR6ujRo7iFuheNPXv2bMCAAS4uLunp6VVVVX/99VdYWBhBEPv376f6tHo8i4uL33zzTRcXl3PnzlVXV9+5c2fChAn29vbXr19v3+67u7v36NGjafuUKVMQQjdu3KA3Pnz4ECE0evRoemNVVRVCyN/fn83m8Lim9oXajcB5AwAIIRQdHe3n5ycUCidPnhwcHJyTk1NWVkbvoNfr9+7di/v4+PgcPXq0vr5+2bJlJtm6wWDA/5AmWVuznj17hhCSSCRG+jg7OycnJ/P5fKVSiX90M6hUqkePHu3atWv69OlisXjw4MHHjx93c3OLiYlhDBYycjxVKtXjx4937tw5bdo0kUgklUpPnDhBkqTx8xVTwQeZIAh6o1gsJggCHyKAQW4AACGExowZQ/2NfzsXFxfTOwiFQnzlARsxYkS/fv3y8/NN8oWSlZVVUVHh5+fX8VW1BF9Go6Y+bYmvr298fLxer1coFLW1tYylqampCKHg4GCqxc7OLjAwsLa2lnFBxsjxTEtL4/F49JHErq6uUqk0Ly/PtNVkHR0dEUJ6vZ7eiF/iRXQ2NjZN99eaQW4AAKH//0Fta2uLEGIMdW36bdK3b1+E0PPnzzs/OhOwt7dHCOGpT42LiYmJiIi4c+cOY6hrXV1dVVWVvb09Y8yoi4sLQkir1dIbWzqeeCUGg0EikdCfrbt16xZCCF/wMZWhQ4cihBj55unTpwihwYMHMzo3Nja2enPbqkBuAICV8vJyxjUfnBVwhkAI8Xi8+vp6eofKykrGShiXMszJzc0NIYQvrLcqMTFxyJAhBw8exHcpMDs7O4lE8urVq+rqanpnfDXJ1dWVzZrt7OwcHR1tbGwaGhqaXuOeNGlSG3apNXhteXl59Eb8MjAwkN6o0+lIksSHCGCQGwBg5dWrV/gRYuyPP/4oLi6WyWTUF4qbmxv+TYpptVo8ATVdz549qfwxZMiQH374oZOj/p/hw4ejJj+iWyISiU6fPi0UCvfu3UtvDw0NRQjRx3rW1dVlZmYKBIKgoCCWkYSFhTU2NjJqV2zfvv2NN94w7RMGEyZMGDZsWEpKCjUq9/Xr1ydOnPDw8KBfFkP/nkzgQwQwyA0AsCKRSNauXatWq/V6fW5u7rx582xtbXfv3k11kMvlxcXF3333XU1NTUFBwbJly6hTCsro0aMfPHjw5MkTtVpdWFjo7++P2wMCApycnDp1gkKZTNa3b9/8/HyW/aVSaUJCAqNx27ZtAwYMiI2NzcjIqK6ufvDgwdy5c589e7Z79258ZYmNbdu2eXl5LVy48MKFC1VVVRUVFQkJCV999VV8fDw1rnfevHkEQTx69IjlOpvF4/EOHDhQUVGxYMECrVZbXl6+ZMmShw8f7t+/H19ho+ARtHK5vCObszTmGxLFERjDanna95ni+6iUqKgoxkTi69atI///qlFwcDB+r0wmc3d3v3fvXlBQkIODg0AgmDBhQnZ2Nn39lZWV0dHRbm5uAoFg/PjxOTk53t7eeD2ff/457nP//n1/f3+hUOjh4bFnzx7qvf7+/r169Wr3IE7EYgwrSZJr1661sbF5+vQpfllaWkrfWW9v76ZvWbx4MWMYbllZWWxs7IABA/h8vkQiCQoKyszMxIvYH0/8kMTAgQP5fH6fPn3kcvnly5fpWwkICBCJRI2NjUZ2Jz09vekXGn00LXbr1q2pU6eKxWKRSBQQEMD41DCFQuHu7l5fX29kcxQrGcNq+XsIucHymP8zxbnBnFtsE5a5obKy0t3dXalUmiGkjnjx4oVAIIiOjjbP5jQaDUEQSUlJLPtbSW6Aa0oAWAuJRJKenp6SkrJnzx6uY2kRSZIxMTFisXjz5s1m2FxhYWFYWJhKpZozZ44ZNteNQG6wFiKRiD5kkMfj9erVSyaTffLJJ4yBHMCCjRo1Kjc398KFCzqdjutYmldSUlJYWJiZmcly4FMHJSQkxMXFxcXFmWFb3QvkBmtRU1Pz+++/I4RCQkJIkmxoaLh///5XX311//59Hx+fBQsWvHz5kusYuyJcByk/P//p06cEQaxfv57riDrK09MzIyNDLBZzHUjzXF1ds7OzpVKpeTa3fft2OGNoFuQGE+vsAv2mWn+PHj1cXFxCQkKuXLmyevXqQ4cORUZGkp1Zs6GbWrVqFf0i7JYtW7iOCABzgNwA0Ndffz127NizZ8+eOHGC61gAAF0C5AaACILA1REYDzoBAKwW5Ib/MVKYviMF+rvFBAB4uzdu3KDq7RipsM+yTH9dXd3GjRuHDh3as2fP3r17z5gx4+zZs69fv6Y6mKeIPwCgPTgZOWtOLMfCsylM35EC/V1hAgD6vWgGqgJlcXExya7Cfqtl+qOjoyUSyaVLl16+fKnValetWoUQunr1Kl7akSL+8MwKA2L3fAMwCXi+wbqwL0zfbl15AgDGG9lX2DdSpj8zM1MqlU6ZMkUgELi4uHzzzTf04pfcFvEHALSCw7xkHix/Y+KSwrgcI2X+/PkIocOHD+OXHT9vYDTimXvxT/UOrp8NI+cN+FoQn8/HZQMkEgmPx6uqqqL3GT16NELoyZMn+CU+b9BqtVSH5cuXI4Ty8/Pxy8WLFyOEFi1apFarmxY/YLOJloSHh3P1/wIAxupfrjsz64y1XVabCtO3W7MTABQXFz9//pzz4sDZ2dkIIT8/Pz6fj48GamGOsIcPH9In4DUy7cGePXv8/PwOHz6M6yH7+/srlUpcyLNNm2iWr68vTkUAIRQREREbG9upUwMBilqt3rVrF9dRdDrIDQj9W5i+qqqqurqanh4Yhek7WKAfTwBA79BFJgAwGAy4iMKSJUvQvxX2a2pqamtrO3K7myCI+fPnz58/v6GhISsrKz4+Piws7Ntvv12xYkXHN9G/f//Zs2e3OzYLExER4efnBwfEbKwhN8D9hv9iU5i+gwX6u+wEACqV6rfffgsNDVUoFLjFJBX2HR0d8ZzDfD5/ypQpeHQTdYTNVsQfANAOkBv+i01h+o4U6EddbAIAg8Hw/PnzM2fOBAYG7tixY+HChceOHaNOSthU2Gfj448/vn37dl1d3fPnz3fs2EGSZEBAgGk3AQDoFFzf8Oh07Mc7GilMj3WkQD/nEwAIhUL6504QhEQiGTFixOLFi/Py8pr2N1Jhn2WZfo1Go1Qq3377bfx8g6+v7/79+6nxVMY3YRyMYWVAMIbVjKxkDCtBWnoJHXyd5NSpU9yGMXLkyLKyMpYzMgLjushn2nUQBHHy5Em432AeycnJERERFv/NCdeUALB2jx8/njlzpk6nKysro554HzVqFDXNMkZfShCEj48PVwE368WLF/v27QsICOjdu7dAIBg0aFBUVBRjDtQ1a9bgX/2gVZAbALBqGo3Gx8dHLpeLxWJnZ2eSJPGICY1GExsbS++Jl6rVavzMTW5uLkchN++zzz5bunRpSEjIvXv3ysvLDx48qNFovL2909LSqD6LFi1SqVQbNmzgMM7uAnJDp7O8CQCsTXepu94OOp1uxowZs2bNwsUWKXZ2dk5OTgkJCUlJSZwE1j4LFy5ctmyZq6trz549/f39jx8//vr169WrV1MdvLy8UlNT4+LikpOTOYyzW4Dc0OlgAgDQZe3YsUOr1W7cuJHRbm9vf+zYMR6Pp1QqHzx4wElsbZWYmJiQkEBvkclkAoGgoKCAfm9AJpOFh4evXLkShkobB7kBACtFkmRiYuLYsWNx7RaGoKCg9evXV1dXKxQKxo2H7kKv19fW1g4fPpzxxGhoaGhRURH9YSbQFOQGYMmsue56q/Lz80tKSmQyWUsdNm3aJJfLb9++bbwAopGDzLKcO+qcgu14JNu6desY7SNHjkQIXbx4sYPrt3BmHzVrbjAW3vJA3XUG1K7nG44cOYIQ2rp1K6M9JydHIpHgv0tLSz08PBBCR48exS3UvWiMzUFutZx7Rwq2t0Sr1bq4uEQi0WrkAAADnUlEQVRHRzddhGt5+fv7t2/NVvJ8A5w3AItl5XXXW/Xs2TPUQrlDirOzc3JyMp/PVyqVuAIKA/uDbKScu8kLtpeXl7///vsTJ07ct29f06VisZggCLz7oCWQG4DFSk1NRQgFBwdTLXZ2doGBgbW1taa6niAUCvEFCmzEiBH9+vXLz883yfdOVlZWRUVF51VXxXcR+Hy+8W6+vr7x8fF6vV6hUFBzQFHYH+QxY8ZQf+NzkeLiYvwyLS2Nx+NNnz6d6uDq6iqVSvPy8trxuKherw8KCho2bNixY8d69OjRbB8bG5um+wLoIDcAy8Rh3XX0b4XdLs7e3h4hRM0Ca0RMTExERMSdO3cYQ13bdJBbKueOV2IwGCQSCf3Zulu3biGEHj582KadamxsVCgU7u7uhw8fbikx4G4CgaBNa7Y2UNQMWCYrr7vOBi4AjC++tyoxMVGj0Rw8eBBnFIzlQTbOVDXhMaVSWVdXl5qaSq3qrbfeOnr0qK+vL9UHT+HF+aQpXRycNwCLZc1119kYPnw4QojlRRuRSHT69GmhULh37156O5uD3CpTFWz/4osv7t69e+bMGTs7OyPd8CeCdx+0BHIDsFjWVne9rWQyWd++fRkVh4yQSqWMh8sQu4PcKjYF2+fNm0cQxKNHj1payaFDh7788subN286ODjQr00xBssihPDoWLlczjI8K8XhGCnzgDGslgfqrjOg9tboXrt2rY2NzdOnT/HL0tJS+peDt7d307csXryYMQbXyEFmWc6dZFGwPSAgQCQSNZ14nEK/H87AGAeMb0jgqdHbwUrGsFr+HkJusDxd5DPFuYHrKEiyA7mhsrLS3d1dqVSaPCTTevHihUAgaPZhhbbSaDQEQSQlJbV7DVaSG+CaEgDWSyKRpKenp6Sk4AnDuyaSJGNiYsRi8ebNmzu4qsLCwrCwMJVKNWfOHJPEZsEgNwBg1UaNGpWbm3vhwgWdTsd1LM0rKSkpLCzMzMxkOfDJiISEhLi4uLi4OJMEZtkgNwDQZhZWd93T0zMjI0MsFnMdSPNcXV2zs7OlUmnHV7V9+3Y4Y2AJnm8AoM1WrVq1atUqrqMAoBPBeQMAAAAmyA0AAACYIDcAAABggtwAAACAySruRRcVFcHU4ZYElwCCz5SO8QQy6DxWcqgJstNmDukiFApFSkoK11EAACyKxX9zWn5uAAAA0FZwvwEAAAAT5AYAAABMkBsAAAAwQW4AAADA9B+kFW+aov6RGwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "execution_count": 4
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Bây giờ ta tiến hành training model."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "model_1.fit(**hyperparameters_1)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 79).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 79).\n",
      "320/320 [==============================] - ETA: 0s - loss: 0.4508 - accuracy: 0.8185WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 79).\n",
      "320/320 [==============================] - 31s 86ms/step - loss: 0.4508 - accuracy: 0.8185 - val_loss: 0.3506 - val_accuracy: 0.8549\n",
      "Epoch 2/10\n",
      "320/320 [==============================] - 28s 86ms/step - loss: 0.2847 - accuracy: 0.8943 - val_loss: 0.3400 - val_accuracy: 0.8558\n",
      "Epoch 3/10\n",
      "320/320 [==============================] - 26s 80ms/step - loss: 0.2424 - accuracy: 0.9119 - val_loss: 0.3460 - val_accuracy: 0.8681\n",
      "Epoch 4/10\n",
      "320/320 [==============================] - 25s 79ms/step - loss: 0.2164 - accuracy: 0.9212 - val_loss: 0.3432 - val_accuracy: 0.8663\n",
      "Epoch 5/10\n",
      "320/320 [==============================] - 26s 80ms/step - loss: 0.1960 - accuracy: 0.9278 - val_loss: 0.3669 - val_accuracy: 0.8584\n",
      "Epoch 6/10\n",
      "320/320 [==============================] - 26s 82ms/step - loss: 0.1812 - accuracy: 0.9369 - val_loss: 0.3840 - val_accuracy: 0.8593\n",
      "Epoch 7/10\n",
      "320/320 [==============================] - 14s 44ms/step - loss: 0.1814 - accuracy: 0.9343 - val_loss: 0.3922 - val_accuracy: 0.8602\n",
      "Epoch 8/10\n",
      "320/320 [==============================] - 14s 44ms/step - loss: 0.1626 - accuracy: 0.9439 - val_loss: 0.4019 - val_accuracy: 0.8593\n",
      "Epoch 9/10\n",
      "320/320 [==============================] - 14s 44ms/step - loss: 0.1515 - accuracy: 0.9471 - val_loss: 0.4074 - val_accuracy: 0.8619\n",
      "Epoch 10/10\n",
      "320/320 [==============================] - 14s 44ms/step - loss: 0.1386 - accuracy: 0.9514 - val_loss: 0.4378 - val_accuracy: 0.8575\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "source": [
    "!tensorboard --logdir ./tb_log_sentiment --port 6006"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "2021-10-29 22:18:10.219562: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/cuda/include:/usr/lib/cuda/lib64:\n",
      "2021-10-29 22:18:10.219585: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "2021-10-29 22:18:11.176327: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2021-10-29 22:18:11.176710: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/cuda/include:/usr/lib/cuda/lib64:\n",
      "2021-10-29 22:18:11.176770: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/cuda/include:/usr/lib/cuda/lib64:\n",
      "2021-10-29 22:18:11.176824: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/cuda/include:/usr/lib/cuda/lib64:\n",
      "2021-10-29 22:18:11.178553: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/cuda/include:/usr/lib/cuda/lib64:\n",
      "2021-10-29 22:18:11.178620: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/cuda/include:/usr/lib/cuda/lib64:\n",
      "2021-10-29 22:18:11.178711: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1835] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.\n",
      "Skipping registering GPU devices...\n",
      "\n",
      "NOTE: Using experimental fast data loading logic. To disable, pass\n",
      "    \"--load_fast=false\" and report issues on GitHub. More details:\n",
      "    https://github.com/tensorflow/tensorboard/issues/4784\n",
      "\n",
      "Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all\n",
      "TensorBoard 2.7.0 at http://localhost:6006/ (Press CTRL+C to quit)\n",
      "^C\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "![](./images/12.png)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "> **Nhận xét**:\n",
    "> * Hai biểu đồ trên lần lượt thể hiện giá trị accuracy và loss value qua 10 epoch với đường màu xanh là của **validation data** và hồng là của **training data** được vẽ bằng TensorBoard. Có thể thấy kể từ epoch thứ 2 trở đi thì loss value có xu hướng đi xuống ở training data và đi lên ở valudation. Lúc này nhờ cơ thế CheckPoint của TensorFlow mà các weight tại thời điểm này được lưu lại và ta có thể kiểm tra sau này."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Tiếp theo, ta sẽ tokenizer cho test data để tiến hành đánh giá trên test data."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "prediction_1 = model_1.predict(X_test['normalize_comment'])\n",
    "prediction_1"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 79).\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input</th>\n",
       "      <th>output_proba</th>\n",
       "      <th>output_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bán hàng kỉ quấn đầy đủ chất lượn...</td>\n",
       "      <td>(4.148319e-06, 0.9999958)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>đặt xl áo ghi xl nhưng bé tí tẹo ngang ...</td>\n",
       "      <td>(0.9960705, 0.003929518)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>kẹp rất rất đẹp săn được giá sale h...</td>\n",
       "      <td>(0.809507, 0.19049297)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>không nghĩ áo đẹp giao hàng nhanh không ...</td>\n",
       "      <td>(0.22817881, 0.7718212)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>hôm nay mới nhận được nhưng nhìn qua...</td>\n",
       "      <td>(0.99997556, 2.4480925e-05)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2837</th>\n",
       "      <td>đặt dây buộc tóc nhưng chỉ nhận đưo...</td>\n",
       "      <td>(0.9999796, 2.0352623e-05)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2838</th>\n",
       "      <td>chất vải đẹp có vẻ mát giao hàng nhanh...</td>\n",
       "      <td>(0.0008592206, 0.99914074)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2839</th>\n",
       "      <td>dây nhìn tạm không đẹp mắc sai dây pha...</td>\n",
       "      <td>(0.9999949, 5.076978e-06)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2840</th>\n",
       "      <td>quần đẹp chất lượng ok nhưng chữ cu...</td>\n",
       "      <td>(0.9882366, 0.011763366)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2841</th>\n",
       "      <td>ok hơi chật tí</td>\n",
       "      <td>(0.0049098604, 0.9950901)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2842 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  input  \\\n",
       "0     bán hàng kỉ quấn đầy đủ chất lượn...   \n",
       "1     đặt xl áo ghi xl nhưng bé tí tẹo ngang ...   \n",
       "2     kẹp rất rất đẹp săn được giá sale h...   \n",
       "3     không nghĩ áo đẹp giao hàng nhanh không ...   \n",
       "4     hôm nay mới nhận được nhưng nhìn qua...   \n",
       "...                                                 ...   \n",
       "2837  đặt dây buộc tóc nhưng chỉ nhận đưo...   \n",
       "2838  chất vải đẹp có vẻ mát giao hàng nhanh...   \n",
       "2839  dây nhìn tạm không đẹp mắc sai dây pha...   \n",
       "2840  quần đẹp chất lượng ok nhưng chữ cu...   \n",
       "2841                                 ok hơi chật tí   \n",
       "\n",
       "                     output_proba  output_class  \n",
       "0       (4.148319e-06, 0.9999958)             1  \n",
       "1        (0.9960705, 0.003929518)             0  \n",
       "2          (0.809507, 0.19049297)             0  \n",
       "3         (0.22817881, 0.7718212)             1  \n",
       "4     (0.99997556, 2.4480925e-05)             0  \n",
       "...                           ...           ...  \n",
       "2837   (0.9999796, 2.0352623e-05)             0  \n",
       "2838   (0.0008592206, 0.99914074)             1  \n",
       "2839    (0.9999949, 5.076978e-06)             0  \n",
       "2840     (0.9882366, 0.011763366)             0  \n",
       "2841    (0.0049098604, 0.9950901)             1  \n",
       "\n",
       "[2842 rows x 3 columns]"
      ]
     },
     "metadata": {},
     "execution_count": 7
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "Model.confusionMatrix(y_test, prediction_1['output_class'])"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "    Negative       0.86      0.86      0.86      1410\n",
      "    Positive       0.87      0.87      0.87      1432\n",
      "\n",
      "    accuracy                           0.87      2842\n",
      "   macro avg       0.87      0.87      0.87      2842\n",
      "weighted avg       0.87      0.87      0.87      2842\n",
      "\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAEWCAYAAACZnQc8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAoVklEQVR4nO3dd5wV1fnH8c9XEKSqgDEKdlGjBruxa4INJLbYojFojNg1msQeTUxiihpjbBGxYIk1gr0Q1KAmqNgVNWL7CViwo6iUfX5/zFm44Ja7u3d27w7ft6953Zkzc+ecu6zPnnvmzDOKCMzMrBgWaesGmJlZ5Tiom5kViIO6mVmBOKibmRWIg7qZWYE4qJuZFYiDurWYpC6Sbpf0iaSbWnCe/STdV8m2tRVJW0p6ua3bYQsfeZ76wkPSvsBxwBrAdOBp4PcR8XALz7s/cBSwWUTMbmk7q52kAPpHxKS2bovZgtxTX0hIOg74K3AmsDSwPHARsEsFTr8C8L+FIaCXQ1LHtm6DLcQiwkvBF2Bx4DNgzwaO6UwW9Kem5a9A57RvG2Ay8HPgPeBt4MC07zfATGBWquMg4NfANSXnXhEIoGPaPgB4jezbwuvAfiXlD5e8bzPgceCT9LpZyb4Hgd8Cj6Tz3Af0qeez1bb/+JL27woMBv4HfAicXHL8xsB/gY/TsRcAndK+cemzfJ4+794l5z8BeAe4urYsvWeVVMf6aXtZYBqwTVv/bngp3uKe+sJhU2AxYFQDx5wCbAKsC6xDFthOLdn/TbI/Dn3JAveFkpaMiNPJev83RET3iLisoYZI6gb8DRgUET3IAvfTdRzXC7gzHdsb+Atwp6TeJYftCxwIfAPoBPyigaq/SfYz6AucBlwK/AjYANgS+JWkldKxc4BjgT5kP7uBwOEAEbFVOmad9HlvKDl/L7JvLcNKK46IV8kC/jWSugJXACMj4sEG2mvWLA7qC4fewPvR8PDIfsAZEfFeREwj64HvX7J/Vto/KyLuIuulrt7M9tQAa0vqEhFvR8QLdRyzE/BKRFwdEbMj4jrgJeD7JcdcERH/i4gvgBvJ/iDVZxbZ9YNZwPVkAfu8iJie6p9I9seMiHgiIsanet8ALgG2LuMznR4RX6X2zCciLgUmAY8Cy5D9ETWrOAf1hcMHQJ9GxnqXBd4s2X4zlc09xwJ/FGYA3ZvakIj4nGzI4lDgbUl3SlqjjPbUtqlvyfY7TWjPBxExJ63XBt13S/Z/Uft+SatJukPSO5I+Jfsm0qeBcwNMi4gvGznmUmBt4PyI+KqRY82axUF94fBf4CuyceT6TCUbOqi1fCprjs+BriXb3yzdGRH3RsR2ZD3Wl8iCXWPtqW3TlGa2qSkuJmtX/4joCZwMqJH3NDiNTFJ3susUlwG/TsNLZhXnoL4QiIhPyMaRL5S0q6SukhaVNEjSn9Nh1wGnSlpKUp90/DXNrPJpYCtJy0taHDipdoekpSXtksbWvyIbxqmp4xx3AatJ2ldSR0l7A2sCdzSzTU3RA/gU+Cx9izhsgf3vAis38ZznARMi4qdk1wr+3uJWmtXBQX0hERHnkM1RP5Vs5sVbwJHA6HTI74AJwLPAc8CTqaw5dY0BbkjneoL5A/EiqR1TyWaEbM3XgyYR8QEwhGzGzQdkM1eGRMT7zWlTE/2C7CLsdLJvETcssP/XwEhJH0vaq7GTSdoF2JF5n/M4YH1J+1WsxWaJbz4yMysQ99TNzArEQd3MrEAc1M3MCsRB3cysQKo28dDMt57xFVz7mq6rDG7rJlgVmj1zSmP3ETRq1vuvlR1zFu2zcovry0vVBnUzs1ZVM6fxY9oBB3UzM4Co6x649sdB3cwMoMZB3cysMMI9dTOzAplTjAd3eUqjmRlkF0rLXRoh6XJJ70l6vqTsLEkvSXpW0ihJS5TsO0nSJEkvS9qhpHzHVDZJ0onlfAwHdTMzyC6Ulrs07kqyJG6lxgBrR8QAsscongQgaU1gH2Ct9J6LJHWQ1AG4EBhElqH0h+nYBjmom5lBdqG03KURETGOLAtpadl9JQ+aGQ/0S+u7ANenp2a9TvaErI3TMikiXouImWRP7Gr0QfEO6mZmZBdKy10kDZM0oWQZ1ngN8/kJcHda70uWCrvW5FRWX3mDfKHUzAyaNKUxIoYDw5tTjaRTgNnAtc15f2Mc1M3MAObMyr0KSQeQPfxlYMx7mMUUYLmSw/ox77GN9ZXXy8MvZmZQ6QulXyNpR7IneO0cETNKdt0G7COps6SVgP7AY8DjQH9JK0nqRHYx9bbG6nFP3cwMKnpHqaTrgG2APpImA6eTzXbpDIyRBDA+Ig6NiBck3QhMJBuWOSIi5qTzHAncC3QALo+IFxqtu1ofZ+csjVYXZ2m0ulQiS+NXz48pO+Z0Xns7Z2k0M6tqzv1iZlYcUZP/hdLW4KBuZgbuqZuZFYqzNJqZFYiffGRmViDuqZuZFYjH1M3MCqQgD8lwUDczA/fUzcyKJN2Z3+45qJuZgXvqZmaF4tkvZmYF4p66mVmBePaLmVmBePjFzKxAPPxiZlYgDupmZgXi4RczswLxhVIzswIpyPDLInlXIGkFSdum9S6SeuRdp5lZk0VN+UsVyzWoSzoYuBm4JBX1A0bnWaeZWbPU1JS/VLG8h1+OADYGHgWIiFckfSPnOs3Mmq7Kg3W58g7qX0XETEkASOoIRM51mpk1XRQjNOUd1P8t6WSgi6TtgMOB23Ou08ys6WYXY/ZL3hdKTwSmAc8BhwB3AafmXKeZWdMV5EJp3j31XYGrIuLSnOsxM2uZgoyp591T/z7wP0lXSxqSxtTNzKpPRPlLFcs1qEfEgcCqwE3AD4FXJY3Is04zs2bxlMbyRMQsSXeTzXrpQjYk89O86zUza5IqD9blyjWoSxoE7A1sAzwIjAD2yrNOM7PmiDl+8HQ5fgzcABwSEV/lXJeZWfMVpKee95j6DyNitAO6mVW9Ck5plHS5pPckPV9S1kvSGEmvpNclU7kk/U3SJEnPSlq/5D1D0/GvSBpazsfIJahLeji9Tpf0ackyXdKnedRpZtYiNVH+0rgrgR0XKDsRGBsR/YGxaRtgENA/LcOAiyH7IwCcDnyHLN3K6bV/CBqSS1CPiC3Sa4+I6Fmy9IiInnnUaWbWIhWc/RIR44APFyjeBRiZ1keSTRqpLb8qMuOBJSQtA+wAjImIDyPiI2AMX/9D8TV5Z2m8upwyM7M2N2dO2YukYZImlCzDyqhh6Yh4O62/Ayyd1vsCb5UcNzmV1VfeoLwvlK5VupFuPtog5zrbhV+ddRHjHn2SXksszqgR5wBwziVX8+D4J1i0Y0eWW3ZpfvvLw+nZvRsffzKd4874C8+/PIlddtiGU446CIDPZ3zB0GNPm3vOd6d9yJBtt+SEww9oi49kFXbp8HPYafC2vDftfdZdbyAAAwasyUUX/JFu3bvy5puT2f/HRzJ9+mdstOG6XHzxnwGQxBm/PYdbb72nLZvf/jThQmlEDAeGN7eqiAhJudzFlNeY+kmSpgMDSsfTgXeBW/Oos73ZZYdtuPgPJ89XtukGAxg14hxuufRsVui3DCOuGwVAp06LcuQBe/OLQ/af7/huXbtw8yVnzV2WWboPA7fYuNU+g+XrqqtuZKch+81Xdsnfz+LkU85kvfW3ZfTou/nFzw8D4PkXXuI7mwxiw422Z6ch+3HxhX+iQ4cObdHs9quyY+p1eTcNq5Be30vlU4DlSo7rl8rqK29QXmPqf4iIHsBZC4yn946Ik/Kos73ZcMCaLN6j+3xlm224Dh3T/4jrfGs13p2WDcl17bIY6397DTp16lTv+d6YPJUPP/6UDb79rfwaba3qoYcf5cOPPp6vbLX+KzPuofEA/GvsQ+y222AAvvjiS+akedaLLdaZqPJb2atS/gm9bgNqZ7AMZV4H9zbgx2kWzCbAJ2mY5l5ge0lLpguk26eyBuU9pfGk1KCNJW1Vu+RZZ1GMuud+tth43bKPv/uB/7DjNptSm7veimnixP+x8847ALDHD4awXL9l5+7beKP1eObp+3n6ybEcfuSJc4O8lamCPXVJ1wH/BVaXNFnSQcAfge0kvQJsm7Yhy177GjAJuJQsRTkR8SHwW+DxtJyRyhqU94XSnwLjyP66/Ca9/rqB4+defBhx7c15Nq2qDb/2Fjp06MCQgVuW/Z57HniEQd/dIsdWWTX46bDjOOyQoTw6/m569OjGzJmz5u577PGnWGfd77HJZoM58fgj6dy5cxu2tP2Jmpqyl0bPld2js0xELBoR/SLisoj4ICIGRkT/iNi2NkCnWS9HRMQqEfHtiJhQcp7LI2LVtFxRzufI+0LpMcBGwPiI+K6kNYAz6zu49OLDzLeeWSi/P46+90H+Pf4JRpx1Wtm97pdffYM5c2pYa7WVc26dtbWXX36VQTvtC0D//iszeNDArx3z0kuT+OyzGay91uo88eSzrd3E9qsg32zyTr37ZUR8CSCpc0S8BKyec53t1sOPPc0VN9zK+b89gS6Lld/Luuv+Rxj0vc1zbJlVi6WW6g1kM1xOPukYLhmezRBeccXl5l4YXX75vqy++iq88eZb9Z7H6pD/hdJWkXdPfbKkJYDRwBhJHwFv5lxnu3D87//K489M5ONPpjNwn0M5YuhejLhuFDNnzWbYCb8FYMC3+nPaz7LprzvsdwSfzZjBrFmzuf+Rxxn+p1NZZYV+ANz77/9y0Zm+/lw011x9IVtvtSl9+vTijdcm8JszzqZ7924cdtgBAIwefRdXjrwBgM0335jjf3kEs2bNpqamhiOPPpkPPvioDVvfDhUk94ta6yq5pK2BxYF7ImJmY8cvrMMv1rCuqwxu6yZYFZo9c0qLZwh8fto+ZcecbmdcX7UzEvJOvdurZPO59OpgbWbVp8qfPVquvIdfniSbPP8RIGAJ4B1J7wIHR8QTOddvZlaeKh8rL1feF0rHAIMjok9E9CbLRnYH2TzMi3Ku28ysbDF7TtlLNcs7qG8SEXPvgIqI+4BNUyYyT6I1s+rh2S9leVvSCcD1aXtvsvwHHYBiDGCZWTEUZEw97576vmRJaEYDo8jG1/cFOuBnlZpZNXFPvXER8T5wlKRuEfH5Arsn5Vm3mVlTRJUH63LlnftlM0kTgRfT9jqSfIHUzKrP7DnlL1Us7+GXc8keyfQBQEQ8AzhLo5lVHw+/lCci3logMVV1/5kzs4VTlQfrcuUd1N+StBkQkhYly9r4Ys51mpk1WVEeLJJ3UD8UOI/sYalTgPuAI3Ku08ys6dxTb1ya/bJfoweambU1B/X6STqtgd0REb/No14zs+aK2cW4+SivnvqCc9IBugEHAb3JnrtnZlY9ihHT8wnqEXFO7bqkHmQXSA8kSxdwTn3vMzNrK0W5+Si3MfWUS/04sjH1kcD6EeFHsZhZdXJQr5+ks4DdyR4i/e2I+CyPeszMKqYgwy+N3lEqac80hIKkUyXdImn9Rt72c2BZ4FRgqqRP0zJd0qctb7aZWWVFTZS9VLNyeuq/ioibJG0BbAucBVwMfKe+N0RE3ukHzMwqKmZXd7AuVznBt/a2/p2A4RFxJ9ApvyaZmbWBmiYsVaycnvoUSZcA2wF/ktSZ/BOBmZm1qoI8I6Os4LwXcC+wQ0R8DPQCfplno8zMWl1BeuqNBvWImAG8B2yRimYDr+TZKDOz1hY15S/VrNHhF0mnAxsCqwNXAIsC1wCb59s0M7PWE7PbugWVUc6Y+m7AesCTABExtXaKo5lZUVR7D7xc5QT1mRERkgJAUrec22Rm1uqKEtTLuVB6Y5r9soSkg4F/AZfm2ywzs1YWKn+pYo321CPibEnbAZ+SjaufFhFjcm+ZmVkrKkpPvazcLymIO5CbWWFFTeV64JKOBX4KBPAcWZbaZcgy1fYGngD2j4iZ6d6fq4ANgA+AvSPijebWXU7ul+kluVu+lDTH+VvMrGhq5qjspSGS+gJHAxtGxNpAB2Af4E/AuRGxKvAR2fMlSK8fpfJz03HNVs489R4R0TMiegJdgB8AF7WkUjOzalPheeodgS6SOgJdgbeB7wE3p/0jgV3T+i5pm7R/oKRmf21o0u3+kRkN7NDcCs3MqlHUqOxF0jBJE0qWYXPPEzEFOBv4P7Jg/gnZcMvHEXNnw08G+qb1vsBb6b2z0/G9m/s5yrn5aPeSzUXIbkT6srkVmplVo2hCksaIGE72vIivkbQkWe97JeBj4CZgxxY3sEzlXCj9fsn6bOANsgabmRVGBS+Ubgu8HhHTACTdQnYH/hKSOqbeeD9gSjp+CrAcMDkN1yxOdsG0WcqZ0nhgc09uZtZeNHYBtAn+D9hEUlfgC2AgMAF4ANiDbAbMUODWdPxtafu/af/9EU353jC/eoO6pPPJpuPUKSKObm6lZmbVplI99Yh4VNLNZKlVZgNPkQ3V3AlcL+l3qeyy9JbLgKslTQI+JJsp02wN9dQntOTEZmbtSVTwTtGIOB04fYHi14CN6zj2S2DPStVdb1CPiJH17TMzK5qF5o5SSUsBJwBrAovVlkfE93Jsl5lZq6qp8pwu5Spnnvq1wItk03N+Qzb75fEc22Rm1uoiVPZSzcoJ6r0j4jJgVkT8OyJ+QnZnlJlZYVQqTUBbK2ee+qz0+raknYCpZM8pNTMrjEom9GpL5QT130laHPg5cD7QEzg211aZmbWyooyplxPUH42IT8jyEXw35/aYmbWJah8rL1c5Y+qPSLpP0kEpp4GZWeFElL9Us3JS764GnAqsBTwh6Q5JP8q9ZWZmragmVPZSzcpKvRsRj0XEcWR3Q33IvNy/ZmaFUFOjspdqVs7NRz2B3cjyEawCjKKOW13NzNqzau+Bl6ucC6XPAKOBMyLiv/k2Z56uqwxuraqsHfli6kNt3QQrqKJcKC0nqK/ckjSQZmbtwULTU3dAN7OFQVECXTk9dTOzwptT06RHNlctB3UzM6AgmXf95CMzM4Cg+GPqfvKRmS00agoyqO4nH5mZATULQU8d8JOPzGzhUJThFz/5yMwMmIPKXqqZn3xkZkY2+6XcpZr5yUdmZlR/sC6Xn3xkZkZxxtTLSRNwR1r1k4/MrLCqPKNu2cqZ/XIFddyElMbWzcwKYaGZ0gjcUbK+GFlu9an5NMfMrG3MaesGVEg5wy//LN2WdB3wcG4tMjNrAzVaeHrqC+oPfKPSDTEza0sFyRJQ1pj6dOb/vO+Q3WFqZlYYC82Uxojo0RoNMTNrS0WZ/dLoHaWSxpZTZmbWnhU+TYCkxST1AvpIWlJSr7SsCPRttRaambWCGpW/NEbSEpJulvSSpBclbZri5xhJr6TXJdOxkvQ3SZMkPStp/ZZ8joZ66ocATwBrpNfa5VbggpZUamZWbSqc++U84J6IWANYhywp4onA2IjoD4xN2wCDyCag9AeGARe35HM0lE/9POA8SUdFxPktqcTMrNpVavZLSquyFXAAQETMBGZK2gXYJh02EniQbNLJLsBVERHA+NTLXyYi3m5O/eVkaayRtERJg5eUdHhzKjMzq1ZNGX6RNEzShJJlWMmpVgKmAVdIekrSCEndgKVLAvU7wNJpvS/wVsn7J9OCIe5ygvrBEfFx7UZEfAQc3NwKzcyqUVOGXyJieERsWLIMLzlVR2B94OKIWA/4nHlDLZC9P8hpanw5Qb2DNO9WK0kdgE55NMbMrK3MUflLIyYDkyPi0bR9M1mQf1fSMgDp9b20fwqwXMn7+6WyZiknqN8D3CBpoKSBwHWpzMysMCp1oTQi3gHekrR6KhoITARuA4amsqFkk05I5T9Os2A2AT5p7ng6lJcm4ASyK7KHpe0xwKXNrdDMrBpV+I7So4BrJXUCXgMOJOtE3yjpIOBNYK907F3AYGASMCMd22zl3FFaA/w9LUjakuxhGUe0pGIzs2pSyQHuiHga2LCOXQPrODaoYDwtK6GXpPWAH5L9ZXkduKVSDTAzqwZFSRNQb1CXtBpZIP8h8D5wA6CI8NOPzKxwFoaEXi8BDwFDImISgCQ/m9TMCqkoD8loaPbL7sDbwAOSLk0zXwryBcXMbH6VzP3SluoN6hExOiL2Icv98gDwM+Abki6WtH0rtc/MrFVUOPdLm2l0nnpEfB4R/4iI75NNin8KPyTDzAommrBUsyY9zi6lCBieFjOzwqip+nBdnuY8o9TMrHCKcqHUQd3MjOofKy9XOblfmk3SapLGSno+bQ+QdGqedZqZNUfhZ79UyKXAScAsgIh4Ftgn5zrNzJqshih7qWZ5D790jYjHSjL3AszOuU4zsyar7lBdvryD+vuSViH9vCTtQXZDk5lZVSnKmHreQf0IsumPa0iaQpYMbL+c6zQza7I5Bemr5x3U34yIbdPz+RaJiOk512dm1ixF6annfaH0dUnDgU2Az3Kuy8ys2YpyoTTvoL4G8C+yYZjXJV0gaYuc6zQza7KipAnINahHxIyIuDEidgfWA3oC/86zTjOz5lhoEnq1lKStJV0EPAEsxrzn8pmZVY05RNlLNcv1QqmkN8iyOt4I/DIiPs+zPjOz5qr2sfJy5d1THxARu0XEdQ7o9bt0+DlMnfwMTz81dm7ZgAFr8vC423jqyX8xetSV9OjRfe6+E44/kpcmPswLz49j++22bosmW05OPfMvbLXTPuz6o0Pnlp19wQi+/8OD2e3Hh3H0SWfw6fT55xy8/c57bLTtblzxj5uz7XenceCRJ7DzfsPYZb9DuPrG0a35Edotj6k3QNLxafX3kv624JJHne3ZVVfdyE5D5p++f8nfz+LkU85kvfW3ZfTou/nFzw8D4Fvf6s9ee+3CgHW/x05D9uP8v53JIovkPopmrWTXwdvx97/8br6yTTdaj1FX/51RV13Misv1ZcTVN8y3/8/nD2fLTeY9uL5jhw788qiDue3a4fxj+Llcf8sdvPr6m63S/vbMs18a9mJ6nUA2lr7gYiUeevhRPvzo4/nKVuu/MuMeGg/Av8Y+xG67DQZg5+/vwI033srMmTN54423ePXVN9h4o/Vau8mWkw3X/TaL9+wxX9nm39mAjh07ADBgrTV497335+4bO+4/9F3mm6yy0gpzy5bq04s1V18VgG7durLyCsvx7rQPWqH17ZsvlDYgIm5PqzMiYmTpAszIo86imTjxf+y88w4A7PGDISzXb1kAll32m7w1eerc4yZPeZtl+36zTdporW/UnfexxaYbATBjxhdcfs1NHP6T+m/SnvL2u7z4yqsMWGv11mpiuxVN+K+a5f29/aQyywCQNEzSBEkTamoW7iH4nw47jsMOGcqj4++mR49uzJw5q62bZG3skpHX0aFDB4Zs/10ALrz8Gvbfeze6du1S5/EzZnzBsaf8jhOOPoTu3bq1ZlPbJc9+aYCkQcBgoO8CY+g9aSBLY0TMfVRex059q/snl7OXX36VQTvtC0D//iszeNBAAKZOfWdurx2gX99lmDrlnTZpo7We0XeOYdwjjzHib3+gNuvpcy+8zJgHHuYvF13G9M8+RxKdO3Vi3z12Ztbs2fzslN+x0/bfZbttNm/j1rcP1T6sUq68pjROJRtP35n5x9CnA8fmVGehLLVUb6ZN+wBJnHzSMVwy/GoAbr/jPq6+6kLO/etwll12aVZddSUee/ypNm6t5enh8RO4/B83ceUFf6bLYovNLb/q4rPnrl942TV07bIY++6xMxHBaX/4KyuvsBxD99m9LZrcLtVEMfqRuQT1iHgGeEbStRHh/OmNuObqC9l6q03p06cXb7w2gd+ccTbdu3fjsMMOAGD06Lu4cmQ242HixP9x882389wzDzB7zhyOPuYUamqK0sewX57+Rx5/6lk+/vhTBu76Iw4/aH9GXH0DM2fN4uCfnQJkF0tPP/6oes/x1LMvcPs9Y+m/yor8YOgRABxzyFC22mzjVvkM7VUxQjoocvjrJOnGiNhL0nPM/7MSEBExoLFzLOzDL1a3L6Y+1NZNsCq0aJ+VW/yQuX1X2K3smPOPN0dV7UPt8hp+OSa9Dsnp/GZmFVXts1rKldfwS+3Tjd4HvoiIGkmrkWVtvDuPOs3MWmJ2QYJ63lMaxwGLSeoL3AfsD1yZc51mZk3meerlUUTMAHYHLoqIPYG1cq7TzKzJKn1HqaQOkp6SdEfaXknSo5ImSbpBUqdU3jltT0r7V2zJ58g9qEvalOy5pHemsg4512lm1mQRUfZSpmOYlzIF4E/AuRGxKvARcFAqPwj4KJWfm45rtryD+s/I7iAdFREvSFoZeCDnOs3MmqySCb0k9QN2AkakbQHfA25Oh4wEdk3ru6Rt0v6Bqr3DrBlyzaceEf8G/i2pu6TuEfEacHSedZqZNUdTbv+XNAwYVlI0PN0RX+uvwPFAbXa23sDHJfftTAb6pvW+wFsAETFb0ifp+HmZ25og74dkfBu4CuiVbWoa8OOIeCHPes3MmqopKXVLU5osSNIQ4L2IeELSNhVpXBPkGtSBS4DjIuIBgPQBLwU2y7leM7MmqeCNmJsDO0saTPYIz57AecASkjqm3no/YEo6fgqwHDBZUkdgcaDZuZLzHlPvVhvQASLiQcDp4sys6lRq9ktEnBQR/SJiRWAf4P6I2I/seuIe6bChwK1p/ba0Tdp/f7TgL0zeQf01Sb+StGJaTgVey7lOM7Mma4V56icAx0maRDZmflkqvwzoncqPA05syefIe/jlJ8BvgFvIcsA8lMrMzKpKHo+pS6MTD6b114CvZVWLiC+BPStVZ1751BcDDgVWBZ4Dfh4RfsqDmVWtOVGMbKd59dRHArPIeuaDgG+RzVk3M6tK1X77f7nyCuprRsS3ASRdBjyWUz1mZhXhh2Q0bO5QS5pMn1M1ZmaVUYyQnl9QX0fSp2ldQJe0XfuQjJ451Wtm1ix5XChtC3nlU3fSLjNrVxzUzcwKxLNfzMwKxLNfzMwKpIK5X9qUg7qZGR5TNzMrFPfUzcwKZE7ZTx+tbg7qZmb4jlIzs0Lx7BczswJxT93MrEDcUzczKxD31M3MCsRpAszMCsTDL2ZmBRLuqZuZFYfTBJiZFYjTBJiZFYh76mZmBTKnxmPqZmaF4dkvZmYF4jF1M7MC8Zi6mVmBuKduZlYgvlBqZlYgHn4xMysQD7+YmRWIU++amRVIUeapL9LWDTAzqwY1EWUvDZG0nKQHJE2U9IKkY1J5L0ljJL2SXpdM5ZL0N0mTJD0raf2WfA4HdTMzoCZqyl4aMRv4eUSsCWwCHCFpTeBEYGxE9AfGpm2AQUD/tAwDLm7J53BQNzMju1Ba7tLIed6OiCfT+nTgRaAvsAswMh02Etg1re8CXBWZ8cASkpZp7udwUDczo2lBXdIwSRNKlmF1nVPSisB6wKPA0hHxdtr1DrB0Wu8LvFXytsmprFl8odTMDJp0mTQihgPDGzpGUnfgn8DPIuJTSaXvD0m5XJmt2qA+e+YUNX7UwkHSsPRLZDaXfy8qq5IxR9KiZAH92oi4JRW/K2mZiHg7Da+8l8qnAMuVvL1fKmsWD7+0D3V+tbOFnn8vqpCyLvllwIsR8ZeSXbcBQ9P6UODWkvIfp1kwmwCflAzTNFnV9tTNzNqpzYH9geckPZ3KTgb+CNwo6SDgTWCvtO8uYDAwCZgBHNiSylWUW2OLTNKEiNiwrdth1cW/F1YXD7+0Dx43tbr498K+xj11M7MCcU/dzKxAHNTNzArEQb3CJIWkc0q2fyHp1znUc/IC2/+pdB2WD0lzJD0t6XlJN0nq2sT3Lyvp5rS+rqTBJft2lnRi/e+2onNQr7yvgN0l9cm5nvmCekRslnN9VjlfRMS6EbE2MBM4tClvjoipEbFH2lyXbDpc7b7bIuKPFWuptTsO6pU3m2xWwrEL7pC0lKR/Sno8LZuXlI9JaTpHSHqz9o+CpNGSnkj7hqWyPwJdUm/v2lT2WXq9XtJOJXVeKWkPSR0knZXqfVbSIbn/JKwcDwGrprSso9O/zXhJAwAkbZ3+nZ+W9JSkHpJWTL38TsAZwN5p/96SDpB0gaTF0+/RIuk83SS9JWlRSatIuif9Xj0kaY02/PxWaU1JYuOlrOxtnwE9gTeAxYFfAL9O+/4BbJHWlye74wzgAuCktL4jWRqKPmm7V3rtAjwP9K6tZ8F60+tuwMi03oksUVAXsrsPT03lnYEJwEpt/fNaGJeSf6uOZHcVHgacD5yeyr8HPJ3Wbwc2T+vd03tWBJ5PZQcAF5Sce+52Ovd30/rewIi0Phbon9a/A9zf1j8TL5VbfEdpDiJL3nMVcDTwRcmubYE1SxL79ExJf7YgC8ZExD2SPip5z9GSdkvry5HlXP6ggervBs6T1JnsD8S4iPhC0vbAAEm1X9sXT+d6vbmf05qtS8mdhg+R3VL+KPADgIi4X1JvST2BR4C/pG9kt0TE5NLEUI24gSyYPwDsA1yUft82A24qOU/nln8kqxYO6vn5K/AkcEVJ2SLAJhHxZemB9f1PKmkbsj8Em0bEDEkPAos1VGlEfJmO24Hsf+jra08HHBUR9zbtY1gOvoiIdUsL6vsdiIg/SrqTbNz8EUk7AF/WefDX3QacKakXsAFwP9AN+HjB+q04PKaek4j4ELgROKik+D7gqNoNSeum1UdIeSBSj3rJVL448FEK6GuQPUWl1qyUCa4uN5Dlj9gSuCeV3QscVvseSatJ6ta8T2c5eAjYD+b+MX8/feNbJSKei4g/AY8DC45/Twd61HXCiPgsvec84I6ImBMRnwKvS9oz1SVJ6+TxgaxtOKjn6xygdBbM0cCG6WLYRObNevgNsL2k54E9yRLoTycLyB0lvUiWDGh8ybmGA8/WXihdwH3A1sC/ImJmKhsBTASeTPVcgr+pVZNfAxtIepbs37o2m9/P0kXRZ4FZZMNrpR4gG9J7WtLedZz3BuBH6bXWfsBBkp4BXiB78o4VhNMEVIE0/j0nImZL2hS42F+Pzaw53FOrDsuTpeRchGze8sFt3B4za6fcUzczKxCPqZuZFYiDuplZgTiom5kViIO6zaelGQQXONeVtXewppw2azZw7DaSmpyUTNIblUieVqnzmLU1B3VbUIMZBCU1a8ZURPw0IiY2cMg2ZLevm1kLOKhbQ2ozCG6TsvndBkysL+NjujvxAkkvS/oX8I3aE0l6UNKGaX1HSU9KekbSWEkrkv3xODZ9S9hS9We07C3pPqWMlmTpD+Yj6VBJZ5VsHyDpgrT+tayXC7x3xXRzVu323Hz49WU3lLRn+mbzjKRxLf2hm7WE56lbnVKPfBDz0gysD6wdEa+nYPhJRGyUbpx6RNJ9wHrA6sCawNJkd7BevsB5lwIuBbZK5+oVER9K+jtZ9sKz03H/AM6NiIclLU+W5uBbwOnAwxFxhrIUw6VpGGr9E/gv8Mu0vTfw+7T+k1RfF+BxSf+MiIYSpJUaDhwaEa9I+g5wEVlGxdOAHSJiiqQlyjyXWS4c1G1BdWUQ3Ax4LCJqMzrWl/FxK+C6iJgDTJV0fx3n34Qsc+TrMDdHTl3qy2i5FbB7eu+dmj+jJal8mqTXJG0CvEKWL+WRtLupWS8BUMPZDR8BrpR0I3BLY+cyy5ODui2ovgyCn5cWUUfGR5U8Vq0CmpTRsg7XkyVJewkYFRGh8rJezmb+Ycna/YtQT3bDiDg09dx3Ap6QtEETev9mFeUxdWuO+jI+jiN7Ck8HScsA363jveOBrSStlN7bK5UvmG2wvoyW44B9U9kg5mW0XNAoskRVP2Re+uGGsl7Wehf4Rhq77wwMgSxHPvVkN1SWSfHRiDgNmEb2DcCsTTioW3PUl/FxFNlwx0TgKrJx7flExDSypzDdoixLYG32wNuB3WovlNJwRsutJL1ANgzzf3U1MCI+Al4EVoiIx1JxQ1kva983i+wRcY8BY8h6+rXqy254lqTn0s/iP8Azdf/YzPLn3C9mZgXinrqZWYE4qJuZFYiDuplZgTiom5kViIO6mVmBOKibmRWIg7qZWYH8P66PlIt5yeF2AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     }
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "> **Nhận xét**:\n",
    "> * Ta có thể thấy rõ ràng rằng dù độ chính xác của model LSTM khá cao hơn 85%, với việc để các hyper-params cho TensorFlow tự quyết thì model có khả năng dự đoán cân bằng trên cả hai class là negative và postive, ta không cần phải kiểm tra lại bằng ROC-AUC."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Do ta có áp dụng checkpoint trong model để lưu lại các bộ tham số tốt nhất trong quá trình training, ta có thể lấy bộ tham số mà có validation loss thấp nhất để update trọng số cho model."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "source": [
    "model_uw_1 = DeepModel.SentimentLSTM()\n",
    "model_uw_1.define(**hyperparameters_1)\n",
    "model_uw_1.model.load_weights(\"./models/lstm_model_comments_1/weights/02-0.340042.h5\")"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "source": [
    "prediction_uw_1 = model_uw_1.predict(X_test['normalize_comment'])\n",
    "prediction_uw_1"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 79).\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input</th>\n",
       "      <th>output_proba</th>\n",
       "      <th>output_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bán hàng kỉ quấn đầy đủ chất lượn...</td>\n",
       "      <td>(0.00073411025, 0.99926585)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>đặt xl áo ghi xl nhưng bé tí tẹo ngang ...</td>\n",
       "      <td>(0.9971385, 0.002861474)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>kẹp rất rất đẹp săn được giá sale h...</td>\n",
       "      <td>(0.9313881, 0.068611994)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>không nghĩ áo đẹp giao hàng nhanh không ...</td>\n",
       "      <td>(0.011878509, 0.9881215)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>hôm nay mới nhận được nhưng nhìn qua...</td>\n",
       "      <td>(0.9889897, 0.011010231)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2837</th>\n",
       "      <td>đặt dây buộc tóc nhưng chỉ nhận đưo...</td>\n",
       "      <td>(0.9959667, 0.0040333555)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2838</th>\n",
       "      <td>chất vải đẹp có vẻ mát giao hàng nhanh...</td>\n",
       "      <td>(0.010884253, 0.9891158)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2839</th>\n",
       "      <td>dây nhìn tạm không đẹp mắc sai dây pha...</td>\n",
       "      <td>(0.99751866, 0.002481401)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2840</th>\n",
       "      <td>quần đẹp chất lượng ok nhưng chữ cu...</td>\n",
       "      <td>(0.91258466, 0.08741527)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2841</th>\n",
       "      <td>ok hơi chật tí</td>\n",
       "      <td>(0.23734112, 0.7626589)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2842 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  input  \\\n",
       "0     bán hàng kỉ quấn đầy đủ chất lượn...   \n",
       "1     đặt xl áo ghi xl nhưng bé tí tẹo ngang ...   \n",
       "2     kẹp rất rất đẹp săn được giá sale h...   \n",
       "3     không nghĩ áo đẹp giao hàng nhanh không ...   \n",
       "4     hôm nay mới nhận được nhưng nhìn qua...   \n",
       "...                                                 ...   \n",
       "2837  đặt dây buộc tóc nhưng chỉ nhận đưo...   \n",
       "2838  chất vải đẹp có vẻ mát giao hàng nhanh...   \n",
       "2839  dây nhìn tạm không đẹp mắc sai dây pha...   \n",
       "2840  quần đẹp chất lượng ok nhưng chữ cu...   \n",
       "2841                                 ok hơi chật tí   \n",
       "\n",
       "                     output_proba  output_class  \n",
       "0     (0.00073411025, 0.99926585)             1  \n",
       "1        (0.9971385, 0.002861474)             0  \n",
       "2        (0.9313881, 0.068611994)             0  \n",
       "3        (0.011878509, 0.9881215)             1  \n",
       "4        (0.9889897, 0.011010231)             0  \n",
       "...                           ...           ...  \n",
       "2837    (0.9959667, 0.0040333555)             0  \n",
       "2838     (0.010884253, 0.9891158)             1  \n",
       "2839    (0.99751866, 0.002481401)             0  \n",
       "2840     (0.91258466, 0.08741527)             0  \n",
       "2841      (0.23734112, 0.7626589)             1  \n",
       "\n",
       "[2842 rows x 3 columns]"
      ]
     },
     "metadata": {},
     "execution_count": 10
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "source": [
    "Model.confusionMatrix(y_test, prediction_uw_1['output_class'])"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "    Negative       0.85      0.90      0.87      1410\n",
      "    Positive       0.89      0.84      0.87      1432\n",
      "\n",
      "    accuracy                           0.87      2842\n",
      "   macro avg       0.87      0.87      0.87      2842\n",
      "weighted avg       0.87      0.87      0.87      2842\n",
      "\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAEWCAYAAACZnQc8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAApCklEQVR4nO3dd5wV1fnH8c8XUEQUEFAU0FiCMWpsGLtoolGxBPWnsceoEY1GE01iRI2oSYwmdo1GwCj2jo1YiL3Egg2xE0sEC6IoNgR2n98fcxYvsOXucmf3MnzfvuZ1Z87MnXPusj539syZ5ygiMDOzYmjX1g0wM7PKcVA3MysQB3UzswJxUDczKxAHdTOzAnFQNzMrEAd1m2+SOkm6XdKnkm6Yj/PsI+meSratrUjaXNKrbd0OW/jI49QXHpL2Bo4GVgM+A54D/hwRj8znefcDjgA2iYhZ89vOaicpgH4RMaGt22I2N1+pLyQkHQ2cA5wK9AJWAC4EBlXg9N8CXlsYAno5JHVo6zbYQiwivBR8AboCnwO7N3JMR7Kg/25azgE6pn1bAhOB3wCTgfeAA9K+k4EZwMxUx0HAScCVJedeEQigQ9r+GfAG2V8LbwL7lJQ/UvK+TYCngE/T6yYl+x4A/gg8ms5zD9Czgc9W1/5jStq/M7A98BrwMXBcyfEbAP8BPknHXgAsmvY9lD7LF+nz7lFy/t8D7wNX1JWl96yS6lgvbfcGPgS2bOvfDS/FW3ylvnDYGFgMGNXIMccDGwHrAGuTBbYTSvYvS/bl0IcscP9d0lIRMZTs6v+6iFgiIi5prCGSOgPnAQMjYkmywP1cPcd1B0anY3sAZwGjJfUoOWxv4ABgGWBR4LeNVL0s2c+gD3AiMBzYF+gPbA78QdJK6dga4CigJ9nPbivgMICIGJCOWTt93utKzt+d7K+WwaUVR8R/yQL+lZIWBy4FRkbEA42016xFHNQXDj2AKdF498g+wCkRMTkiPiS7At+vZP/MtH9mRPyL7Cr1Oy1sTy2wpqROEfFeRLxYzzE7AK9HxBURMSsirgFeAXYqOebSiHgtIr4Crif7QmrITLL7BzOBa8kC9rkR8Vmq/yWyLzMi4umIeDzV+xZwMbBFGZ9paER8ndozh4gYDkwAngCWI/sSNas4B/WFw0dAzyb6ensDb5dsv53KZp9jri+FL4ElmtuQiPiCrMviUOA9SaMlrVZGe+ra1Kdk+/1mtOejiKhJ63VB94OS/V/VvV/SqpLukPS+pGlkf4n0bOTcAB9GxPQmjhkOrAmcHxFfN3GsWYs4qC8c/gN8TdaP3JB3yboO6qyQylriC2Dxku1lS3dGxN0R8SOyK9ZXyIJdU+2pa9OkFrapOS4ia1e/iOgCHAeoifc0OoxM0hJk9ykuAU5K3UtmFeegvhCIiE/J+pH/LmlnSYtLWkTSQEl/TYddA5wgaWlJPdPxV7awyueAAZJWkNQVGFK3Q1IvSYNS3/rXZN04tfWc41/AqpL2ltRB0h7A6sAdLWxTcywJTAM+T39F/GKu/R8AKzfznOcCYyPi52T3Cv4x3600q4eD+kIiIs4kG6N+AtnIi3eAXwK3pEP+BIwFxgEvAM+kspbUNQa4Lp3raeYMxO1SO94lGxGyBfMGTSLiI2BHshE3H5GNXNkxIqa0pE3N9Fuym7Cfkf0Vcd1c+08CRkr6RNJPmjqZpEHAdnzzOY8G1pO0T8VabJb44SMzswLxlbqZWYE4qJuZFYiDuplZgTiom5kVSNUmHpo55Q3fwbV5dOq9eVs3warQrBmTmnqOoEnNiTmL9Fx5vuvLS9UGdTOzVlVb0/QxCwAHdTMzgKjvGbgFj4O6mRlArYO6mVlhhK/UzcwKpKYYE3c5qJuZgW+UmpkVirtfzMwKxDdKzcyKoyg3Sp0mwMwMsiv1cpcmSPqnpMmSxpeU/U3SK5LGSRolqVvJviGSJkh6VdK2JeXbpbIJko4t52M4qJuZAdTMLH9p2mVkE6OUGgOsGRFrAa+RZgSTtDqwJ7BGes+FktpLag/8HRhINuvXXunYRjmom5lBdqO03KWpU0U8RDazV2nZPSWTtz8O9E3rg4BrI+LriHgTmABskJYJEfFGRMwArk3HNspB3cwMmtX9ImmwpLEly+Bm1nYgcGda70M2vWSdiamsofJG+UapmRk0a0hjRAwDhrWkGknHA7OAq1ry/qY4qJuZQasMaZT0M7IJ1beKbyaIngQsX3JY31RGI+UNclA3MwOitqwboC0maTvgGGCLiPiyZNdtwNWSzgJ6A/2AJwEB/SStRBbM9wT2bqoeB3UzM6jolbqka4AtgZ6SJgJDyUa7dATGSAJ4PCIOjYgXJV0PvETWLXN4RNSk8/wSuBtoD/wzIl5ssu5v/gKoLp75yOrjmY+sPpWY+Wj607eUHXMW67+zZz4yM6tqTuhlZlYgBUkT4KBuZgZO6GVmViieJMPMrEB8pW5mVhxpFOECz0HdzAx8pW5mVige/WJmViC+UjczKxCPfjEzKxB3v5iZFYi7X8zMCsRB3cysQNz9YmZWIL5RamZWIAXpfmmXdwWSviVp67TeSdKSeddpZtZsUVv+UsVyDeqSDgZuBC5ORX2BW/Ks08ysRWpry1+qWN7dL4cDGwBPAETE65KWyblOM7Pmq/JgXa68g/rXETEjTbKKpA6A5x41s+pTpfM1N1feQf1BSccBnST9CDgMuD3nOs3Mmm9WMUa/5H2j9FjgQ+AF4BDgX8AJOddpZtZ8BblRmveV+s7A5RExPOd6zMzmT0H61PO+Ut8JeE3SFZJ2TH3qZmbVJ6L8pYrlGtQj4gDg28ANwF7AfyWNyLNOM7MW8ZDG8kTETEl3ko166UTWJfPzvOs1M2uWKg/W5co1qEsaCOwBbAk8AIwAfpJnnWZmLRE1nni6HD8FrgMOiYivc67LzKzlfKXetIjYK8/zm5lVTJUPVSxXLkFd0iMRsZmkz5jzCVIBERFd8qjXzKzFaqt7VEu5chn9EhGbpdclI6JLybKkA7qZVaUKjn6R9E9JkyWNLynrLmmMpNfT61KpXJLOkzRB0jhJ65W8Z/90/OuS9i/nY+SdpfGKcsrMzNpcTU35S9MuA7abq+xY4N6I6Afcm7YBBgL90jIYuAiyLwFgKLAhWWLEoXVfBI3J++GjNUo30sNH/XOuc4FwwqlnMWCHPdl530Nnl51xwQh22utgdvnpLzhyyClM++zz2ftenfAm+ww+ikH7HMIu+/2Cr7+eAcDMmTM56fRz2WHPn7PTXgcz5v5HWv2zWD6GDzuTdyc+z3PP3jvPvqN+fQizZkyiR485/x9fv//aTP/ybXbddYfWamZxVPBKPSIeAj6eq3gQMDKtjyQb3l1XfnlkHge6SVoO2BYYExEfR8RUYAzzflHMI5egLmlI6k9fS9K0tHwGfADcmkedC5qdt/8R/zjrT3OUbfz9dRl1xT8YdflFrLh8H0ZccR0As2bVcOwpf+UPvzuCW6+6mEsvOJ0OHdoDcPHIa+m+VDdGXzuCW6+6mPXX/V6rfxbLx+WXX88OO+4zT3nfvr350dYDePvtiXOUt2vXjr+cejxjxjzYWk0sltooe5E0WNLYkmVwGTX0ioj30vr7QK+03gd4p+S4iamsofJG5dWn/peIWBL421z96T0iYkgedS5o1l/ne3TtMuckUJtu2H92sF5rjdX4YPIUAB578mlWXWUlVuu3MgDdunahffvsuFGj7+Hn++0BZP9TL9Wta2t9BMvZw488wcdTP5mn/MwzTuLY4/5MzPW4+i8PP5CbR41m8ocftVILC6YZCb0iYlhErF+yDGtWVdk/Xi53ZvMe0jgk9QH1AxYrKX8oz3qLYNToe9huqy0AePudSUhi8FHHM/WTTxm49RYcuM/us7tnLhh+OU89O47l+yzHcUcfRs/uTXa72QJqp522YdKk9xg37qU5ynv3XpadB23HVj/anRHrr9M2jVvQ5T/65QNJy0XEe6l7ZXIqnwQsX3Jc31Q2iezBzdLyB5qqJO8bpT8HHgLuBk5Oryc1cvzsP2lGXH5Nnk2rahePvIb27duz4zY/AGBWTQ3PjnuR04cew+UXncG9Dz7G42Ofpaamhg8mT2Gd732XGy69gLXX/C5nXODUOkXVqdNiDPn9EZx08hnz7DvrzJMZctyp81y9W/mitrbspYVuA+pGsOzPN13RtwE/TaNgNgI+Td00dwPbSFoqXRxvk8oalfcTpb8Cvg88HhE/kLQacGpDB6c/YYYBzJzyxkL523nL6DE89OiTjDjvL9TNGNVrmZ70X3vN2V0rm2/8fV569b9s2H8dOi3Wka232BSAbX6wOTff3uS/uS2gVlllRVZccQWeGTsGgL59l+OpJ+5m4013oP96a3HVlRcC0LNndwZu90NmzZrFbbf596FsFUwTIOkasqvsnpImko1iOQ24XtJBwNt8kzLlX8D2wATgS+AAgIj4WNIfgafScadExNw3X+eRd1CfHhHTJSGpY0S8Iuk7Ode5wHrk8bH88+obuOyCv9Jpsdm9VWy6QX8uvepGvpo+nUU6LMLY515gvz12QRJbbLohTz07jg37r8MTY59jlZVWaMNPYHkaP/4Vevdde/b2hNceZ8ONB/LRR1Pp952NZ5dfMuJsRv/r3w7ozVXB7pdGnqbfqp5jg2w+5/rO80/gn82pO++gPlFSN+AWYIykqWTfUAu93w09jaeeHccnn0xjq5335bCD9mPEFdcxY+ZMDv718UB2s3ToMUfQtcuS/HTPXdnzoF8hic03/j5bbLIBAEcfdiBDTjmD0869mO7duvKn445uy49lFXTlFX9niwEb07Nnd956Yywnn3IGl152bVs3q7gKkvtFrdUHJ2kLoCtwV0TMaOr4hbX7xRrXqffmbd0Eq0KzZkzS/J7jixP3LDvmdD7l2vmuLy95p97tXrL5Qnp1sDaz6uOEXmV5hmyozlSyZF7dgPclfQAcHBFP51y/mVl5nNCrLGOA7SOiZ0T0IMtxcAdwGHBhznWbmZUtZtWUvVSzvIP6RhEx+xZ8RNwDbJzyG3TMuW4zs/I1I01ANcu7++U9Sb8H6m7Z70H2VFV7oBgdWGZWDAXpU8/7Sn1vskdbbwFGkfWv7w20x3OVmlk18ZV60yJiCnCEpM4R8cVcuyfkWbeZWXNElQfrcuWd+2UTSS8BL6fttSX5BqmZVZ9ZNeUvVSzv7pezyRK9fwQQEc8DA3Ku08ys+dz9Up6IeKcuMVVS3V9zZrZwqvJgXa68g/o7kjYBQtIiZFkbX865TjOzZitK2uK8g/qhwLlkUzBNAu6hgWxkZmZtylfqTUujX+adZNHMrNo4qDdM0omN7I6I+GMe9ZqZtVTMKsbDR3ldqc89Jh2gM3AQ0ANwUDez6lKMmJ5PUI+IM+vWJS1JdoP0ALJ0AWc29D4zs7ZSlIePcutTT7nUjybrUx8JrBcRU/Oqz8xsvjioN0zS34BdySaR/l5EfJ5HPWZmFVOQ7pcmnyiVtHvqQkHSCZJulrReE2/7DdAbOAF4V9K0tHwmadr8N9vMrLKiNspeqlk5V+p/iIgbJG0GbA38DbgI2LChN0RE3ukHzMwqKmZVd7AuVznBt+6x/h2AYRExGlg0vyaZmbWB2mYsVaycK/VJki4GfgScLqkj+ScCMzNrVQWZI6Os4PwT4G5g24j4BOgO/C7PRpmZtbqCXKk3GdQj4ktgMrBZKpoFvJ5no8zMWlvUlr9Usya7XyQNBdYHvgNcCiwCXAlsmm/TzMxaT8xq6xZURjl96rsA6wLPAETEu3VDHM3MiqLar8DLVU5QnxERISkAJHXOuU1mZq2uKEG9nBul16fRL90kHQz8Gxieb7PMzFpZqPylipVzo/QM4EbgJrJ+9RMj4vy8G2Zm1poqeaNU0lGSXpQ0XtI1khaTtJKkJyRNkHSdpEXTsR3T9oS0f8X5+RxljTePiDER8buI+G1EjJmfCs3MqlHUquylMZL6AEcC60fEmkB7YE/gdODsiPg2MJUsFTnpdWoqPzsd12Ll5H75rCR3y3RJNc7fYmZFU1ujspcydAA6SeoALA68B/yQrNcDssy1O6f1QWmbtH8rSS3u42nyRmlEzB7pkioaBGzU0grNzKpRc26UShoMDC4pGhYRwwAiYpKkM4D/AV+Rzc38NPBJxOyBkxPJ5m4mvb6T3jtL0qdkkwlNacnnaNbj/pG5Bdi2JZWZmVWr5nS/RMSwiFi/ZBlWdx5JS5Fd/K5Elq22M7Bda32Och4+2rVksx3Zg0jTc2uRmVkbiMoladwaeDMiPgSQdDPZw5rdJHVIV+t9gUnp+EnA8sDE1F3TFfiopZWXM059p5L1WcBbZN9CZmaF0dQN0Gb4H7CRpMXJul+2AsYC9wO7kU3ruT9wazr+trT9n7T/voiWf8WU06d+QEtPbma2oCjzBmiTIuIJSTeSPYU/C3iWbBa40cC1kv6Uyi5Jb7kEuELSBOBjspEyLdZgUJd0PtDgt0VEHDk/FZuZVZMKXqkTEUOBoXMVvwFsUM+x04HdK1V3Y1fqYytViZlZtYsqf1K0XA0G9YgY2dA+M7OiKUrul3JGvywN/B5YHVisrjwifphju8zMWlVtQa7UyxmnfhXwMtmYy5PJRr88lWObzMxaXYTKXqpZOUG9R0RcAsyMiAcj4kCyx13NzAqjwmkC2kw549Rnptf3JO0AvEs2T6mZWWFUcvRLWyonqP9JUlfgN8D5QBfgqFxbZWbWyorSp15OUH8iIj4FPgV+kHN7zMzaRLX3lZernD71RyXdI+mglKjGzKxwIspfqlk5Mx+tCpwArAE8LekOSfvm3jIzs1ZUGyp7qWblznz0ZEQcTfaI68d8k9DdzKwQamtV9lLNynn4qAuwC1mSmVWAUdSTv8DMbEFW7Vfg5SrnRunzwC3AKRHxn3yb842VVv1xa1VlC5Avxl3d1k2wgirKjdJygvrK85Pb18xsQbDQXKk7oJvZwqAoga6cK3Uzs8KrqW3WlM1Vy0HdzAwoSOZdz3xkZgYQFL9P3TMfmdlCo7Ygneqe+cjMDKhdCK7UAc98ZGYLh6J0v3jmIzMzoAaVvVQzz3xkZkY2+qXcpZp55iMzM6o/WJfLMx+ZmVGcPvVy0gTckVY985GZFVaVZ9QtWzmjXy6lnoeQUt+6mVkhLDRDGoE7StYXI8ut/m4+zTEzaxs1bd2ACimn++Wm0m1J1wCP5NYiM7M2UKtiXKm3JC1ZP2CZSjfEzKwtRTOWpkjqJulGSa9IelnSxpK6Sxoj6fX0ulQ6VpLOkzRB0jhJ683P52gyqEv6TNK0ugW4newJUzOzwqjwOPVzgbsiYjVgbbIHOI8F7o2IfsC9aRtgINnFcj9gMHDR/HyOcrpflpyfCszMFgSVGv2ShoAPAH4GEBEzgBmSBgFbpsNGAg+QXSAPAi5PExI9nq7yl4uI91pSfzlX6veWU2ZmtiBrTpoASYMljS1ZBpecaiXgQ+BSSc9KGiGpM9CrJFC/D/RK632Ad0rePzGVtUhj+dQXAxYHeqa+n7rvsS7zU6GZWTVqzpV6RAwDhjWwuwOwHnBERDwh6Vy+6Wqpe39IyiXZb2PdL4cAvwZ6A0/zTVCfBlyQR2PMzNpKBdMETAQmRsQTaftGsqD+QV23iqTlgMlp/yRg+ZL3901lLdJg90tEnBsRKwG/jYiVI2KltKwdEQ7qZlYolRr9EhHvA+9I+k4q2gp4CbgN2D+V7Q/cmtZvA36aRsFsBHza0v50KO/ho1pJ3SLiE4DUFbNXRFzY0krNzKpNhdMEHAFcJWlR4A3gALKL6OslHQS8DfwkHfsvYHtgAvBlOrbFygnqB0fE3+s2ImKqpIMBB3UzK4xKZmmMiOeA9evZtVU9xwZweKXqLieot5ekVDGS2gOLVqoBZmbVoKYYD5SWFdTvAq6TdHHaPiSVmZkVxsKUT/33ZE85/SJtjwGG59YiM7M2UJSg3uTDRxFRGxH/iIjdImI3sru45+ffNDOz1lPJ3C9tqZwrdSStC+xFdrf2TeDmPBtlZtbaCj9JhqRVyQL5XsAU4DpAEeHZj8yscIrS/dLYlforwMPAjhExAUCS5yY1s0IqyiQZjfWp7wq8B9wvabikraAg8z2Zmc2lVuUv1ayxNAG3RMSewGrA/WR5YJaRdJGkbVqpfWZmraLC+dTbTDmjX76IiKsjYieyRDPP4kkyzKxgFqrRL3UiYipZusmGUk6amS2Qaqs+XJenWUHdzKyoinKj1EHdzIzq7ysvV5N96vND0qqS7pU0Pm2vJemEPOs0M2uJwo9+qZDhwBBgJkBEjAP2zLlOM7NmqyXKXqpZ3t0vi0fEk9IcX22zcq7TzKzZqjtUly/voD5F0iqkn5ek3cgeaDIzqypF6VPPO6gfTjb8cTVJk8iSge2Tc51mZs1WU5Br9byD+tsRsbWkzkC7iPgs5/rMzFqkKFfqed8ofVPSMGAj4POc6zIza7Gi3CjNO6ivBvybrBvmTUkXSNos5zrNzJqtKGkCcg3qEfFlRFwfEbsC6wJdgAfzrNPMrCUWmoRe80vSFpIuBJ4GFiObPcnMrKrUEGUv1SzXG6WS3iLL6ng98LuI+CLP+szMWqra+8rLlffol7UiYlrOdSzQluuzLOdeeCo9l+lBRHD1yBu55OIrOeHk37D1tlswc+Ys3n7zHY7+5QlMm/YZu+y2A4ceccDs9393jVXZbsvdeWn8q234KaxSTjzvMh4c+wLduy7JqPNPAuDMS2/kwaeeZ5EOHVh+2aU55cif0WWJxQEYceOdjBrzCO3atePYg/dk0/XWAGC7g4eweKeOtG/Xjvbt2nPtWce31UdaYBQjpGdzjlb+pNIxEfFXSedTz88qIo5s6hx9u69ZlJ9xo5bp1ZNlei3N+HEv03mJxbnzvus5aL8jWa73sjz60BPU1NRw3NBsFsFTTz57jveu9t1+jLjyPDbrP7Atmt4m/vvY39u6Cbka++JrLL5YR44/59LZQf2xZ19kg7VWo0P79pw98iYAjtr///jv/97l92eO4OozhjD5408ZfOJZ3H7hn2jfvh3bHTyEa848jqW6LNmGn6b1dFxti/nOyHLIiruXHXMufuuGqs0Ak1ef+svpdSxZX/rciyWTP5jC+HHZj+uLz7/k9dfeYNnlevHQ/Y9RU5MlA31m7DiW691rnvcO+r/tue3mO1u1vZav9ddYla5LdJ6jbJN116BD+/YArLXqynwwZSoA9z/5PNtt/n0WXWQR+vbqyQrLLsP4199s9TYXRVFulObS/RIRt6fVLyPihtJ9knbPo84i6Lt8b9Zc67s8+/S4Ocr32GcXbh911zzH77TLdhy07xGt1TyrAqPufZTtNlsfgMkfTWWt76w8e1+vnkvxwUefzN4+ZOg5SGL3bQew27YDWrupC5woSAdM3n3qQ4AbyigDQNJgYDBAt8WXo3PH7vm2roos3rkTw0aezUnHnc7nn31zP/mIowdTM6uGm2+4Y47j1+3/PaZ/9RWvvjyhtZtqbWTY9aPp0K4dO2yxYZPHjjztGHr1WIqPPpnGIUPPYcW+y7L+Gqu2QisXXNU+qqVcuQR1SQOB7YE+ks4r2dWFRrI0RsTsqfIWlj51gA4dOjBs5DmMunE0d97x79nlu+81iK23HcAeO/98nvf8eNeB3HKTu14WFrfe+xgPjX2B4X88irqsp8v0WIr3U1cMwAdTptKrRzcAevVYCoAe3brww43WYfxrbzmoN6Hau1XKlVef+rtk/enTmbMv/TZg25zqXGCdcd4pTHjtDYZfePnssi232pRfHHkgB+x9BNO/mj7H8ZLYadC27k9fSDzyzHguvfluzjv+cDp17Di7fMsN1uauh59ixsyZTPxgCm+/N5k1+63El9O/5osvs9+ZL6d/zX+efYlvf6t3WzV/gVEbUfZSDkntJT0r6Y60vZKkJyRNkHSdpEVTece0PSHtX3F+PkdeferPA89LuioinD+9Ed/fcF122/PHvPzia9z94I0AnP7HcznltCEs2nFRrrl5OJDdLB3ym1MA2GiT9Xn33ff539sT26zdlo9jzhjO2PGv8sm0z9n6wGM4bK8fc8mNdzJj5iwOGZqNflpr1ZX5w2H78u0VerPNpv3Z+ZdDad+uPccdshft27fj4w+n8eu/XARATU0NAwdswGbrrdmWH2uBkEPXwK/IBo10SdunA2dHxLWS/gEcBFyUXqdGxLcl7ZmO26OlleY1pPH6iPiJpBeY82clICJirabOsTB1v1j5ij6k0VqmEkMa9/7WLmXHnKvfHtVofZL6AiOBPwNHAzsBHwLLRsQsSRsDJ0XEtpLuTuv/kdQBeB9YOloYnPO6Ufqr9LpjTuc3M6uo5ox+KR3UkQxL9wTrnAMcA9Q9KNAD+KSk52Ii0Cet9wHeAUgB/9N0/JRmfgQgv+6XutmNpgBfRUStpFXJsja6I9jMqs6sZgT10kEdc5O0IzA5Ip6WtGVFGtcMeSf0eghYTFIf4B5gP+CynOs0M2u2aMZ/TdgU+HHKfXUt8EPgXKBb6l4B6AtMSuuTgOUB0v6uwEct/Rx5B3VFxJfArsCFEbE7sEbOdZqZNVulniiNiCER0TciVgT2BO6LiH2A+4Hd0mH7A7em9dvSNmn/fS3tT4dWCOrphsA+wOhU1j7nOs3Mmi0iyl5a6PfA0ZImkPWZX5LKLwF6pPKjgWPn53Pk/UTpr8meIB0VES9KWpns28rMrKrkkXo3Ih4AHkjrbwAb1HPMdKBi6VNyDeoR8SDwoKQlJC2RPlSTGRrNzFpbUdIE5Nr9Iul7kp4FXgRekvS0JPepm1nVKcrE03l3v1wMHB0R9wOk4T3DgU1yrtfMrFnyeBCzLeQd1DvXBXTI+pckdW7sDWZmbaEoCb3yDupvSPoDcEXa3hd4I+c6zcyarSj51PMe0nggsDRwM3AT0DOVmZlVFfepN0LSYsChwLeBF4DfRMTMPOoyM6uEmihGB0xe3S8jgZnAw8BA4LtkY9bNzKpSUbpf8grqq0fE9wAkXQI8mVM9ZmYVUe7kF9Uur6A+u6slpZLMqRozs8ooRkjPL6ivLWlaWhfQKW3XTZLRpeG3mpm1vmq/AVquvPKpO2mXmS1QHNTNzArEo1/MzArEo1/MzArEuV/MzArEfepmZgXiK3UzswKpKUieRgd1MzP8RKmZWaF49IuZWYH4St3MrEB8pW5mViC+UjczKxCnCTAzKxB3v5iZFUj4St3MrDicJsDMrECcJsDMrEB8pW5mViA1tcXoU2/X1g0wM6sG0Yz/GiNpeUn3S3pJ0ouSfpXKu0saI+n19LpUKpek8yRNkDRO0nrz8zkc1M3MyPrUy12aMAv4TUSsDmwEHC5pdeBY4N6I6Afcm7YBBgL90jIYuGh+PoeDupkZWZ96uUtjIuK9iHgmrX8GvAz0AQYBI9NhI4Gd0/og4PLIPA50k7RcSz+Hg7qZGc27Upc0WNLYkmVwfeeUtCKwLvAE0Csi3ku73gd6pfU+wDslb5uYylrEN0rNzGjejdKIGAYMa+wYSUsANwG/johpkkrfH5JyGW7joG5mRmWHNEpahCygXxURN6fiDyQtFxHvpe6Vyal8ErB8ydv7prIWcfeLmRmVu1Gq7JL8EuDliDirZNdtwP5pfX/g1pLyn6ZRMBsBn5Z00zSbr9TNzKho6t1Ngf2AFyQ9l8qOA04Drpd0EPA28JO071/A9sAE4EvggPmp3EHdzIzKZWmMiEcANbB7q3qOD+DwilSOg7qZGeBJMszMCqXWqXfNzIrDWRrNzArEQd3MrECKEdJBRfl2KjJJg9MTbGaz+ffC6uOHjxYM9eaVsIWefy9sHg7qZmYF4qBuZlYgDuoLBvebWn38e2Hz8I1SM7MC8ZW6mVmBOKibmRWIg3qFSQpJZ5Zs/1bSSTnUc9xc249Vug7Lh6QaSc9JGi/pBkmLN/P9vSXdmNbXkbR9yb4fSzq24Xdb0TmoV97XwK6SeuZczxxBPSI2ybk+q5yvImKdiFgTmAEc2pw3R8S7EbFb2lyHLBd33b7bIuK0irXUFjgO6pU3i2xUwlFz75C0tKSbJD2Vlk1LysdIelHSCElv130pSLpF0tNp3+BUdhrQKV3tXZXKPk+v10raoaTOyyTtJqm9pL+lesdJOiT3n4SV42Hg25K6p3/rcZIel7QWgKQt0r/zc5KelbSkpBXTVf6iwCnAHmn/HpJ+JukCSV3T71G7dJ7Okt6RtIikVSTdlX6vHpa0Wht+fqu05kzh5KWsaa4+B7oAbwFdgd8CJ6V9VwObpfUVyKa7ArgAGJLWtyNLQ9EzbXdPr52A8UCPunrmrje97gKMTOuLks1S3ons6cMTUnlHYCywUlv/vBbGpeTfqgPZlGa/AM4HhqbyHwLPpfXbgU3T+hLpPSsC41PZz4ALSs49ezud+wdpfQ9gRFq/F+iX1jcE7mvrn4mXyi1O6JWDyGYOvxw4EviqZNfWwOols4p3STOOb0YWjImIuyRNLXnPkZJ2SevLA/2Ajxqp/k7gXEkdyb4gHoqIryRtA6wlqe7P9q7pXG+29HNai3UqmebsYbL5LJ8A/g8gIu6T1ENSF+BR4Kz0F9nNETGxdFb6JlxHFszvB/YELky/b5sAN5Scp+P8fySrFg7q+TkHeAa4tKSsHbBRREwvPbCh/0klbUn2RbBxRHwp6QFgscYqjYjp6bhtyf6HvrbudMAREXF38z6G5eCriFintKCh34GIOE3SaLJ+80clbQtMr/fged0GnCqpO9AfuA/oDHwyd/1WHO5Tz0lEfAxcDxxUUnwPcETdhqR10uqjpElo0xX1Uqm8KzA1BfTVgI1KzjVT0iINVH8d2eS1mwN3pbK7gV/UvUfSqpI6t+zTWQ4eBvaB2V/mU9JffKtExAsRcTrwFDB3//dnwJL1nTAiPk/vORe4IyJqImIa8Kak3VNdkrR2Hh/I2oaDer7OBEpHwRwJrJ9uhr3EN6MeTga2kTQe2B14n+x/1ruADpJeJpuJ/PGScw0DxtXdKJ3LPcAWwL8jYkYqGwG8BDyT6rkY/6VWTU4C+ksaR/ZvvX8q/3W6KToOmEnWvVbqfrIuveck7VHPea8D9k2vdfYBDpL0PPAiMKhyH8PamtMEVIHU/10TEbMkbQxc5D+PzawlfKVWHVYArk/Dz2YAB7dxe8xsAeUrdTOzAnGfuplZgTiom5kViIO6mVmBOKjbHOY3g+Bc57qs7gnWlNNm9UaO3VJSs5OSSXqrEsnTKnUes7bmoG5zazSDoKQWjZiKiJ9HxEuNHLIl2ePrZjYfHNStMXUZBLdM2fxuA15qKONjejrxAkmvSvo3sEzdiSQ9IGn9tL6dpGckPS/pXkkrkn15HJX+SthcDWe07CHpHqWMlmTpD+Yg6VBJfyvZ/pmkC9L6PFkv53rviunhrLrt2fnwG8puKGn39JfN85Iemt8futn88Dh1q1e6Ih/IN2kG1gPWjIg3UzD8NCK+nx6celTSPcC6wHeA1YFeZE+w/nOu8y4NDAcGpHN1j4iPJf2DLHvhGem4q4GzI+IRSSuQpTn4LjAUeCQiTlGWYrg0DUOdm4D/AL9L23sAf07rB6b6OgFPSbopIhpLkFZqGHBoRLwuaUPgQrKMiicC20bEJEndyjyXWS4c1G1u9WUQ3AR4MiLqMjo2lPFxAHBNRNQA70q6r57zb0SWOfJNmJ0jpz4NZbQcAOya3jtac2a0JJV/KOkNSRsBr5PlS3k07W5u1ksA1Hh2w0eByyRdD9zc1LnM8uSgbnNrKIPgF6VF1JPxUSXTqlVAszJa1uNasiRprwCjIiJUXtbLWczZLVm3vx0NZDeMiEPTlfsOwNOS+jfj6t+sotynbi3RUMbHh8hm4WkvaTngB/W893FggKSV0nu7p/K5sw02lNHyIWDvVDaQbzJazm0UWaKqvfgm/XBjWS/rfAAsk/ruOwI7QpYjnwayGyrLpPhERJwIfEj2F4BZm3BQt5ZoKOPjKLLujpeAy8n6tecQER+SzcJ0s7IsgXXZA28Hdqm7UUrjGS0HSHqRrBvmf/U1MCKmAi8D34qIJ1NxY1kv6943k2yKuCeBMWRX+nUaym74N0kvpJ/FY8Dz9f/YzPLn3C9mZgXiK3UzswJxUDczKxAHdTOzAnFQNzMrEAd1M7MCcVA3MysQB3UzswL5f8qgPg+CvKGbAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     }
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "> **Nhận xét**:\n",
    "> * Ta thấy rằng nếu nhìn vào các chỉ số như **precision**, **recall** và **F1-score** thì model được update bởi weight ở tại epoch thứ hai này hoạt động cho hiệu suất thậm chí tốt hơn một ít dù chỉ mới trải qua 2 epoch."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Ta sẽ đào tạo một model thứ hai với các hyper-params là `num_word` và `maxlen` lần lượt là 3000 và 100, ở project hai ta đã tính được toàn bộ data của ta có khoảng loanh quanh 3000 từ và các comment hay có độ dài là 100 và sử dụng 15 neuron ở LSTM layer.\n",
    "\n",
    "Ta sẽ định nghĩa model sau đó cho training, dự đoán và đánh giá."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "source": [
    "hyperparameters_2 = {\n",
    "    'pX': X_train['normalize_comment'], \n",
    "    'py': y_train, \n",
    "    'pnum_words': 3000, \n",
    "    'pseq_length': 100,\n",
    "    'pno_units': 15, \n",
    "    'pdropout': 0.3, \n",
    "    'pbatch_size': 32, \n",
    "    'pepochs': 10, \n",
    "    'psave_weights': './models/lstm_model_comments_2.h5'\n",
    "}\n",
    "\n",
    "model_2 = DeepModel.SentimentLSTM()\n",
    "model_2.define(**hyperparameters_2)\n",
    "model_2.model.summary()\n",
    "plot_model(model_2.model, to_file='./images/lstm_model_comments_2.png', show_shapes=True, show_layer_names=True)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (None, 100, 100)          358500    \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, 15)                6960      \n",
      "_________________________________________________________________\n",
      "output (Dense)               (None, 2)                 32        \n",
      "=================================================================\n",
      "Total params: 365,492\n",
      "Trainable params: 365,492\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhUAAAGVCAIAAADG3pRsAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdfVxUVf448DPDDMMwTyCowEiJFLhRjQasYLoIGEggJoGIomXhsm6Kk7krGOr3VZKrUUklRWLtbxMUpJfugg/lYmoquoMGpvkU4BMwyIM8xjP398ep2+0ODJfLwL3A5/0Xc+6ZM5+5l5nP3HPPPUdAEAQCAAAABkjIdQAAAABGJMgfAAAA2ID8AQAAgA3IHwAAANgQcR2AMREREVyHAAAAXFq3bp23tzfXUfSO1+cfOTk59+/f5zoKMMKcP3/+/PnzXEfBI/A5GrlycnLu3bvHdRR94vX5B0Lo9ddfX7RoEddRgJEEn7YeOHCA60D4QiAQwOdohBIIBFyHYAyvzz8AAADwFuQPAAAAbED+AAAAwAbkDwAAAGxA/gAAoL179wp+JZfLaVvv3LkTGhra2NhYU1NDVps+fXpbWxu1GnWrQCDw8PAYxncwAEeOHHFxcRGJ+hw9VFRUFBwcbGVlpVAo5s6de/bs2YHWiY+Pz8rKoj0lPj6e3DleXl4meS/cgvwBwC+am5sff/zxkJAQrgPhzCeffEIQRHNzM7WwqKjIw8MjICBAqVTa2toSBKHT6XC5Vqul1sRbCwoKbGxsCIIoLCwc1ugZKCkpCQ0NTUhIqKqq6qvOhQsXZs6cqVAorl27VlZWNmXKlDlz5nzzzTcDqrNy5cqEhIRNmzZRn/WPf/yDIAiCIMzMzEz+1jgB+QOAXxAE0dPT09PTw1UAcrl81qxZXL16rxobG+fPn//iiy+uXr2aWi6RSGxsbNLS0vbt28dVbCxs2rRp5syZFy9eVCgUvVbo6el59dVXraysvvjiC3t7e1tb208++cTZ2TkmJqa9vZ15HWdn54MHDyYlJWVnZw/Te+MC5A8AfqFQKEpKSo4cOcJ1IDyyY8cOvV6/efNmWrmFhUVGRoZQKIyNjb158yYnsbGwZ8+e+Ph4Iz1Xp0+fvnr1anh4uFQqxSVmZmZRUVH37t3Ly8tjXgchpNFowsPD33jjja6uriF7QxyD/AEA6B1BEOnp6TNmzHBwcDDcGhgYmJiY2NTUFBERQbsQwlvkN35fTpw4gRCiXbnBD/Pz85nXwRYuXHj//v3Dhw8PNm6+gvwBAEIIHTp0iLy2ib8NqSW3b9+OjIy0srKysbEJCQkpKSnBz0pOTsYVJk2apNPp/P39FQqFpaWlr68veUF169atuA7ZN3Xs2DFcYmtrS22npaXl7NmzeJOR38jDpri4uKqqSqPR9FVhy5YtAQEBly9fXrNmjZF2amtr161b5+zsbG5ubm1tHRQU9O233+JNTHYyVl1dHRcXN3nyZHNz8/Hjx4eFhRUVFZnkbVJdv34dITRp0iRqoVqtRgiRp1lM6mDTpk1DCH399dcmj5MnIH8AgBBCL7zwAkEQCxYs6LVEq9Vqtdry8vKsrKwTJ05ERUXhOuvXrycIQqPR1NfXr127duvWrXq9/vTp03V1dX5+fqdOnUIIJSYmEgQhk8nIlufNm0cQhLu7O1mC25HJZM8++yy+xErt9PDz87OxsRn+Sb2uXLmCDL4oqYRCYUZGhqOjY3p6ekZGRq919Hq9p6dnZmZmSkpKTU3NhQsXLC0t/f3909PTEbOdjBCqrKz09PTMzs5OTU2tq6s7efJkXV2dt7d3QUGBad9yfX09Qoh6sBBCeEDaw4cPmdfBcFLBu3FUgvwBQP9iYmK8vb1lMtncuXODg4N1Ol1NTQ21QktLS2pqKq7j4eGxd+/ejo6OtWvXmuTVe3p6cFIxSWvMVVZWIoRUKpWROra2ttnZ2WKxODY2Fv8wp0lISCgrK9u5c2dISIhSqXRxccnMzLS3t4+Li6MNgjKykxMSEu7cufP+++8///zzcrnczc1t//79BEEYP+8xFbznjU9F1WsdpVIpEAjwbhyVIH8A0D9PT0/yb0dHR4RQRUUFtYJMJsOdFdhTTz3l4OBQXFxsku8O8uf24JsaENyPJxaLjVfz8vJKTk5uaWmJiIhobW2lbT148CBCKDg4mCyRSCT+/v6tra20jh0jO/nQoUNCoZA6tNrOzs7Nze3ixYumnVrYysoKIdTS0kItxA/xJoZ1SCKRyHCfjBqQPwDoH/U3uLm5OUKINszX8ItjwoQJCKEHDx4MfXRDxcLCAiHU2dnZb824uLjIyMgrV67Qhvm2t7c3NDRYWFjQxstOnDgRIaTX66mFfe1k3EhPT49KpaLen3jp0iWE0K1bt1i/QUNTp05FCNFyUnl5OULIxcWFeR1SV1dXvxftRy7IHwCYQG1tLa1/CWcOnEUQQkKhsKOjg1oBd6NT8W2ybnt7e4RQQ0MDk8rp6emurq6ff/75l19+SRZKJBKVStXW1tbU1EStjHuu7OzsmLQskUisrKxEIlFnZydhwNfXdwBvqT+4tYsXL1IL8UN/f3/mdbDGxkaCIPBuHJUgfwBgAm1tbfiubOyHH36oqKjQaDTkd4e9vT3+iYrp9fq7d+/SGrG0tCRzjKur62effTbEUffjySefRAY/tPsil8u/+uormUyWmppKLV+4cCFCiDqGtb29PT8/XyqVBgYGMowkLCysq6uLNkfI9u3bH3nkEdPeXeHj4/PEE0/k5OSQI5K7u7v379/v6OhIdsExqYPhI45346gE+QMAE1CpVBs3biwoKGhpaSksLIyOjjY3N09JSSErBAQEVFRUfPzxx83NzSUlJWvXriVPTUjPPPPMzZs37927V1BQUFpaOnv2bFzO1fgrjUYzYcKE4uJihvXd3NzS0tJohdu2bXNyctJqtXl5eU1NTTdv3lyyZEllZWVKSgruxWJi27Ztzs7Or7zyytGjRxsaGurq6tLS0t56663k5GRyoHN0dLRAICgrK2PYZq+EQuGePXvq6upWrFih1+tra2tfe+21W7du7d69G/fmMayD4RHGAQEBgwmJ1wzPB/kDIZSVlcV1FGCECQ8PDw8PH+iz8GVe0tKlS2ljQ998803i9z1UwcHB+LkajUatVv/444+BgYEKhUIqlfr4+Jw5c4bafn19fUxMjL29vVQqnTVrlk6nI8fvbtiwAde5fv367NmzZTKZo6Pjrl27yOfOnj3b2tr63Llz7HYIk88R7nTC819Rbdy4USQSlZeX44fV1dXUPeDu7m7Y1KpVq/D8V6SamhqtVuvk5CQWi1UqVWBgYH5+Pt7EfCfjm0imTJkiFovHjx8fEBBw/Phx6qv4+fnJ5fKuri4jbzM3N9fwO3D37t20apcuXQoKClIqlXK53M/Pj3YomdeJiIhQq9UdHR20cjMzsxkzZhiJk8Tz70DIH2C0YZc/BgPnj+F8xQEZTP6or69Xq9WxsbFDFp1pPHz4UCqVxsTEcB3Ib4qKigQCwb59+ww3jZr8Af1XAIA+qVSq3NzcnJycXbt2cR1LnwiCiIuLUyqVb7/9Ntex/KK0tDQsLCwhIWHx4sVcxzKEIH9wRi6XUwcjJicncx3RL3gbGBhqq1atMlz/Y/r06YWFhUePHm1sbOQqMOOqqqpKS0vz8/MZDugaBmlpaUlJSUlJSdRCcv2P7u5urgIzMa5PgIxBDM7dmpqaHnvsMbKTdGT5/vvvEUILFizgOhA63gbGxHD2X7377rvUTxPuvucbJp8jwE88P3Yj/vyDgDUbBmFEB88HeN4q0tatW7mOCIDhw/0cn4OE12zgOgoAABhzRvz5BwAAAE6M7PwxytZsGEHBd3V1ZWVlPffcc3Z2dlKp9KmnnkpJScG9iPX19dTL77hLp6uriywJDw/HjRhZ0YG6K27cuLFo0SIbGxv8kDbxLQCAMxxdd2EEMbt2hBcPaG1tpZUsWLDg3Llzzc3Nx48fl0qlnp6e1GdpNBqZTObt7Y3r6HS6p59+2tzc/OTJk2Qd6noMmLu7O+3eKMM6mK+v77hx4woKCoxE3utlaj4E3+/1c3wf1jvvvFNXV1ddXf3hhx8KhULqxYDAwEChUPjTTz9Rn+Xt7Z2RkYH/rqioePTRRydOnHj48OGmpqYrV674+PhYWFhQ75LDu8LHx+fbb79taWk5f/68mZlZdXV1X1Fhw3//B88x/BwBHuL5sRvZ5x/Gjeg1G7gNnok5c+YkJCRYW1vb2tquWbNmyZIlKSkp5BDPdevW9fT0vP/++2T9s2fP3r17NyIiAj9kvqLDhg0b5syZY2lpOWPGjK6uLvIUCgDArdGcP0b0mg3cBt+vkJAQcglSTKPRdHZ2Xr16FT8MCAh46qmn/vnPf9bW1uKSd999d82aNeRiEsxXdPjjH/840PBycnIE4FcIocjISK6jAGwM9D9/mI348VdGsFuzoaKi4sGDB5xPuczz4BsaGt57772DBw/ev3+fOg/5zz//TP6t1WpfffXV1NTUTZs23bx588SJE1988QXehFd0QH2sbXfr1i3qmqm0hUKZ8PLyev311wf6rNEqMjJSq9UO//JTYPAiIyO5DsGY0Zw/+oXXbKAm+RG0ZgO3wc+fP/+7775LSUmJioqytbUVCAQ7d+58/fXXqf11S5cu3bhx48cff/z3v//9vffee+mll6ytrfEmvKJDc3Nza2vrIAcd9GrSpEmLFi0yebMjVGRkpLe3N+yQkYjn+WM091/1a0Sv2cBV8CKR6OrVq2fPnrWzs4uLixs/fjxOQoaLdEokkr/+9a8PHjx47733MjIyaNdmhm1FBwDAEBnT+WNEr9kwpMEbZ2ZmNmfOHL1e/+6779bU1LS2tn777beffvqpYc2//vWvUqk0MTFx7ty5jz32GHUTkxUdAAC8xt3Qr/6h/saujeg1G2jd+u+++y5Pgu/3esO1a9eqq6tjY2MdHR3FYvHEiRNffvnl+Ph4vJW2JsTKlSsRQqdOnTLcA0ZWdKDtigH9o8L4XZp+P0eAt3h+7AQE2wGmw0AgEGRlZQ1Rv+20adNqamoYrs3JNyMo+C+++GLXrl2FhYXD9op4iPCBAweG7RV5bkg/R2BI8fzYjen+KzAMPv3003Xr1nEdBejH3r17yTGjtPnbEUJ37twJDQ1tbGysqakhq02fPp1cAByjbhUIBB4eHsP4DgbgyJEjLi4uRrpJi4qKgoODraysFArF3LlzaVfpmNSJj4/PysqiPYWcv10gEHh5eZnkvXAL8gcwvfT09IULFzY3N3/66acPHz7k7a8nQIPXH2xubqYWFhUVeXh4BAQEKJVKW1tbgiDwwI2ioiKtVkutibcWFBTgaQ6G86SToZKSktDQ0ISEhKqqqr7qXLhwYebMmQqF4tq1a2VlZVOmTJkzZ84333wzoDorV65MSEjYtGkT9Vn/+Mc/cLePmZmZyd8aNzjrOWMADU3f34hYs6EvIyL43bt3I4REItHTTz998eLFYX71Yb7+0dccMPxpn8nnqK/1axsaGiZNmkRbv1an00kkEhsbG4RQZmYm7Slk/uChqKiobdu2dXZ2qtVqMzMzwwrd3d1ubm729vY///wzLunq6nJ1dXV0dGxra2Neh/h1/dpe9zysXzuCjeg1G0ZE8HgZ6s7OzuLi4meeeYbrcAB7O3bs0Ov1mzdvppVbWFhkZGQIhcLY2NibN29yEhsLe/bsiY+PN9Jzdfr06atXr4aHh0ulUlxiZmYWFRV17969vLw85nUQQhqNJjw8/I033hjF49HHYv4AADBBEER6evqMGTMcHBwMtwYGBiYmJjY1NUVERNAuhPAW+Y3flxMnTiCEaFdu8MP8/HzmdbCFCxfev3//8OHDg42bryB/gLELDyB2dnY2Nze3trYOCgoiJ/UazBz4PJljf/CKi4urqqo0Gk1fFbZs2RIQEHD58mXDWS+pjOxnJmsWYEZm+zeh69evI4So0+cghNRqNUKIPM1iUgfDM9R9/fXXJo+TJyB/gDFKr9d7enpmZmampKTU1NRcuHDB0tLS398/PT0dIZSYmEj8/laYefPmEQRB3kODfu1LpF6fwD0VuFyj0dTX169du3br1q16vf706dN1dXV+fn6nTp0aZPvYUN+gihC6cuUKMviipBIKhRkZGY6Ojunp6RkZGb3WMb6fX3jhBeLXifq1Wq1Wqy0vL8/Kyjpx4kRUVBTZSGVlpaenZ3Z2dmpqal1dHTk5qeF9QoOE5/ih3QKFB6Q9fPiQeR0MJxW8G0clyB9gjEpISCgrK9u5c2dISIhSqXRxccnMzLS3t4+LizMyOGdA+LxAABN4Lude57gk2draZmdni8Xi2NhY/MOchvl+NrJmAfPZ/ocC3snGJ4vrtY5SqRQIBMMzJTYnIH+AMQpPXhAcHEyWSCQSf3//1tZWU3U48HmBACbwVQ1yyv2+eHl5JScnt7S0REREGE6Dxnw/G1mzgPls/4OEp7VuaWmhFuKH5IzXTOqQRCKR4T4ZNSB/gLEITyBvYWGhUCio5RMnTkQI6fV6k7xKr3Pso19nSuY/CwsLhFBnZ2e/NePi4iIjI69cubJ69Wpq+YD2c19rFuBGenp6VCoV9f7ES5cuIYRu3brF+g0amjp1KkKIlpPwPKQuLi7M65C6urr6vWg/ckH+AGORRCJRqVRtbW1NTU3UctyjYmdnhx8Ocg58PMc+tWQELRCAEMJzOeOVWvqVnp7u6ur6+eef41tJMIb72Tg8279IJOrs7DS8BcHX13cAb6k/uLWLFy9SC/FDf39/5nWwxsZGgiA4X0xo6ED+AGPUwoULEULUsZXt7e35+flSqTQwMBCXDHIO/BG9QABC6Mknn0QGP7T7IpfLv/rqK5lMlpqaSi1nsp/7NWyz/fv4+DzxxBM5OTnkiOTu7u79+/c7OjqSXXBM6mD44OLdOCpB/gBj1LZt25ycnLRabV5eXlNT082bN5csWVJZWZmSkoJ7V9Cg58Af0QsEIIQ0Gs2ECROKi4sZ1ndzc0tLS6MVMtnP/WIy2390dLRAICgrK2PYZq+EQuGePXvq6upWrFih1+tra2tfe+21W7du7d69G/fmMayD4RHGAQEBgwmJ10xwD/uQQfy+dx/wE/P5S2pqarRarZOTk1gsVqlUgYGB+fn51AqDmcCf8wUCSEw+R33NX7Jx40aRSFReXo4fVldXU789aHP1Y6tWraLNX2JkPzNfs8DIbP+Yn5+fXC7v6uoy8jZzc3MNvwN3795Nq3bp0qWgoCClUimXy/38/GhHjXmdiIgItVrd0dFBKx8185dA/gCjDU/W/8D5g+soCGJw+aO+vl6tVtPmv+Khhw8fSqVSPHEOT+D5r/bt22e4adTkD+i/AgD0SaVS5ebm5uTk7Nq1i+tY+kQQRFxcnFKpfPvtt7mO5RelpaVhYWEJCQmLFy/mOpYhBPkDAPCLVatWGa7/MX369MLCwqNHjzY2NnIVmHFVVVWlpaX5+fkMB3QNg7S0tKSkpKSkJGohuf5Hd3c3V4GZFuQPAEwMz1tVXFxcXl4uEAgSExO5jqh/0dHRZKcEbf0PhNDkyZPz8vKUSiUnsfXLzs7uzJkzbm5uXAfym+3btxueeZDrfxAEMaQDH4bNsE7HBsBYsH79+vXr13MdBQBDDs4/AAAAsAH5AwAAABuQPwAAALAB+QMAAAAbfL9+bvL1YcCoh+drys7O5joQHoHPERgKAmIo158ZJA5nHgUAAD7IyspatGgR11H0jtf5AwBuCQQCPn96AeAWXP8AAADABuQPAAAAbED+AAAAwAbkDwAAAGxA/gAAAMAG5A8AAABsQP4AAADABuQPAAAAbED+AAAAwAbkDwAAAGxA/gAAAMAG5A8AAABsQP4AAADABuQPAAAAbED+AAAAwAbkDwAAAGxA/gAAAMAG5A8AAABsQP4AAADABuQPAAAAbED+AAAAwAbkDwAAAGxA/gAAAMAG5A8AAABsQP4AAADABuQPAAAAbED+AAAAwAbkDwAAAGxA/gAAAMAG5A8AAABsQP4AAADABuQPAAAAbED+AAAAwIaI6wAA4JHdu3fX1dVRS/7973+XlZWRD1esWDFhwoRhjwsAPhIQBMF1DADwxV/+8pe0tDSJRGK4qbOz09raWq/Xi0TwqwsAhKD/CgCqqKgohFB7b8zMzJYsWQLJAwASnH8A8BuCINRqdWVlZa9bz5075+3tPcwhAcBbcP4BwG8EAsHSpUvNzc0NNzk4OHh5eQ1/SADwFuQPAH4nKiqqo6ODVmhubv7SSy8JBAJOQgKAn6D/CgC6xx9//KeffqIVXr58+amnnuIkHgD4Cc4/AKCLjo4Wi8XUksceewySBwA0kD8AoIuOju7q6iIfisXiFStWcBgPAPwE/VcA9GLatGmXL1/Gnw6BQFBSUuLk5MR1UADwC5x/ANCL5cuXm5mZIYQEAoG7uzskDwAMQf4AoBdRUVE9PT0IITMzs+XLl3MdDgB8BPkDgF7Y29s/++yzAoGgp6cnIiKC63AA4CPIHwD0btmyZQRBzJkzx87OjutYAOAlgiIrK4vrcAAAAPBUeHg4NWX0MhkcZBEwFkRGRmq1WuPzWX3wwQd//vOfZTLZsEXFlYKCgp07d8JnHxjxwQcf0Ep6yR+LFi0almAA4FJkZKS3t7fx//ZZs2Y5ODgMW0jc2rlzJ3z2gREHDhyglcD1DwD6NHaSBwAsQP4AAADABuQPAAAAbED+AAAAwAbkDwAAZ+7cuRMaGtrY2FhTUyP41fTp09va2qjVqFsFAoGHhwdXARt35MgRFxcXI4scFxUVBQcHW1lZKRSKuXPnnj17dqB14uPj+TNMDvIHAAPQ3Nz8+OOPh4SEcB3IaFBUVOTh4REQEKBUKm1tbQmC0Ol0uFyr1VJr4q0FBQU2NjYEQRQWFnIUcp9KSkpCQ0MTEhKqqqr6qnPhwoWZM2cqFIpr166VlZVNmTJlzpw533zzzYDqrFy5MiEhYdOmTUP4ZhiD/AHAABAE0dPTg6fG4oRcLp81axZXr25CjY2N8+fPf/HFF1evXk0tl0gkNjY2aWlp+/bt4yo2FjZt2jRz5syLFy8qFIpeK/T09Lz66qtWVlZffPGFvb29ra3tJ5984uzsHBMT097ezryOs7PzwYMHk5KSsrOzh+m99Q3yBwADoFAoSkpKjhw5wnUgI96OHTv0ev3mzZtp5RYWFhkZGUKhMDY29ubNm5zExsKePXvi4+ON9FydPn366tWr4eHhUqkUl5iZmUVFRd27dy8vL495HYSQRqMJDw9/4403qKvUcALyBwBguBEEkZ6ePmPGjF7vsAkMDExMTGxqaoqIiKBdCOEt8hu/LydOnEAI0a7c4If5+fnM62ALFy68f//+4cOHBxv34ED+AICpQ4cOkZdw8fcateT27duRkZFWVlY2NjYhISElJSX4WcnJybjCpEmTdDqdv7+/QqGwtLT09fUlL41u3boV1yH7po4dO4ZLbG1tqe20tLScPXsWbzLya5fniouLq6qqNBpNXxW2bNkSEBBw+fLlNWvWGGmntrZ23bp1zs7O5ubm1tbWQUFB3377Ld7E5NBg1dXVcXFxkydPNjc3Hz9+fFhYWFFRkUneJtX169cRQpMmTaIWqtVqhBB5msWkDjZt2jSE0Ndff23yOAfGcP5EAoAxACGUlZXF4okLFixACLW2ttJKFixYcO7cuebm5uPHj0ulUk9PT+qzNBqNTCbz9vbGdXQ63dNPP21ubn7y5Emyjkwme/bZZ6nPcnd3x1eMjdTBfH19x40bV1BQwOIdEcP+2f/yyy8RQu+88w6tXKfTqVQq/Hd1dbWjoyNCaO/evbiEvH6OVVZWOjk5TZw4MTc3t6Gh4caNG2FhYQKBYPfu3WSdfg9NRUXFo48+OnHixMOHDzc1NV25csXHx8fCwuLcuXPs3pparTYzMzMsf+655xBC58+fpxbeunULIfTMM88wr4M1NDQghGbPns0uSHbCw8Np8yfC+QcAphETE+Pt7S2TyebOnRscHKzT6WpqaqgVWlpaUlNTcR0PD4+9e/d2dHSsXbvWJK/e09ODP9ImaW2oVVZWIoRUKpWROra2ttnZ2WKxODY2Fv8wp0lISCgrK9u5c2dISIhSqXRxccnMzLS3t4+Li6MNgjJyaBISEu7cufP+++8///zzcrnczc1t//79BEEYP+8xFeLXBZIHWkepVAoEArwbOQT5AwDT8PT0JP/GP5wrKiqoFWQyGe52wJ566ikHB4fi4mKTfAucPHmyrq7O+HTC/IF7/8RisfFqXl5eycnJLS0tERERra2ttK0HDx5ECAUHB5MlEonE39+/tbWV1rFj5NAcOnRIKBRSB2Tb2dm5ubldvHjx/v37LN5aX6ysrBBCLS0t1EL8EG9iWIckEokM98kwg/wBgGlQf02bm5sjhGjDfA2/AiZMmIAQevDgwdBHxy8WFhYIoc7Ozn5rxsXFRUZGXrlyhTbMt729vaGhwcLCgjZeduLEiQghvV5PLezr0OBGenp6VCoV9f7ES5cuIYRwx5GpTJ06FSFEy0nl5eUIIRcXF+Z1SF1dXf1etB9qkD8AGCa1tbW0/iWcOXAWQQgJhcKOjg5qhfr6elojxvs6Rgp7e3uEEO7E71d6erqrq+vnn3+Or5pgEolEpVK1tbU1NTVRK+OeK4ZLRkokEisrK5FI1NnZadjd7+vrO4C31B/c2sWLF6mF+KG/vz/zOlhjYyNBEHg3cgjyBwDDpK2tDd9fjf3www8VFRUajYb8FrC3t8c/NjG9Xn/37l1aI5aWlmSOcXV1/eyzz4Y46iHx5JNPIoMf2n2Ry+VfffWVTCZLTU2lli9cuBAhRB3D2t7enp+fL5VKAwMDGUYSFhbW1dVFmyNk+/btjzzyiGnvrvDx8XniiSdycnLIEcnd3d379+93dHQku+CY1MHw/wnejRyC/AHAMFGpVBs3biwoKGhpaSksLIyOjjY3N09JSSErBAQEVFRUfPzxx83NzSUlJWvXriVPTUjPPPPMzZs37927V1BQUFpaOnv2bFzu5+dnY2Nz/vz54Xs/g6DRaCZMmFBcXMywvjPp2fwAACAASURBVJubW1paGq1w27ZtTk5OWq02Ly+vqanp5s2bS5YsqaysTElJwb1YTGzbts3Z2fmVV145evRoQ0NDXV1dWlraW2+9lZycTA6Pjo6OFggEZWVlDNvslVAo3LNnT11d3YoVK/R6fW1t7WuvvXbr1q3du3fj3jyGdTA8wjggIGAwIZkA9XwNxu+CsQMNfPwuvmBLWrp0aUFBAbXkzTffJH7fQxUcHIyfq9Fo1Gr1jz/+GBgYqFAopFKpj4/PmTNnqO3X19fHxMTY29tLpdJZs2bpdDp3d3fczoYNG3Cd69evz549WyaTOTo67tq1i3zu7Nmzra2tWY86Hf7P/saNG0UiUXl5OX5YXV1N3W/u7u6GT1m1ahVtNHNNTY1Wq3VychKLxSqVKjAwMD8/H29ifmjwTSRTpkwRi8Xjx48PCAg4fvw49VX8/PzkcnlXV5eRt5Obm2v47UodSYxdunQpKChIqVTK5XI/Pz/aPwDzOhEREWq1uqOjw0hIJmc4fhfyBxijWOSPwcD5Y9hebqCG/7NfX1+vVqtjY2OH80VZePjwoVQqjYmJ4TqQ3xQVFQkEgn379g3z6/L3/g/qPbombHb//v24WdrZ30DrDDO5XC4wajDzj8KuBnygUqlyc3NzcnJ27drFdSx9IggiLi5OqVS+/fbbXMfyi9LS0rCwsISEhMWLF3MdC2+uf6xfv54gCCPzGbCzePFigiBoQxeY1OF2mu7m5ubvv/8eIbRgwQLDXwHG77rqF+xqwBPTp08vLCw8evRoY2Mj17H0rqqqqrS0ND8/n+GArmGQlpaWlJSUlJTEdSAI8Sd/8A3B9TTdY8eo39X4hK+4uLi8vFwgECQmJnIdEY9Mnjw5Ly9PqVRyHUjv7Ozszpw54+bmxnUgv9m+fTsfzjywkTr/2lDD03RzHUXvDO8JGNH4vKtNYv369evXr+c6CgBMD84/RpJZs2b985//5DoKAABAiHX+MDLjMXXa5Dt37kRGRioUChsbm2XLlj18+PD27dvz589XKBT29vYrV66k3TuKXb9+PTg4WKVS0ea47velyae/8MILKpVKJpPNnj37zJkzvb6EkTrspummtWxpafnHP/4xLy9v7ty5+IkxMTGsdnafYFcP264GAPSCemGW4Rg+JjMe42mTw8LCCgsLm5ub//WvfyGEgoKCFixY8P333zc1NX366acIoddff53askajUalUvr6+Z86caWpqMpzjut+XvnXrlpWVlVqt/uabb5qami5fvhwQEDB58mSJREK+CpM6BKtpumktX7lyZe7cuePHj6e1zGSqbXz93NAXX3xBqwm72siuNgIN7/hdnoOx+6Bfprn/46WXXkIIZWRkkCWVlZUSiYR6yw/+/B8+fJgswdegTp06RZY4OTm5urpSW8aDgqhfrJcvX0YIaTQahi8dERGBEMrJySErlJeXSyQS6tcKkzpE319qubm5ZEl4eDhCqLq6uq+WHzx4YGlpSWvZx8en31u9eh1/9eyzz/aVP2BX97qrjYD8QQX5A/TLMH+wuX5ufMZj6l0F1IUYHRwcrl69Si1Rq9WGExhYWFjMmDGDfEid49re3r7flz527BhCiDr7jYODg4uLC3X1LiZ1jOh1Lmi8SJxhy+PHj586derVq1epLZw8eZLJCw0I7Oped7VxtFuUxzK8K7Kzs7kOBPAX7esdsRh/hWc8Rn2s/XLr1i3qC1CH5QmFQjMzM0tLS7LEzMzMcNSmjY0NbYbRCRMmVFRUPHjwYNy4ccZfevz48U1NTRYWFnK5nNYC+YXV3t7ebx3jjMwF3WvL1tbWTJplotcLDBjsajTwXb1z586dO3cO6CmjW2RkJNchAF7D3QCkAecPPONxc3Nza2vrUCy/bDilMznHNZOXVigUTU1Nzc3N1G+Wuro6avz91mGnr5Z5u7oD7OqsrKxFixYNMpjRITs7OzIykhghyxcCTuBOYyo246+GdMbj5uZmak8LbY7rfl86KCgI/dq5gdXU1Ny4cYNan0kddgxb1uv1DH9rM+fh4bF///7BtwO7GgAwGGzyB5MZj1mTyWSrV6++cOFCr3Nc9/vS77zzzrhx47Ra7fHjx5ubm3/88cfo6GhaLweTOuzQWr5y5cqKFSsMZz7gyVTbY2FXAwCGEPViOvMxGEZmPDacNpm6Zg5CaNu2bd999x21ZMuWLe+++y7+W61W/+9///P19ZXL5b3Ocd3vZMs3btx44YUXlEolHvGZl5dHTrj06quvMqkzmGm6yZYtLS1nzpx56tSpOXPmWFpaUiPsd6ptmUxm/KjhqTdhV/e7q41AMP6KAsZfgX4Zjr8SEJQPJ/SBDoWpU6e2trbeuXOH60BGvwHtaoFAANc/SPDZB/3C1z8OHDhAlsD8Jaak1+vHjRvX2dlJlty+fbukpMTPz4/DqEYl2NUAcA7yh4k9fPgwNjb23r17P//88//+97/IyEilUrlp0yau4xqFYFdz6M6dO6GhoY2NjTU1NeR8M9OnTycX7saoWwUCAfWeJF45cuSIi4uLkcu3RUVFwcHBVlZWCoVi7ty5tHElzOvwJ574+HjcaTkYkD9Myc7O7r///W99ff2f/vQna2vr0NDQxx9//H//+9+UKVO4Dm20gV3NoaKiIg8Pj4CAAKVSaWtrSxAEvvBWVFSk1WqpNfHWgoICvO7sYNY9GyIlJSWhoaEJCQlVVVV91blw4cLMmTMVCsW1a9fKysqmTJkyZ86cb775ZqB1eBXPypUrExISBvt7i3oxBK6hgbEDDeP1c5lM9uyzz/K5feaf/YaGhkmTJtHWndXpdBKJxMbGBiGUmZlJewqZP3goKipq27ZtnZ2darXazMzMsEJ3d7ebm5u9vf3PP/+MS7q6ulxdXR0dHdva2pjX4WE8eB1c5p8C/q5fCwAYEXbs2KHX6zdv3kwrt7CwyMjIEAqFsbGxI+hGnD179sTHxxvpKTp9+vTVq1fDw8OlUikuMTMzi4qKunfvXl5eHvM6PIxHo9GEh4e/8cYbrO/bg/wBAGCKIIj09PQZM2Y4ODgYbg0MDExMTGxqaoqIiKBdCOEt8hu2LydOnEC/n1+OfJifn8+8Dj/jWbhw4f379w8fPjygIEmQPwAwBt8E4+zsbG5ubm1tHRQU9O233+JNW7duxZeFZ82ahUuOHTuGS/Akj+jXxWtbWlrOnj2LN+GflrhcIBBMmjRJp9P5+/srFAraKiyDaX+IFBcXV1VV4cmbe7Vly5aAgIDLly+vWbPGSDtG9irzBWD6XZ/GJK5fv44Qos0bqFarEULkaRaTOvyMZ9q0aQihr7/+ml0wkD8A6JNer/f09MzMzExJSampqblw4YKlpaW/v396ejpCKDExkfj9zZ7z5s0jCMLd3Z0sWb9+PfH76xO4rwCXazSa+vr6tWvXbt26Va/Xnz59uq6uzs/P79SpU4NsHzP5TAdXrlxBBl9MVEKhMCMjw9HRMT09PSMjo9c6xvfqCy+8QPw6gb9Wq9VqteXl5VlZWSdOnIiKiiIbqays9PT0zM7OTk1NraurO3nyZF1dnbe3t8nnVMbLRdNu6cVzKDx8+JB5HX7Gg5MKPqwsQP4AoE8JCQllZWU7d+4MCQlRKpUuLi6ZmZn29vZxcXFGhscMSEtLS2pqqre3t0wm8/Dw2Lt3b0dHx9q1a03SeE9PD04qJmkNIVRZWYn6mJWZZGtrm52dLRaLY2Nj8Q9hGuZ7NSYmBu+ZuXPnBgcH63S6mpoaspE7d+68//77zz//vFwud3Nz279/P0EQxs97TAXvUtr01SzqcB6PUqkUCAT4sLIA+QOAPuHpVYKDg8kSiUTi7+/f2trK+pSfRiaT4T4EjLoKy+AbJ3+VD74pDF/VEIvFxqt5eXklJye3tLRERES0trbStjLfq70uAIMfGl+fhsVb64uVlRVCqKWlhVqIH+JNDOvwNh6RSGR4jBiC/AFA7/BSNxYWFgqFglo+ceJEhJBerzfJqxh+nidMmID4Ou2/hYUFQoh6239f4uLiIiMjr1y5snr1amr5gPaqkQVgGhoaenp6VCoV9f7ES5cuIYRu3brF+g0amjp1KkKIlpPKy8sRQi4uLszr8Daerq6ufi/a9wXyBwC9k0gkKpWqra2tqamJWo77WMi5foVCYUdHB7UC7n2mMtKxUFtbS+tfIldhMUn7poXn9jdcOaZX6enprq6un3/++ZdffkkWMtyrxuH1aUQiUWdnp+FtCr6+vgN4S/3BrV28eJFaiB+SM4EyqcPPeBobGwmCwIeVBcgfAPRp4cKFCCHq6Mb29vb8/HypVEounWtvb49/2WF6vf7u3bu0diwtLckc4Orq+tlnn5Gb2traqLMm01ZhGXz7pvXkk08igx+2fZHL5V999ZVMJktNTaWWM9mr/RrSVYiofHx8nnjiiZycHHJEcnd39/79+x0dHckuOCZ1+BkP/tfCh5UNat6G+8/B2IEY3H9eWVnp5OQ0ceLE3NzcxsbGGzduhIWFCQSCzz77jKyD+2c++uijpqamn376adGiRWq1mna79bx581Qq1d27d8+dOycSiX788UdcrtFoVCqVv7//uXPnmpubdTrd008/bW5ufvLkSZO07+vrO27cuIKCgn73BsPPfk9Pz4QJEwzvddfpdCqVqten7N27FyFEDZjJXsXjr1pbW8mSDRs2IIS+//57/LCqqsrZ2XnKlClHjhypr6+vra399NNPLS0tqcd06dKlCKHS0tJ+3xdBEH3d700QREFBgYWFxeLFiysrK2tqamJjY0Ui0bFjxwZah2/xEASRmZmJEDp48CCTkAzvP4f8AcYoJvmDIIiamhqtVuvk5CQWi1UqVWBgYH5+PrVCfX19TEyMvb29VCqdNWuWTqcjx9du2LAB17l+/frs2bNlMpmjo+OuXbvI52o0GrVa/eOPPwYGBioUil5XYRlM+/2uNENi/tnfuHGjSCQqLy/HD6urq6m/R93d3Q2fsmrVKlrCM7JXmS8A0+/6NH5+fnK5vKury8jbyc3NNfxVvXv3blq1S5cuBQUFKZVKuVzu5+dHO0YM6/AtHoIgIiIi1Gp1R0eHkZBIkD8A+AXD/DGkcP7gNgaM+We/vr5erVbT5r/ioYcPH0ql0piYGK4D+QXf4iF+nf8KL0bHBMx/BQAYFJVKlZubm5OTs2vXLq5j6RNBEHFxcUql8u233+Y6FoT4Fw9CqLS0NCwsLCEhYfHixawbgfwBABiY6dOnFxYWHj16tLGxketYeldVVVVaWpqfn89wQNdQ41s8CKG0tLSkpKSkpKTBNDKEU+UAAPqSnJz8t7/9Df8tEAjefPPNrVu3chvSgEyePHmgM8sOJzs7uzNnznAdxW/4Fg9CaPv27YNvBPIHABxYv379+vXruY4CgEGB/isAAABsQP4AAADABuQPAAAAbED+AAAAwEYv188jIiKGPw4Aht8HH3xw4MABrqPgBTylFXz2gRHnz5/38vKilggIytwABQUF77///rBHBQBP5efnP/nkk3hqcQCAt7f3unXryIe/yx8AACqBQJCVlbVo0SKuAwGAj+D6BwAAADYgfwAAAGAD8gcAAAA2IH8AAABgA/IHAAAANiB/AAAAYAPyBwAAADYgfwAAAGAD8gcAAAA2IH8AAABgA/IHAAAANiB/AAAAYAPyBwAAADYgfwAAAGAD8gcAAAA2IH8AAABgA/IHAAAANiB/AAAAYAPyBwAAADYgfwAAAGAD8gcAAAA2IH8AAABgA/IHAAAANiB/AAAAYAPyBwAAADYgfwAAAGAD8gcAAAA2IH8AAABgA/IHAAAANiB/AAAAYAPyBwAAADYgfwAAAGAD8gcAAAA2BARBcB0DAHyxfPny77//nnx47949GxsbS0tL/FAsFufl5Tk4OHAUHQD8IuI6AAB4xNXV9csvv6SWNDQ0kH8/8cQTkDwAIEH/FQC/iY6OFggEvW4Si8Uvv/zy8IYDAK9B/xUAv+Ph4XHp0iXDz4VAICgtLZ08eTIXQQHAR3D+AcDvLF++3MzMjFYoFAq9vLwgeQBABfkDgN9ZvHhxT08PrVAoFC5fvpyTeADgLcgfAPzOhAkTfHx8aKcgBEGEhYVxFRIA/AT5AwC6ZcuWUa9/mJmZzZ07d8KECRyGBAAPQf4AgO7FF18UiX4b2k4QRHR0NIfxAMBPkD8AoFMqlUFBQWQKEYlEoaGh3IYEAA9B/gCgF9HR0d3d3QghkUi0YMECpVLJdUQA8A7kDwB6ERISgqct6e7uXrp0KdfhAMBHkD8A6IWFhcWLL76IEJLJZPPmzeM6HAD4aGTPf1VQUHDv3j2uowCj06RJkxBCnp6e//73v7mOBYxaixYt4joE9kb2/CURERE5OTlcRwEAACyN6G/gkX3+gRAKDw8/cOAA11EAFBERgRAaZcciKSkpPj7ecDoTJgQCQVZW1oj+dQmGVHZ2dmRkJNdRDApc/wCgTxs2bGCXPAAYCyB/ANAn6l2EAAAayB8AAADYgPwBAACADcgfAAAA2ID8AQD37ty5Exoa2tjYWFNTI/jV9OnT29raqNWoWwUCgYeHB1cBG3fkyBEXFxcjV4+KioqCg4OtrKwUCsXcuXPPnj3Lrg5/4omPj8/KymId4QgF+QNwrLm5+fHHHw8JCeE6EM4UFRV5eHgEBAQolUpbW1uCIHQ6HS7XarXUmnhrQUGBjY0NQRCFhYUchdynkpKS0NDQhISEqqqqvupcuHBh5syZCoXi2rVrZWVlU6ZMmTNnzjfffDPQOryKZ+XKlQkJCZs2bRpohCMbMZKFh4eHh4dzHQUgiEEci8bGxilTpgQFBZk8JIZkMtmzzz5r8mYRQllZWf1Wa2homDRpUmxsLLVQp9NJJBIbGxuEUGZmJu0pZP7goaioqG3btnV2dqrVajMzM8MK3d3dbm5u9vb2P//8My7p6upydXV1dHRsa2tjXoeH8RQVFeGbfhjGhs9XBvR2+AbOPwDHFApFSUnJkSNHuA6EGzt27NDr9Zs3b6aVW1hYZGRkCIXC2NjYmzdvchIbC3v27ImPjzfSU3T69OmrV6+Gh4dLpVJcYmZmFhUVde/evby8POZ1eBiPRqMJDw9/4403urq6BhTkyAX5AwDOEASRnp4+Y8YMBwcHw62BgYGJiYlNTU0RERG0CyG8RX7D9uXEiRMIIdqVG/wwPz+feR1+xrNw4cL79+8fPnx4QEGOXJA/AJcOHTpEXg3GX5HUktu3b0dGRlpZWdnY2ISEhJSUlOBnJScn4wqTJk3S6XT+/v4KhcLS0tLX15e8qrl161ZcZ9asWbjk2LFjuMTW1pbaTktLy9mzZ/GmYb5hsLi4uKqqSqPR9FVhy5YtAQEBly9fXrNmjZF2amtr161b5+zsbG5ubm1tHRQU9O233+JNTPYnVl1dHRcXN3nyZHNz8/Hjx4eFhRUVFZnkbVJdv34d/To3JUmtViOEyNMsJnX4Gc+0adMQQl9//bVpg+QtyB+ASy+88AJBEAsWLOi1RKvVarXa8vLyrKysEydOREVF4Trr168nCEKj0dTX169du3br1q16vf706dN1dXV+fn6nTp1CCCUmJhIEIZPJyJbnzZtHEIS7uztZgtuhXv+g9jz4+fnZ2NicP39+6N7+lStXkMEXE5VQKMzIyHB0dExPT8/IyOi1jl6v9/T0zMzMTElJqampuXDhgqWlpb+/f3p6OmK2PxFClZWVnp6e2dnZqampdXV1J0+erKur8/b2LigoMO1brq+vRwhRjwtCSC6XI4QePnzIvA4/48FJBR/WsQDyB+CvmJgYb29vmUw2d+7c4OBgnU5XU1NDrdDS0pKamorreHh47N27t6OjY+3atSZ59Z6eHpxUTNJaryorKxFCKpXKSB1bW9vs7GyxWBwbG4t/CNMkJCSUlZXt3LkzJCREqVS6uLhkZmba29vHxcXRBh0Z2Z8JCQl37tx5//33n3/+eblc7ubmtn//foIgjJ/3mAreyQKBYJB1OI9HqVQKBAJ8WMcCyB+Avzw9Pcm/HR0dEUIVFRXUCjKZDPcYYE899ZSDg0NxcbFJPsDkb/DBN9UX3GUnFouNV/Py8kpOTm5paYmIiGhtbaVtPXjwIEIoODiYLJFIJP7+/q2trbSOFCP789ChQ0KhkDqK2s7Ozs3N7eLFi/fv32fx1vpiZWWFEGppaaEW4od4E8M6vI1HJBIZHqPRCvIH4C/qD3Nzc3OEUE9PD7WC4ad3woQJCKEHDx4MfXQmYGFhgRDq7Ozst2ZcXFxkZOSVK1dWr15NLW9vb29oaLCwsFAoFNTyiRMnIoT0ej21sK/9iRvp6elRqVTU+xMvXbqEELp16xbrN2ho6tSpCCFaTiovL0cIubi4MK/D23i6urr6vWg/akD+ACNYbW0trX8JZw6cRRBCQqGwo6ODWgH3ZVMNT5dIr+zt7RFCDQ0NTCqnp6e7urp+/vnnX375JVkokUhUKlVbW1tTUxO1Mu65srOzY9KyRCKxsrISiUSdnZ2GY/x9fX0H8Jb6g1u7ePEitRA/9Pf3Z16Hn/E0NjYSBIEP61gA+QOMYG1tbfhWbeyHH36oqKjQaDTkB9je3h7/TsT0ev3du3dpjVhaWpI5xtXV9bPPPhviqH/z5JNPIoMftn2Ry+VfffWVTCZLTU2lli9cuBAhRB0z2t7enp+fL5VKAwMDGUYSFhbW1dVFm5Nj+/btjzzyiGnvZvDx8XniiSdycnLIEcnd3d379+93dHQku+CY1OFnPPifDR/WsQDyBxjBVCrVxo0bCwoKWlpaCgsLo6Ojzc3NU1JSyAoBAQEVFRUff/xxc3NzSUnJ2rVryVMT0jPPPHPz5s179+4VFBSUlpbOnj0blw/D+CuNRjNhwoTi4mKG9d3c3NLS0miF27Ztc3Jy0mq1eXl5TU1NN2/eXLJkSWVlZUpKCu7FYmLbtm3Ozs6vvPLK0aNHGxoa6urq0tLS3nrrreTkZHJMc3R0tEAgKCsrY9hmr4RC4Z49e+rq6lasWKHX62tra1977bVbt27t3r0b9+YxrMPDeBBCeMRzQEDAYEIaSYbw3vahB/OX8Ae7Y4Gv/ZKWLl1KGzD65ptvEr/voQoODsbP1Wg0arX6xx9/DAwMVCgUUqnUx8fnzJkz1Pbr6+tjYmLs7e2lUumsWbN0Oh05fnfDhg24zvXr12fPni2TyRwdHXft2kU+d/bs2dbW1ufOnWO3QxCz+Us2btwoEonKy8vxw+rqauqbdXd3N3zKqlWraPOX1NTUaLVaJycnsVisUqkCAwPz8/PxJub7E99EMmXKFLFYPH78+ICAgOPHj1Nfxc/PTy6Xd3V1GXk7ubm5hl8yu3fvplW7dOlSUFCQUqmUy+V+fn60o8awDt/iIQgiIiJCrVZ3dHQYCYk0CuYvGdnRQ/7gj+E/Fjh/DOcrDgjD/FFfX69Wq2nzX/HQw4cPpVJpTEwM14H8gm/xEL/Of7Vv3z6G9UdB/hj9/VfUe5W5jqVPcrmcOu4lOTm512rd3d2ffvrpzJkzVSqVWCx2cHB4/vnnP/7449u3b+MK06ZNE/QnPj6e+tDIDWJ/+9vfyGpbt24dijcOVCpVbm5uTk7Orl27uI6lTwRBxMXFKZXKt99+m+tYEOJfPAih0tLSsLCwhISExYsXcx3L8Bn9+YO8V5nrQIxpbm7+/vvvEUILFiwgCGL9+vW9Vlu2bNlrr732wgsvXL16tamp6bvvvps+fXpcXBx1Zp4DBw6Qvw5iY2MRQkePHiVLIiMj5XI5QRD45RBCfX0Ca2trP/30U4TQ0qVLCYJITEw07VsGpOnTpxcWFh49erSxsZHrWHpXVVVVWlqan5/PcEDXUONbPAihtLS0pKSkpKQkrgMZVqM/f7Ajl8vJeZN4QqfT7du379VXX/373/8+adIkCwsLZ2fnpKSkVatWsWtQKpU++uijR48e7XUZiQ8++ADfYsZD+JyyuLi4vLxcIBCMgtw2efLkvLw8pVLJdSC9s7OzO3PmjJubG9eB/IJv8SCEtm/fPqbOPDDIHyPG1atXEUKurq608kWLFpF/FxUVhYeHG2lk//795LetUCiMj49HCBn2TdXX13/yyScbNmwYfNhDAZ9TkqBvDQBOQP4YMfBYzOPHj9PKfXx8aLNCMbdixQq1Wv2f//zn8uXL1PIPP/zw+eefd3Z2ZtcsAGAsGKP5o729ffPmzVOnTrW0tBw3btz8+fP/85//dHd3o77n9KbOg33nzp3IyEiFQmFjY7Ns2bKHDx/evn17/vz5CoXC3t5+5cqVtJuBTWL27Nl2dnZff/11UFDQyZMnaTN5sCORSP72t78RBEHtt21ubv7oo482btw4+PYBAKPYGM0fq1ev/vDDDz/66KPa2tpr165NnTp1wYIF3333Hep7Tm/qPNjr1q37+9//rtfrd+7cuXfv3qVLl2q12rfffruysvL//u//0tPTt2zZQn05k9yJJpfLDxw44OjoeOzYMV9fX3t7++jo6H379v3888+DafbPf/7zxIkTc3Jyrl27hkt27drl5+f3hz/8YTDNAgBGvTGaP/Lz893c3J577jmpVDpx4sR33313QPOyvfrqq+7u7jKZbNmyZW5ubkePHl23bt20adPkcnlsbKyTkxNtNVZTzQQ+a9asW7du/b//9/8WLFjQ2tqakZGxZMmSRx55ZP/+/azblEql69at6+npeeeddxBCP//88wcffPDmm28OMlQAwKg3rKut8ce8efM++eSTP//5z6+88oqnp6eZmdmNGzeYP506XtbBweHq1avUErVaTZuR4uTJk4MO+RcSiWT58uXLly/v6uo6ffr07t279+/fHx0d7erqOn36dHZt/vWvf92xY8e+ffu2bNmSm5vr5eX19NNPs2jn/PnzERER7GIYlT744IMDBw5wHQXgKdNOjM+JMXr+sWvXrn/961+lpaX+IIxolAAAIABJREFU/v5KpXLevHm0iTSMo46zFAqFZmZmlpaWZImZmZlJLk4YJxKJ/Pz89u3bt2HDhu7u7pycHNZNyeVyrVbb3d29ZcuW5OTkUTAcFgAwDMbo+YdAIFi2bNmyZcs6OztPnjyZnJwcFhb23nvvrVu3jqzAbYSGzp49GxYWRltRDiHk6+u7ffv2Qa7ruWbNmuTk5MzMzKCgIOq51IB4eXnBz22SQCB4/fXXqaOrAaDKzs6OjIzkOopBGaPnH1ZWVngpULFY/Nxzz+GxVdQZsDmc09uQSCS6fv06QRAPHjwwvAiP7/5j3XmFqVSqdevWqVQqOPkAADA0RvMHQugvf/nL5cuX29vbHzx4sGPHDoIg/Pz8yK19zenNjglnAl+0aFFmZmZFRUV7e/vt27eTk5Pfeustd3f35cuXD7LlzZs319fXz5w5c/BBAgDGhCGal3F4MJnz9d1336W+Xzx/dVFRUWxs7B/+8Ad8/4eXl9fu3bvJUVJEb3N6G86DTV25CCG0bds2PAKYtGXLFtxavzOBy2Qy44fp2rVr3d3dZ86cWb9+/YwZMxwcHEQikUKh8PDweOedd1paWmgNfvHFF7QWmpqaen25wMDAXkOiPf2jjz4yvp9hLmQaxGz+XTBmjYL5dwXEoAeVcgiP9oE+dz6AY0EjEAiysrLg+gfoC77+MaK/gcdu/xUAY8GdO3dCQ0MbGxtramrICRSmT59OLsWKUbcKBALWYyiG2pEjR1xcXMglEWlmzZpluGCBVqslK8THx+Nf/cAkIH8AMGoVFRV5eHgEBAQolUpbW1uCIHCna1FREfVbFSGEtxYUFOCVDXudkplbJSUloaGhCQkJhkMQmVu5cmVCQsKmTZtMGNhYBvkDjEhDPcE+DyfwH6jGxsb58+e/+OKLq1evppZLJBIbG5u0tLR9+/ZxFRsLmzZtmjlz5sWLFxUKhZFqOp2O1ke/c+dOcquzs/PBgweTkpKys7OHPuTRD/IHAKPTjh079Hr95s2baeUWFhYZGRlCoTA2NvbmzZucxMbCnj174uPj++q5Yk6j0YSHh7/xxht4XjswGJA/ABiFCIJIT0/HQ/UMtwYGBiYmJjY1NUVERNAuhPCWVCo1VVMLFy68f/8+9X4vwA7kDzDcamtr161b5+zsbG5ubm1tHRQU9O233+JNW7duxdc8yb6jY8eO4RJbW1tc0tcE+9SF7nU6nb+/v0KhsLS09PX1PXv27ODbH1mKi4urqqqMLNu8ZcuWgICAy5cvr1mzxkg7Rg4WdUWD27dvR0ZGWllZ2djYhISElJSUUBuprq6Oi4ubPHmyubn5+PHjw8LCioqKTPI2DX355ZfTpk2TyWQqlWr27NmZmZmGdaZNm4YQ+vrrr4cohjFkmMcLmxbcc8AfDI9FZWWlk5PTxIkTc3NzGxoabty4ERYWJhAIdu/eTdahTp6Pubu74+u6RupgGo1GJpN5e3ufO3euublZp9M9/fTT5ubmJ0+eNEn7vr6+48aNKygo6PedIk7v//jyyy8RQu+88w6tXKfTqVQq/Hd1dTVeonjv3r24hLx+jjE5WHhFgwULFuAdfvz4calU6unpSVaoqKh49NFHJ06cePjw4aampitXrvj4+FhYWBi5Hco4tVptZmbW66Znn3122bJlFy9ebG5uvn79+rJlyxBCa9asoVVraGhACM2ePZtdAKYyCu7/GNnRQ/7gD4bH4uWXX0YI7du3jyxpa2tzcHCQSqV6vR6XDDJ/IIS+//57sgQvrajRaIw8l3n7Pj4+xm8FJXGbP3bs2IEQwre+UlHzB0EQBQUFYrFYJpNdu3aNMMgfTA4Wzh+5ublkHbyCcnV1NX740ksvIYQyMjLICpWVlRKJxN3dnd1bM5I/DP3xj39ECJ0/f55WLhAIHnvsMXYBmMooyB/QfwWGFZ7nODg4mCyRSCT+/v6tra2m6k+QyWS4gwJ76qmnHBwciouLKysrB9/4yZMn6+rqvL29B9/UkMJXNcRisfFqXl5eycnJLS0tERERra2ttK3MD5anpyf5Nz6nqaiowA8PHTokFApDQkLICnZ2dm5ubhcvXhyGCcxxMsvNzaWVi0Qiw/cLBgryBxg+7e3tDQ0NFhYWtCGYeGl3vV5vklexsrKilUyYMAEh9ODBA5O0PyJYWFgghDo7O/utGRcXFxkZeeXKFdow3wEdLJVKRf5tbm6OEMJLGOBGenp6VCoV9Z6+S5cuIYRu3brF+g0yZG9vj3o79F1dXSa8ID9mjbwLg2DkkkgkKpWqoaGhqamJ+q2E7wizs7PDD4VCITn5MVZfX09rysgE+7W1tQRBUCvgrw+cRQbf/oiAvzdxR3+/0tPTi4qKPv/8c5x1MIYHyziJRGJlZdXc3Nza2srJMAR8GkQeeqyxsZEgCLyLwGDA+QcYVgsXLkQIUYdOtre35+fnS6XSwMBAXGJvb19eXk5W0Ov1d+/epbVjZIL9trY26tSWP/zwQ0VFhUajIb8vBtn+iPDkk08ixivcyeXyr776SiaTpaamUsuZHKx+hYWFdXV1kUPgsO3btz/yyCOmvQMjPT3d3d2dWkIQBL5PcP78+dRyfPTxLgKDAfkDDKtt27Y5OTlptdq8vLympqabN28uWbKksrIyJSUFd4wghAICAioqKj7++OPm5uaSkpK1a9fSfj8ioxPsq1SqjRs3FhQUtLS0FBYWRkdHm5ubp6SkkBUG074Jp+IfUhqNZsKECbR1lI1wc3NLS0ujFTI5WP3atm2bs7PzK6+8cvTo0YaGhrq6urS0tLfeeis5OZk8I4mOjhYIBGVlZQzb7MulS5dee+21n376qa2t7caNG3gs1po1a2bMmEGthkcPBwQEDPLlwMi++g/jr/iD+bGoqanRarVOTk5isVilUgUGBubn51Mr1NfXx8TE2NvbS6XSWbNm6XQ68nflhg0bcB3DCfYxjUajVqt//PHHwMBAhUIhlUp9fHzOnDljqvb7nYqfhLiev33jxo0ikai8vBw/rK6upn7wex3+tGrVKto4NCMHy3BFA+L3U8kGBwfjmvgmkilTpojF4vHjxwcEBBw/fpz6Kn5+fnK5vKury8jbMbwGjhCijiRua2s7cODAwoULnZ2dcefbnDlzMjMzDZuKiIhQq9UdHR397sMhNQrGX43s6CF/8AdPjgXOH1xHQRA8yB/19fVqtTo2NpbDGJh4+PChVCqNiYkZnpcrKioSCATUQclcGQX5A/qvABidVCpVbm5uTk7Orl27uI6lTwRBxMXFKZXKt99+exherrS0NCwsLCEhYfHixcPwcqMe5A8ARq3p06cXFhYePXq0sbGR61h6V1VVVVpamp+fz3BA1yClpaUlJSUlJSUNw2uNBZA/wCiB560qLi4uLy8XCASJiYlcR8QLkydPzsvLUyqVXAfSOzs7uzNnzri5uQ3Py23fvh3OPEwI7v8Ao8T69evXr1/PdRQAjCFw/gEAAIANyB8AAADYgPwBAACADcgfAAAA2ID8AQAAgA0B8fspB0aWiIiInJwcrqMAAACWRvQ38MjOHwUFBffu3eM6CjBqRUZGarVa/q8WBUauRYsWcR0CeyM7fwAwpAQCQVZW1oj+hAMwdOD6BwAAADYgfwAAAGAD8gcAAAA2IH8AAABgA/IHAAAANiB/AAAAYAPyBwAAADYgfwAAAGAD8gcAAAA2IH8AAABgA/IHAAAANiB/AAAAYAPyBwAAADYgfwAAAGAD8gcAAAA2IH8AAABgA/IHAAAANiB/AAAAYAPyBwAAADYgfwAAAGAD8gcAAAA2IH8AAABgA/IHAAAANiB/AAAAYAPyBwAAADYgfwAAAGAD8gcAAAA2IH8AAABgA/IHAAAANiB/AAAAYAPyBwAAADYgfwAAAGBDxHUAAPDInTt3uru7qSVVVVWlpaXkQwcHBwsLi2GPCwA+EhAEwXUMAPBFcHDwkSNH+toqFourqqqsra2HMyQAeAv6rwD4zeLFi/vaJBQKAwICIHkAQIL8AcBvwsLC+uqeIghi2bJlwxwPAHwG+QOA38hkspCQELFYbLhJIpGEhIQMf0gA8BbkDwB+Z+nSpV1dXbRCsVgcFhYmk8k4CQkAfoL8AcDvPP/883K5nFbY2dm5dOlSTuIBgLcgfwDwO+bm5hEREebm5tRCpVI5d+5crkICgJ8gfwBAt2TJko6ODvKhWCyOioqiZRQAANz/AQBdT0+PnZ1ddXU1WXLq1Kk//elPHIYEAA/B+QcAdEKhcOnSpeQorPHjx8+aNYvbkADgIcgfAPQiKiqqs7MTIWRubv7yyy8LhfBJAYAO+q8A6AVBEJMnT7579y5CqLCw0N3dneuIAOAd+FUFQC8EAsHy5csRQlOmTIHkAUCvxuL8u++//35BQQHXUQC+a2xsRAhZWFhERERwHQsYAQ4cOMB1CMNtLJ5/FBQUnD9/nusogCmdP3/e5MdUqVRaWVk5OjqattnhkZOTc//+fa6jGCvu37+fk5PDdRQcGIvnHwghLy+vMfhjYRTDpwgmP6b//e9/R+htgwKB4PXXX1+0aBHXgYwJ2dnZkZGRXEfBgbF4/gEAQyM0eQAwPCB/AAAAYAPyBwAAADYgfwAAAGAD8gcAY92dO3dCQ0MbGxtramoEv5o+fXpbWxu1GnWrQCDw8PDgKmDjjhw54uLiIhL1Pjho1qxZAgNarZasEB8fn5WVNVzBjmyQP8CY1tzc/Pjjj4/lhQWLioo8PDwCAgKUSqWtrS1BEDqdDpdTv1URQnhrQUGBjY0NQRCFhYUchdynkpKS0NDQhISEqqoq1o2sXLkyISFh06ZNJgxstIL8AcY0giB6enp6enq4CkAul3M4OWNjY+P8+fNffPHF1atXU8slEomNjU1aWtq+ffu4io2FTZs2zZw58+LFiwqFwkg1nU5H/N7OnTvJrc7OzgcPHkxKSsrOzh76kEc2yB9gTFMoFCUlJUeOHOE6EG7s2LFDr9dv3ryZVm5hYZGRkSEUCmNjY2/evMlJbCzs2bMnPj6+r54r5jQaTXh4+BtvvGG4kjGggvwBwBhFEER6evqMGTMcHBwMtwYGBiYmJjY1NUVERNAuhPCWVCo1VVMLFy68f//+4cOHTdXgqAT5A4xdhw4dIq+g4q9Iasnt27cjIyOtrKxsbGxCQkJKSkrws5KTk3GFSZMm6XQ6f39/hUJhaWnp6+t79uxZXGfr/2/vXoOauNoAAJ9FQgjBLCgKiLZarL0EGxSsMDajhRraImWgptgK05bBSakDUkttUautilZKOzpTHKmMo8NFRR1U7FjrUPlBjS0XE2otasVxlHBXbpGLyH4/zvftt12SsCQhi+z7/CJnT3bPbjRvdvfs++7YgfvQ16Z+/vln3OLl5cVcj9Fo/O233/Ai2384j4per29ublYoFOY6bN26VaVS1dbWpqSkWFhPe3v7+vXr/f39XVxcPD0933jjjYsXL+JFXI4n1trampqaOnv2bBcXl2nTpsXGxup0Orvs5nD5+fmBgYFSqZQkSaVSWVRUNLxPYGAgQuj8+fNjNIYJghKelStXrly5ku9RAHuy5TONjo5GCPX29rJaoqOjL1261NPTc+HCBYlEsmjRIua7FAqFVCoNDQ3FfSorK1966SUXF5fy8nK6j1QqXbJkCfNdQUFB+OazhT7Yq6++OmXKFK1Wa91OIYSOHTtmuU9+fj5CaOfOnaz2yspKkiTx362trTgDWEFBAW6h759jjY2Nc+bM8fb2Li0t7ezsvH79emxsLEEQBw4coPuMeDwNBsPTTz/t7e39008/dXd3X716denSpa6urpcuXbJu9/38/CZNmmRy0ZIlSxISEqqrq3t6eurq6hISEhBCKSkprG6dnZ0IIaVSyWVzeL6WdUN9oglxnyF+TDxjET9KS0uZ60cItba20i34Z/uVK1foltraWoSQQqGgW2yJH0uXLvX09LT6C5RL/MjKykII5eTksNqZ8YOiKK1WKxKJpFLp33//TQ2LHx988AFC6MiRI3RLX1/fjBkzJBJJU1MTbhnxeL7//vsIocLCQrpDY2OjWCwOCgoa1V7TLMSP4V5++WWE0OXLl1ntBEHMnTuXyxoEGz/g+hUApi1atIj+G/8GNxgMzA5SqRRf5cDmz58/Y8YMvV7f2Nho+9bLy8vv378fGhpq+6rMwZfs6DK95oSEhGRnZxuNRrVa3dvby1paUlKCEIqMjKRbxGJxeHh4b28v6+KPheN56tQpJycn5ixqHx8fuVxeXV3tgCzCOJiVlpay2p2dnYfvL2CC+AGAaSRJ0n+7uLgghFjTfD08PFhvmT59OkKopaVl7EdnB66urgghXKbXstTU1Li4uKtXr7Km+fb393d2drq6urLmy3p7eyOEmpqamI3mjideydDQEEmSzGf6ampqEEI3b960egc58vX1RaY+tcHBQTvekJ+QBJq/HQDbtbe3UxRFEATdgr+DcBRBCDk5OQ0MDDDf0tHRwVoJ8+0Ohr838YX+EeXl5el0uoMHD+Kog4nFYpIkOzs7u7u7mSEEP77n4+PDZc1isdjDw6Onp6e3t9fBMwgwfBpEf2pYV1cXRVH4EAFz4PwDACv19fXhR7WxP//802AwKBQK+kvH19e3oaGB7tDU1IQLqjO5ubnRMea555778ccfx3jU/xcQEIAQ4niByN3d/eTJk1KpdN++fcz2mJgYhBBznmt/f39ZWZlEIomIiOA4ktjY2MHBQXr2GrZ79+6nnnrKvk9g5OXlsaoRUxSFnxOMiopituMPDh8iYA7EDwCsRJLkxo0btVqt0WisqqqKj493cXHZu3cv3UGlUhkMhh9++KGnp+fWrVvr1q1j/chFCC1cuPDGjRt3797VarX19fVKpRK3h4WFTZ06dUwLZSoUiunTp+v1eo795XJ5bm4uq3HXrl1z5sxJS0s7e/Zsd3f3jRs33nvvvcbGxr179+KrWFzs2rXL398/MTHx3LlznZ2d9+/fz83N3bZtW3Z2Nn1GEh8fTxDE7du3Oa7TnJqamrVr1/7zzz99fX3Xr1/Hc7FSUlIWL17M7IZnD6tUKhs3N8HxfP+eDzD/auKx7jPF935pq1ev1mq1zJZNmzZRFMVsiYyMxO9VKBR+fn7Xrl2LiIiYPHmyRCJZunRpRUUFc/0dHR1JSUm+vr4SieSVV16prKykf/x+/vnnuE9dXZ1SqZRKpbNmzWJOhVIqlWM9/4qiqI0bNzo7Ozc0NOCXra2tzJ01Of0pOTmZNYWsra0tLS1tzpw5IpGIJMmIiIiysjK8iPvxxA+RPPPMMyKRaNq0aSqV6sKFC8ythIWFubu7Dw4OWtid4ffAEULMmcR9fX3Hjx+PiYnx9/fHF9+WLVtWVFQ0fFVqtdrPz29gYGDEY0gJeP6VEPcZ4sfE4/jPFMcPR25xVDjGj46ODj8/P41G44Ah2eLBgwcSiSQpKckxm9PpdARBMCclWybY+AHXrwAQLpIkS0tLT5w4kZOTw/dYzKIoKjU1VSaTbd++3QGbq6+vj42NzcjIWLVqlQM290SD+CFc7u7uzOmSTk5Onp6eCoXi448/rq6u5nt0wEEWLFhQVVV17ty5rq4uvsdiWnNzc319fVlZGccJXTbKzc3NzMzMzMx0wLaedBA/hKunp+fKlSsIoejoaIqiHj16VFdXt23btrq6uuDg4A8//PDhw4d8j3E8wnmr9Hp9Q0MDQRCbN2/me0S2mj179tmzZ2UyGd8DMc3Hx6eiokIulztmc7t374YzD44gfoy5sS7wYK/1T5o0ydvbOzo6+tdff92wYcOhQ4feffdd6t93OwFCKD09nXkJeMeOHXyPCAB+QPwAJnzzzTeLFy8+c+bM0aNH+R4LAGCcgvgBTCAIAmeqYD0sBgAANIgfllgobGBLgYcnooAE3u7ly5fp/EgWKjRwLPPQ39+/ZcuW559/3s3NbcqUKVFRUWfOnHn8+DHdwZFFIAAAtuJl1jC/OD4rwKWwgS0JusdDAQnm/XMWOvOowWCguFVoGLHMQ1JSEkmSv/zyy8OHD5uamtLT0xFCFy9exEttKQIBz/SwIG7PfwC7gOc/AFtGRsbt27f37NmzYsUKmUw2b968oqIiX1/f1NRUnB7Odkajcd++faGhoVKpNDg4uKCgYGBgYN26dXZZ+dDQEP6MrXs7640ZGRl37tz5/vvv33zzTXd3d7lcfvToUYqihlemS0pKwnv02muvRUZGVlZWtrW14UVlZWVyuXz58uUSicTb2/vbb7+dN2+eFZsAAIwLPMYuvnD8rYrTTeM0nDRcrezw4cP4pe3nH6xGXIka/+S3cf1cWDj/wNedRCIRTuFAkqSTk1NnZyezz8KFCxFCd+/exS/x+QddNYiiqE8++QQhpNfr8cvk5GSE0Jo1a7Ra7fBEFFw2YQ4u4QAAjzj9l5tYIH+7aaMqbGA1kwUkDAZDS0sL74mjKyoqEEKhoaEikQgfDfTvEg60mzdvzpw5k35poWxGTk5OaGjo4cOHw8PDEUJKpVKj0eAErqPahEkhISE4XAGEUFxcXFpa2piWnwI0rVa7Z88evkfBA4gfpnEsbGBjgYdxW0BiaGgIJ7RYu3Ytsl+FBoIgEhISEhISHj16VF5enp2dHRsb+913361fv972TcycOfOdd96xemwTTFxcXGhoKBwQhxFm/ID7H2ZxKWxgY4GHcVtAIiMj448//oiJiVGr1bjFLhUaPDw86urqEEIikWj58uV41hZ9hB1WBAIAYBcQP8ziUtjAlgIPaJwVkBgaGmppaTl9+nR4eHhWVlZiYmJhYSF9csOlQgMXH330UW1tbX9/f0tLS1ZWFkVRYWFh9t0EAMBB+L4BwwPucz0tFDbAbCnwwHsBCalUyvyXQBAESZLz589PTk6urq4e3t9ChQaOZR50Op1Go3nhhRfw8x8hISEHDhyg54lZ3oRlMH+XBcH8XQcS7PxdghJegiN8Teb48eP8DiMwMLCtrY1j9VBg2Tj5TMcPgiCOHTsG9z8co7i4OC4uToDfpXD9CgDAdufOnbfeequrq6utrY3OLLBgwYK+vj5mN+ZSgiCCg4P5GrBJDx482L9/f1hY2JQpUyQSybPPPrt69WpWvd4vvvgCnz0AK0D8AAD8i06nCw4OVqlUMpnMy8uLoig8y0On06WlpTF74qVarRY/k1RVVcXTkE377LPPUlJSoqOjr1271t7efvDgQZ1OFxQUdOrUKbrPmjVrMjIyvvzySx7H+eSC+MGDiVdAQmielJz8Vujq6oqKinr77bdxAk2aWCyeOnVqbm7ukSNHeBmYdRITE9etW+fj4+Pm5qZUKouKih4/frxhwwa6g7+/f0lJSWZmZnFxMY/jfEJB/OABFJAA41ZWVlZTU9OWLVtY7a6uroWFhU5OThqN5saNG7yMbbTy8vJyc3OZLQqFQiKR3Lp1i3mvQqFQrFy58tNPP4Vp4qMF8QMA8F8UReXl5S1evBjn0WGJiIjYvHlzd3e3Wq1m3Qh5UhiNxt7e3oCAANZTtzExMffu3WM+7AW4gPgBhEXIOflHpNfrm5ubFQqFuQ5bt25VqVS1tbWWk1paOMgcU/2jsUnmj2fobdq0idUeGBiIEDp//ryN6xcch88Y5h88KzDxQE5+FmTV8x/5+fkIoZ07d7LaKysrSZLEf7e2ts6aNQshVFBQgFvo++cYl4M8Yqp/W5L5m9PU1OTt7Z2UlDR8Ec69plQqrVuzYJ//gPMPICACz8k/osbGRmQmhSXNy8uruLhYJBJpNBqcjYaF+0G2kOrf7sn829vbX3/99WXLlu3fv3/4UplMRhAE3n3AHcQPICAlJSUIocjISLpFLBaHh4f39vba69qFVCrFF0Ow+fPnz5gxQ6/X2+W7qby8/P79+2OXVRff1RCJRJa7hYSEZGdnG41GtVpN1xmjcT/IixYtov/G5zQGgwG/PHXqlJOT04oVK+gOPj4+crm8urraikdujUZjRETEiy++WFhYOGnSJJN9nJ2dh+8LsAziBxAKHnPyo/9lVh7nXF1dEUJ0xWILUlNT4+Lirl69yprmO6qDbC7VP17J0NAQSZLM5xNramoQQjdv3hzVTg0ODqrVaj8/v8OHD5sLHribRCIZ1ZoBpKUDQiHwnPxc4MTP+GbAiPLy8nQ63cGDB3HUwTgeZMvsVS8A02g0/f39JSUl9Krmzp1bUFAQEhJC98Fl4ngvuvPEgfMPICBCzsnPRUBAAEKI4wUid3f3kydPSqXSffv2Mdu5HOQR2SuZ/1dfffXXX3+dPn1aLBZb6IY/Ebz7gDuIH0BAhJaTf7QUCsX06dNZGaIskMvlrAf0ELeDPCIuyfzj4+MJgrh9+7a5lRw6dOjrr7/+/fffJ0+ezLwOxpoojBDCM4NVKhXH4YH/4nHuF19g/u7EAzn5WZC1+ds3btzo7Ozc0NCAX7a2tjK/LoKCgoa/JTk5mTX/2MJB5pjqn+KQzD8sLMzd3X1wcNDcvjDv4bOw5kDjGyQDAwOjPl4URQl4/q4Q9xnix8QzTj5THD/4HgVF2RA/Ojo6/Pz8NBqN3YdkXw8ePJBIJCYf5hgtnU5HEMSRI0esXoNg4wdcvwIA/B9JkqWlpSdOnMjJyeF7LGZRFJWamiqTybZv327jqurr62NjYzMyMlatWmWXsQkKxA8AwL8sWLCgqqrq3LlzXV1dfI/FtObm5vr6+rKyMo4TuizIzc3NzMzMzMy0y8CEBuIHAHYwwXLyz549++zZszKZjO+BmObj41NRUSGXy21f1e7du+HMw2rw/AcAdpCenp6ens73KABwKDj/AAAAYA2IHwAAAKwB8QMAAIA1IH4AAACwhkDvn9+7d6+4uJjvUQC7wSmb4DNlYj3pDcaOYA81QY1ZLZpxS61Wnzhxgu9RAAAmFAF+lwoxfgAAALAd3P8AAABgDYgfAAAArAHxAwAAgDUgfgCqstZlAAAACklEQVQAALDGfwCMwtat6S/qEQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "execution_count": 12
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "source": [
    "model_2.fit(**hyperparameters_2)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Epoch 1/10\n",
      "320/320 [==============================] - 29s 81ms/step - loss: 0.4130 - accuracy: 0.8312 - val_loss: 0.3323 - val_accuracy: 0.8602\n",
      "Epoch 2/10\n",
      "320/320 [==============================] - 25s 79ms/step - loss: 0.2828 - accuracy: 0.8889 - val_loss: 0.3205 - val_accuracy: 0.8716\n",
      "Epoch 3/10\n",
      "320/320 [==============================] - 25s 77ms/step - loss: 0.2386 - accuracy: 0.9107 - val_loss: 0.3210 - val_accuracy: 0.8725\n",
      "Epoch 4/10\n",
      "320/320 [==============================] - 24s 76ms/step - loss: 0.2124 - accuracy: 0.9209 - val_loss: 0.3401 - val_accuracy: 0.8707\n",
      "Epoch 5/10\n",
      "320/320 [==============================] - 25s 77ms/step - loss: 0.1907 - accuracy: 0.9298 - val_loss: 0.3552 - val_accuracy: 0.8654\n",
      "Epoch 6/10\n",
      "320/320 [==============================] - 24s 74ms/step - loss: 0.1752 - accuracy: 0.9363 - val_loss: 0.3658 - val_accuracy: 0.8646\n",
      "Epoch 7/10\n",
      "320/320 [==============================] - 23s 72ms/step - loss: 0.1636 - accuracy: 0.9390 - val_loss: 0.3769 - val_accuracy: 0.8646\n",
      "Epoch 8/10\n",
      "320/320 [==============================] - 23s 73ms/step - loss: 0.1552 - accuracy: 0.9411 - val_loss: 0.3877 - val_accuracy: 0.8593\n",
      "Epoch 9/10\n",
      "320/320 [==============================] - 24s 76ms/step - loss: 0.1420 - accuracy: 0.9483 - val_loss: 0.4108 - val_accuracy: 0.8602\n",
      "Epoch 10/10\n",
      "320/320 [==============================] - 25s 78ms/step - loss: 0.1337 - accuracy: 0.9505 - val_loss: 0.4399 - val_accuracy: 0.8514\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "source": [
    "!tensorboard --logdir ./tb_log_sentiment --port 6006"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "2021-10-29 22:24:59.870715: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/cuda/include:/usr/lib/cuda/lib64:\n",
      "2021-10-29 22:24:59.870745: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "2021-10-29 22:25:01.069888: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2021-10-29 22:25:01.070465: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/cuda/include:/usr/lib/cuda/lib64:\n",
      "2021-10-29 22:25:01.070563: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/cuda/include:/usr/lib/cuda/lib64:\n",
      "2021-10-29 22:25:01.070653: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/cuda/include:/usr/lib/cuda/lib64:\n",
      "2021-10-29 22:25:01.072793: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/cuda/include:/usr/lib/cuda/lib64:\n",
      "2021-10-29 22:25:01.072895: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib/cuda/include:/usr/lib/cuda/lib64:\n",
      "2021-10-29 22:25:01.073019: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1835] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.\n",
      "Skipping registering GPU devices...\n",
      "W1029 22:25:06.098206 139927025317696 server_ingester.py:187] Failed to communicate with data server at localhost:43825: <_InactiveRpcError of RPC that terminated with:\n",
      "\tstatus = StatusCode.DEADLINE_EXCEEDED\n",
      "\tdetails = \"Deadline Exceeded\"\n",
      "\tdebug_error_string = \"{\"created\":\"@1635521106.097862666\",\"description\":\"Deadline Exceeded\",\"file\":\"src/core/ext/filters/deadline/deadline_filter.cc\",\"file_line\":81,\"grpc_status\":4}\"\n",
      ">\n",
      "W1029 22:25:06.399804 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.408351 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.414807 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.421055 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.427793 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.433781 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.440665 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.447593 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.453762 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.460237 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.466703 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.471186 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.477528 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.482867 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.487964 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.494898 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.501519 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.507227 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.513080 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.518752 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.524842 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.535657 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.540503 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.547205 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "W1029 22:25:06.553890 139925910603520 plugin_event_accumulator.py:317] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.\n",
      "Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all\n",
      "TensorBoard 2.7.0 at http://localhost:6006/ (Press CTRL+C to quit)\n",
      "^C\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "![](./images/13.png)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "> **Nhận xét**:\n",
    "> * Với việc ta tự định nghĩa hyper-params set, kết quả đem lại không tốt hơn so với model 1. Có thể thấy rõ accuracy và loss validation trên training data (đường màu xám) và validation data (đường màu cam) bắt đầu khác nhau và khoảng cách khác biệt là lớn - không được ổn định bằng model 1.\n",
    "> * Ta có thể update weight tại epocg thứ 2 xem có cải thiện hơn không."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "source": [
    "prediction_2 = model_2.predict(X_test['normalize_comment'])\n",
    "prediction_2"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input</th>\n",
       "      <th>output_proba</th>\n",
       "      <th>output_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bán hàng kỉ quấn đầy đủ chất lượn...</td>\n",
       "      <td>(7.5561167e-07, 0.9999993)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>đặt xl áo ghi xl nhưng bé tí tẹo ngang ...</td>\n",
       "      <td>(0.92751795, 0.07248204)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>kẹp rất rất đẹp săn được giá sale h...</td>\n",
       "      <td>(0.07028565, 0.9297143)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>không nghĩ áo đẹp giao hàng nhanh không ...</td>\n",
       "      <td>(0.13486566, 0.8651343)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>hôm nay mới nhận được nhưng nhìn qua...</td>\n",
       "      <td>(0.9999882, 1.1852359e-05)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2837</th>\n",
       "      <td>đặt dây buộc tóc nhưng chỉ nhận đưo...</td>\n",
       "      <td>(0.9999896, 1.0315402e-05)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2838</th>\n",
       "      <td>chất vải đẹp có vẻ mát giao hàng nhanh...</td>\n",
       "      <td>(0.00067545846, 0.99932456)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2839</th>\n",
       "      <td>dây nhìn tạm không đẹp mắc sai dây pha...</td>\n",
       "      <td>(0.99999, 1.001526e-05)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2840</th>\n",
       "      <td>quần đẹp chất lượng ok nhưng chữ cu...</td>\n",
       "      <td>(0.981992, 0.018007934)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2841</th>\n",
       "      <td>ok hơi chật tí</td>\n",
       "      <td>(0.0009869847, 0.99901295)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2842 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  input  \\\n",
       "0     bán hàng kỉ quấn đầy đủ chất lượn...   \n",
       "1     đặt xl áo ghi xl nhưng bé tí tẹo ngang ...   \n",
       "2     kẹp rất rất đẹp săn được giá sale h...   \n",
       "3     không nghĩ áo đẹp giao hàng nhanh không ...   \n",
       "4     hôm nay mới nhận được nhưng nhìn qua...   \n",
       "...                                                 ...   \n",
       "2837  đặt dây buộc tóc nhưng chỉ nhận đưo...   \n",
       "2838  chất vải đẹp có vẻ mát giao hàng nhanh...   \n",
       "2839  dây nhìn tạm không đẹp mắc sai dây pha...   \n",
       "2840  quần đẹp chất lượng ok nhưng chữ cu...   \n",
       "2841                                 ok hơi chật tí   \n",
       "\n",
       "                     output_proba  output_class  \n",
       "0      (7.5561167e-07, 0.9999993)             1  \n",
       "1        (0.92751795, 0.07248204)             0  \n",
       "2         (0.07028565, 0.9297143)             1  \n",
       "3         (0.13486566, 0.8651343)             1  \n",
       "4      (0.9999882, 1.1852359e-05)             0  \n",
       "...                           ...           ...  \n",
       "2837   (0.9999896, 1.0315402e-05)             0  \n",
       "2838  (0.00067545846, 0.99932456)             1  \n",
       "2839      (0.99999, 1.001526e-05)             0  \n",
       "2840      (0.981992, 0.018007934)             0  \n",
       "2841   (0.0009869847, 0.99901295)             1  \n",
       "\n",
       "[2842 rows x 3 columns]"
      ]
     },
     "metadata": {},
     "execution_count": 15
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "source": [
    "Model.confusionMatrix(y_test, prediction_2['output_class'])"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "    Negative       0.86      0.86      0.86      1410\n",
      "    Positive       0.86      0.86      0.86      1432\n",
      "\n",
      "    accuracy                           0.86      2842\n",
      "   macro avg       0.86      0.86      0.86      2842\n",
      "weighted avg       0.86      0.86      0.86      2842\n",
      "\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAEWCAYAAACZnQc8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAo30lEQVR4nO3debwf0/3H8dc7uUlks0RUSSxBFFV77VtLS0KL1q6KIva2lCr1s3WjtlpqiajGUmINtRS1VKiE2AkqDSqx77Em997P7485N/km7jJ3mXu/mbyfHvP4zpyZ75zzvbk+37lnznyOIgIzMyuHbl3dADMz6zgO6mZmJeKgbmZWIg7qZmYl4qBuZlYiDupmZiXioG7tJqm3pL9L+lDSte04zx6S7uzItnUVSZtIeqGr22HzH3mc+vxD0u7AEcBKwHTgCeB3EfFAO8+7J3AYsGFE1La3ndVOUgBDI2JyV7fFbG6+Up9PSDoC+BPwe2BxYGngfGC7Djj9MsB/5oeAnoekmq5ug83HIsJLyRdgIeBjYKdmjulFFvRfS8ufgF5p3+bAVOAXwFvA68A+ad9JwAxgZqpjX+BE4IqKcy8LBFCTtvcGppD9tfASsEdF+QMV79sQeAT4ML1uWLHvPuA3wIPpPHcCA5v4bA3t/2VF+7cHhgP/Ad4Djq04fl3gIeCDdOx5QM+07/70WT5Jn3eXivMfDbwBXN5Qlt6zfKpjrbS9JPA2sHlX/254Kd/iK/X5wwbAAsCNzRzza2B9YA1gdbLAdlzF/q+SfTkMIgvcf5a0SEScQHb1PyYi+kXEJc01RFJf4BxgWET0JwvcTzRy3ADg1nTsosCZwK2SFq04bHdgH+ArQE/gyGaq/irZz2AQcDxwMfAjYG1gE+D/JA1Jx9YBhwMDyX52WwAHA0TEpumY1dPnHVNx/gFkf7WMqKw4Iv5LFvCvkNQHuBQYHRH3NdNeszZxUJ8/LAq8E813j+wBnBwRb0XE22RX4HtW7J+Z9s+MiNvIrlK/1sb21AOrSuodEa9HxLONHLMN8GJEXB4RtRFxFfA88L2KYy6NiP9ExGfANWRfSE2ZSXb/YCZwNVnAPjsipqf6J5F9mRERj0bE+FTvy8BFwGY5PtMJEfFFas8cIuJiYDIwAViC7EvUrMM5qM8f3gUGttDXuyTwSsX2K6ls1jnm+lL4FOjX2oZExCdkXRYHAq9LulXSSjna09CmQRXbb7SiPe9GRF1abwi6b1bs/6zh/ZJWlHSLpDckfUT2l8jAZs4N8HZEfN7CMRcDqwLnRsQXLRxr1iYO6vOHh4AvyPqRm/IaWddBg6VTWVt8AvSp2P5q5c6IuCMivkN2xfo8WbBrqT0NbZrWxja1xgVk7RoaEQsCxwJq4T3NDiOT1I/sPsUlwImpe8mswzmozwci4kOyfuQ/S9peUh9JPSQNk/THdNhVwHGSFpM0MB1/RRurfALYVNLSkhYCjmnYIWlxSdulvvUvyLpx6hs5x23AipJ2l1QjaRdgFeCWNrapNfoDHwEfp78iDppr/5vAcq0859nAxIjYj+xewYXtbqVZIxzU5xMRcQbZGPXjyEZevAocCoxNh/wWmAg8BTwNPJbK2lLXXcCYdK5HmTMQd0vteI1sRMhmfDloEhHvAtuSjbh5l2zkyrYR8U5b2tRKR5LdhJ1O9lfEmLn2nwiMlvSBpJ1bOpmk7YCtmf05jwDWkrRHh7XYLPHDR2ZmJeIrdTOzEnFQNzMrEQd1M7MScVA3MyuRqk08NOPVJ30H176kz/LDu7oJVoVqZ0xr6TmCFs18Z0rumNNj4HLtrq8oVRvUzcw6VX1dy8fMAxzUzcwAorFn4OY9DupmZgD1DupmZqURJblS9+gXMzOAutr8Swsk/UXSW5KeqSg7TdLzkp6SdKOkhSv2HSNpsqQXJG1VUb51Kpss6Vd5PoaDupkZZDdK8y4t+ytZvp9KdwGrRsRqZDNuHQMgaRVgV+Dr6T3nS+ouqTvwZ2AYWTK73dKxzXJQNzOD7EZp3qWlU0XcT5awrrLszoo5CcYDg9P6dsDVaYKVl8gmU1k3LZMjYkpEzCCb3KXFOYUd1M3MILtRmnORNELSxIplRMsVzOEnwO1pfRBZ1tQGU1NZU+XN8o1SMzNad6M0IkYCI9tSj6RfA7XAlW15f0sc1M3MoFOGNEram2yegC1idt7zacBSFYcNZvYMX02VN8lB3cwMoG5moaeXtDXZZC+bRcSnFbtuBv4m6UyyuXmHAg+TTaE4VNIQsmC+K9nkLc1yUDczgw59olTSVcDmZBO+TwVOIBvt0gu4SxLA+Ig4MCKelXQNMImsW+aQhknSJR0K3AF0B/4SEc+2WHe1znzkhF7WGCf0ssZ0REKvL569O3fM6fX1LZzQy8ysqpXkiVIHdTMzcO4XM7Myifpib5R2Fgd1MzPwlbqZWam4T93MrEQ885GZWYn4St3MrETcp25mViI5Jr+YFziom5mBr9TNzMokpVuZ5zmom5mBr9TNzErFo1/MzErEV+pmZiXi0S9mZiXi7hczsxJx94uZWYk4qJuZlYi7X8zMSsQ3Ss3MSqQk3S/diq5A0jKStkzrvSX1L7pOM7NWi/r8SxUrNKhL2h+4DrgoFQ0GxhZZp5lZm9TX51+qWNHdL4cA6wITACLiRUlfKbhOM7PWq/JgnVfRQf2LiJghCQBJNUAUXKeZWetFOUJT0UH9X5KOBXpL+g5wMPD3gus0M2u92nKMfin6RumvgLeBp4EDgNuA4wqu08ys9Upyo7ToK/Xtgcsi4uKC6zEza5+S9KkXfaX+PeA/ki6XtG3qUzczqz4R+ZcqVmhQj4h9gBWAa4HdgP9KGlVknWZmbeIhjflExExJt5ONeulN1iWzX9H1mpm1SpUH67wKDeqShgG7AJsD9wGjgJ2LrNPMrC2izhNP5/FjYAxwQER8UXBdZmZtV5Ir9aL71HeLiLEO6GZW9TpwSKOkv0h6S9IzFWUDJN0l6cX0ukgql6RzJE2W9JSktSres1c6/kVJe+X5GIUEdUkPpNfpkj6qWKZL+qiIOs3M2qU+8i8t+yuw9VxlvwLujoihwN1pG2AYMDQtI4ALIPsSAE4A1iNLt3JCwxdBcwoJ6hGxcXrtHxELViz9I2LBIuo0M2uXDhz9EhH3A+/NVbwdMDqtjyYbNNJQfllkxgMLS1oC2Aq4KyLei4j3gbv48hfFlxSdpfHyPGVmZl2uri73ImmEpIkVy4gcNSweEa+n9TeAxdP6IODViuOmprKmyptV9I3Sr1dupIeP1i64znnC/512PvdPeIwBCy/EjaPOAOCMiy7nvvGP0qOmhqWWXJzfHHUwC/brywcfTueIk8/kmRcms91Wm/Prw/YF4LPPv+AXJ5/Jq6+/Sfdu3dhs/bU5fP89uvJjWQe6eOQZbDN8S956+x3WWHMLAFZbbRXOP+8U+vbrwyuvTGXPHx/K9OkfU1NTw8iLTmfNNVelpqaGK664jlP/eF4Xf4J5TCtulEbESGBkW6uKiJBUyFNMRfWpHyNpOrBaZX868CZwUxF1zmu222pzLvjDsXOUbbD2atw46gxuuPh0lhm8BKOuuhGAnj17cOjeu3DkAXt+6Tx77/w9/n7pn7j2wj/yxLMvMO7hxzul/Va8yy67hm22nfNL+qILT+PYX/+eNdfakrFjb+fIXxwEwI47bkuvXj1Zc60tWXe9rdl/vx+xzDKDu6LZ866O7VNvzJupW4X0+lYqnwYsVXHc4FTWVHmziupT/0NE9AdOm6s/fdGIOKaIOuc166y2Cgv17zdH2YbrrE5N9+4ArL7yirz5dtYl16f3Aqz1jZXo2bPnHMf3XqAX666xKgA9etSw8tAhvPn2u53QeusM4x6YwHvvfzBH2YpDl+P+ceMB+Ofd49hhh+EARAR9+/ahe/fu9O7dmxkzZ/LRRx93dpPnbcUn9LoZaBjBshezL3BvBn6cRsGsD3yYumnuAL4raZF0g/S7qaxZRQ9pPCY1aF1JmzYsRdZZFjf+4x42XneN3Md/9PEn3PfQo6y35jeKa5R1uUmT/sP3v78VADv+cFuWGrwkANdffyuffPIpU//3OC/992HOPPNC3p/rC8Fa0IFX6pKuAh4CviZpqqR9gVOA70h6EdgybUOWvXYKMBm4mCxFORHxHvAb4JG0nJzKmlX0jdL9gPvJvl1OSq8nNnP8rJsPo668rsimVbWRV95A9+7d2XaLTXIdX1tXxy9/dzZ77DCMpZZcvOU32DxrvxFHcNABezFh/O3079+XGTNmArDuN9egrq6OpZZZixVWXJ/DDz+AIUOW7uLWzluivj730uK5smd0loiIHhExOCIuiYh3I2KLiBgaEVs2BOg06uWQiFg+Ir4RERMrzvOXiFghLZfm+RxF3yj9GfBNYHxEfEvSSsDvmzq48ubDjFefrO5UaAUZe8d9/Gv8o4w67XgaZoxqyUlnXsQyg77Knj/cpuDWWVd74YX/Mmyb3QEYOnQ5hg/LbqDuuusO3HHnfdTW1vL22+/y738/wtprr85LL/2vK5s7bylJmoCiU+9+HhGfA0jqFRHPA18ruM551gMPP8GlY27i3N8cTe8FeuV6zzl/uZqPP/mUow/eu9jGWVVYbLFFAZDEscf8jItGZiOEX311Gt/afCMA+vTpzXrrrcULL0zusnbOk4q/UdopFAXmBpZ0I7AP8HPg28D7QI+IGN7Se8t+pf7L3/2JR56cxAcfTmfAIgtxyF47M+qqG5kxs5aFF8xuoK628lCO/3k2/HWrPQ7h408/ZebMWvr368vIU4+jb5/efGe3gxiy9CB69sj+6Nptu6354fAtuuxzFa3P8i3+6pTGFZf/mc023YCBAwfw5pvvcNLJp9OvX18OOmhvAMaOvY1jf/0HAPr27cMlo85i5ZWHIonRo8dwxpkXdmHrO1ftjGn5/qxtxicn7pY75vQ98ap211eUQoP6HBVJmwELAf+IiBktHV/2oG5tMz8FdcuvQ4L68bvmD+onX121Qb3o1LsDKjafTq8O1mZWfap87tG8ir5R+hjZ4Pn3AQELA29IehPYPyIeLbh+M7N8qryvPK+ib5TeBQyPiIERsShZNrJbyMZhnl9w3WZmuUVtXe6lmhUd1NePiFlPQEXEncAGKRNZvuEdZmadoSSjX4rufnld0tHA1Wl7F7L8B92BcnRgmVk5lKRPvegr9d3JktCMBW4k61/fHeiO5yo1s2riK/WWRcQ7wGGS+kbEJ3Pt9pMRZlY1osqDdV5F537ZUNIk4Lm0vbok3yA1s+pTW5d/qWJFd7+cRTYl07sAEfEk4CyNZlZ93P2ST0S8Oldiqur+mjOz+VOVB+u8ig7qr0raEAhJPciyNj5XcJ1mZq3WWSlTilZ0UD8QOJtsstRpwJ3AIQXXaWbWer5Sb1ka/eKZkM2s+jmoN03S8c3sjoj4TRH1mpm1VdSW4+Gjoq7U5x6TDtAX2BdYlGzePTOz6lGOmF5MUI+IMxrWJfUnu0G6D1m6gDOaep+ZWVcpy8NHhfWpp1zqR5D1qY8G1oqI94uqz8ysXRzUmybpNOAHZJNIfyMiPi6iHjOzDlOS7pcWnyiVtFPqQkHScZJukLRWC2/7BbAkcBzwmqSP0jJd0kftb7aZWceK+si9VLM8V+r/FxHXStoY2BI4DbgAWK+pN0RE0ekHzMw6VNRWd7DOK0/wbXisfxtgZETcCvQsrklmZl2gvhVLFctzpT5N0kXAd4BTJfWi+ERgZmadqiRzZOQKzjsDdwBbRcQHwADgqCIbZWbW6Upypd5iUI+IT4G3gI1TUS3wYpGNMjPrbFGff6lmLXa/SDoBWAf4GnAp0AO4Atio2KaZmXWeqO3qFnSMPH3qOwBrAo8BRMRrDUMczczKotqvwPPKE9RnRERICgBJfQtuk5lZpytLUM9zo/SaNPplYUn7A/8ELi62WWZmnSyUf6lieW6Ung5cB1xP1q9+fEScW3TDzMw6U0feKJV0uKRnJT0j6SpJC0gaImmCpMmSxkjqmY7tlbYnp/3Ltudz5BpvHhF3RcRREXFkRNzVngrNzKpR1Cv30hxJg4CfAutExKpAd2BX4FTgrIhYAXifLBU56fX9VH5WOq7N8uR+mV6Ru+VzSXXO32JmZVNfp9xLDjVAb0k1QB/gdeDbZL0ekGWu3T6tb5e2Sfu3kNTmPp4Wb5RGxKyRLqmi7YD121qhmVk16qgbpRExTdLpwP+Az8jmZn4U+CBi1sDJqWRzN5NeX03vrZX0IdlkQu+0pf5WPe4fmbHAVm2pzMysWrWm+0XSCEkTK5YRDeeRtAjZxe8Qsmy1fYGtO+tz5Hn46AcVm93IHkT6vLAWmZl1gWhFksaIGEk2X0RjtgReioi3ASTdQPaw5sKSatLV+mBgWjp+GrAUMDV11ywEvNuWzwD5xql/r2K9FniZ7FvIzKw0WroB2gr/A9aX1Ies+2ULYCJwL7Aj2bSeewE3peNvTtsPpf33RLTmK2ZOefrU92nryc3M5hU5b4C2KCImSLqO7Cn8WuBxsqv6W4GrJf02lV2S3nIJcLmkycB7ZCNl2qzJoC7pXKDJb4uI+Gl7KjYzqyYdeKVORJwAnDBX8RRg3UaO/RzYqaPqbu5KfWJHVWJmVu2iyp8UzavJoB4Ro5vaZ2ZWNmXJ/ZJn9MtiwNHAKsACDeUR8e0C22Vm1qnqS3Klnmec+pXAc2RjLk8iG/3ySIFtMjPrdBHKvVSzPEF90Yi4BJgZEf+KiJ+QPe5qZlYaHZwmoMvkGac+M72+Lmkb4DWyeUrNzEqjI0e/dKU8Qf23khYCfgGcCywIHF5oq8zMOllZ+tTzBPUJEfEh8CHwrYLbY2bWJaq9rzyvPH3qD0q6U9K+KVGNmVnpRORfqlmemY9WBI4Dvg48KukWST8qvGVmZp2oPpR7qWZ5Zz56OCKOIHvE9T1mJ3Q3MyuF+nrlXqpZnoePFgR2IEsyszxwI43kLzAzm5dV+xV4XnlulD4JjAVOjoiHim3ObH2WH95ZVdk85LPXxnV1E6ykynKjNE9QX649uX3NzOYF882VugO6mc0PyhLo8lypm5mVXl19q6ZsrloO6mZmQEky73rmIzMzgKD8feqe+cjM5hv1JelU98xHZmZA/XxwpQ545iMzmz+UpfvFMx+ZmQF1KPdSzTzzkZkZ2eiXvEs188xHZmZUf7DOyzMfmZlRnj71PGkCbkmrnvnIzEqryjPq5pZn9MulNPIQUupbNzMrhflmSCNwS8X6AmS51V8rpjlmZl2jrqsb0EHydL9cX7kt6SrggcJaZGbWBeo1/1ypz20o8JWOboiZWVcqSZaAXH3q05nz875B9oSpmVlpzDdDGiOif2c0xMysK5Vl9EuLT5RKujtPmZnZvKz0aQIkLSBpADBQ0iKSBqRlWWBQp7XQzKwT1Cv/0hJJC0u6TtLzkp6TtEGKn3dJejG9LpKOlaRzJE2W9JSktdrzOZq7Uj8AeBRYKb02LDcB57WnUjOzatPBuV/OBv4RESsBq5MlRfwVcHdEDAXuTtsAw8gGoAwFRgAXtOdzNJdP/WzgbEmHRcS57anEzKzaddTol5RWZVNgb4CImAHMkLQdsHk6bDRwH9mgk+2AyyIigPHpKn+JiHi9LfXnydJYL2nhigYvIungtlRmZlatWtP9ImmEpIkVy4iKUw0B3gYulfS4pFGS+gKLVwTqN4DF0/og4NWK90+lHV3ceYL6/hHxQcNGRLwP7N/WCs3MqlFrul8iYmRErFOxjKw4VQ2wFnBBRKwJfMLsrhbI3h8UNDQ+T1DvLs1+1EpSd6BnEY0xM+sqdcq/tGAqMDUiJqTt68iC/JuSlgBIr2+l/dOApSrePziVtUmeoP4PYIykLSRtAVyVyszMSqOjbpRGxBvAq5K+loq2ACYBNwN7pbK9yAadkMp/nEbBrA982Nb+dMiXJuBosjuyB6Xtu4CL21qhmVk16uAnSg8DrpTUE5gC7EN2EX2NpH2BV4Cd07G3AcOBycCn6dg2y/NEaT1wYVqQtAnZZBmHtKdiM7Nq0pEd3BHxBLBOI7u2aOTYoAPjaa6EXpLWBHYj+2Z5CbihoxpgZlYNypImoMmgLmlFskC+G/AOMAZQRHj2IzMrnfkhodfzwDhg24iYDCDJc5OaWSmVZZKM5ka//AB4HbhX0sVp5EtJ/kAxM5tTR+Z+6UpNBvWIGBsRu5LlfrkX+DnwFUkXSPpuJ7XPzKxTdHDuly7T4jj1iPgkIv4WEd8jGxT/OJ4kw8xKJlqxVLNWTWeXUgSMTIuZWWnUV324zqctc5SamZVOWW6UOqibmVH9feV55cn90maSVpR0t6Rn0vZqko4rsk4zs7Yo/eiXDnIxcAwwEyAingJ2LbhOM7NWqydyL9Ws6O6XPhHxcEXmXoDagus0M2u16g7V+RUd1N+RtDzp5yVpR7IHmszMqkpZ+tSLDuqHkA1/XEnSNLJkYHsUXKeZWavVleRaveig/kpEbJnm5+sWEdMLrs/MrE3KcqVe9I3SlySNBNYHPi64LjOzNivLjdKig/pKwD/JumFeknSepI0LrtPMrNXKkiag0KAeEZ9GxDUR8QNgTWBB4F9F1mlm1hbzTUKv9pK0maTzgUeBBZg9L5+ZWdWoI3Iv1azQG6WSXibL6ngNcFREfFJkfWZmbVXtfeV5FT36ZbWI+KjgOuZ5F488g22Gb8lbb7/DGmtm89KuttoqnH/eKfTt14dXXpnKnj8+lOnTP2aZZQbzzFP38cJ/pgAwYcJjHHLor7qy+daBjvv9mdz/4MMMWGRhxl5xIQCnnzeKfz04gZoeNSw1aAl+e+wRLNi/H09PeoETTz0HgCA4+Cd7sOVmG/HSK1M58vg/zDrn1Nde59D99mTPXXboks80ryhHSM/mHO34k0q/jIg/SjqXRn5WEfHTls5R03NQWX7GLdpk4/X4+ONPuPTSs2cF9Yf+fStHH/0b7h83nr332oUhQ5bmhBNPY5llBnPT2NGzjpvffPbauK5uQqEmPvE0fXr35tjfnD4rqD844VHWW3sNamq6c+b5lwBwxMH78tnnn9Ojpgc1Nd15+533+OFeB3PPTVdSU9N91vnq6ur49vZ7ctXFZ7HkVxfvks/UGXoMXK7dGVkOWHan3DHnopevrdoMMEX1qT+XXieS9aXPvViFcQ9M4L33P5ijbMWhy3H/uPEA/PPuceyww/AuaJl1tnXW+AYLLdh/jrKN1lt7VqBe7esr8eZb7wDQe4EFZpV/MWMG6MtxZvzEJ1hq0BKlDugdxTdKmxERf0+rn0bE6MoF+LSIOstm0qT/8P3vbwXAjj/clqUGLzlr35Bll+aRh+/gnn9ex8YbrdtVTbQucOOtd7LxBt+ctf3Us8+z3R4HsMOPD+L4ow6d4yod4Pa7/8XwLTfr7GbOk6IV/1Wzoke/HJOzDABJIyRNlDSxvn7+vqe634gjOOiAvZgw/nb69+/LjBkzAXj99bcYsvy6fHPdrTjyqJO4/LI/079/vy5urXWGi0ZfRffu3dn2u9+aVbba11fipisv4upRZzPq8mv44osZs/bNnDmT+x6YwHe/vUlXNHee49EvzZA0DBgODJJ0TsWuBWkmS2NEzJoqb37qU2/MCy/8l2Hb7A7A0KHLMXxY1oc+Y8YM3nsv+x/3scefZsqUl1lx6HI8+thTXdZWK97YW+/i/gcfZtQ5f0CNdLMsv+zS9OndmxenvMyqK68IwLjxE1l5xeUZOGCRzm7uPKnau1XyKupK/TWy/vTPmbMv/WZgq4LqLJXFFlsUAEkce8zPuGjk5QAMHDiAbt2yf7YhQ5ZmhRWGMOWl/3VZO614D4yfyF/+di3nnnoCvRdYYFb51NfeoLY2m4TttTfe5KVXXmXQErP7zm+76z6Gf2fzzm7uPKs+IvdSzQq5Uo+IJ4EnJV0ZEc6f3oIrLv8zm226AQMHDuDlKRM56eTT6devLwcdtDcAY8fexl9HjwFgk03W58QTjmTmzFrq6+s55NBjeH+um6w27zrqhFN45PGn+OCDj9hi+x9x8L57MuryMcyYOZP9f/5rIOtyOeGXh/HYU89yyeXXUFNTQ7du4rgjD2GRhRcC4NPPPuehRx7nhF+2ONDMkuoO1fkVNaTxmojYWdLTzPmzEhARsVpL55jfu1+scWUf0mht0xFDGndfZofcMedvr9xYtUMai3r46GfpdduCzm9m1qGqfVRLXkV1vzTMbvQO8FlE1EtakSxr4+1F1Glm1h61JQnqRQ9pvB9YQNIg4E5gT+CvBddpZtZqHqeejyLiU+AHwPkRsRPw9YLrNDNrNT9Rmo8kbUA2L+mtqax7M8ebmXWJiMi95CGpu6THJd2StodImiBpsqQxknqm8l5pe3Lav2x7PkfRQf3nZE+Q3hgRz0paDri34DrNzFqtgOnsfsbsPFgApwJnRcQKwPvAvql8X+D9VH5WOq7Nip756F8R8X3gz5L6RcSUPBkazcw6W0emCZA0GNgGGJW2BXwbuC4dMhrYPq1vl7ZJ+7dQY48N51RoUJf0DUmPA88CkyQ9Ksl96mZWdVpzpV6ZpyotI+Y63Z+AXzK7C35R4IOKhzGnAoPS+iDgVYC0/8N0fJsUPUnGRcAREXEvgKTNgYuBDQuu18ysVVrzIGZlnqq5SdoWeCsiHk0xr1MVHdT7NgR0gIi4T1Lfgus0M2u1DhzVshHwfUnDyeZlXhA4G1hYUk26Gh8MTEvHTwOWAqZKqgEWAt5ta+VF3yidIun/JC2bluOAKQXXaWbWah01Tj0ijomIwRGxLLArcE9E7EE2SGTHdNhewE1p/ea0Tdp/T7Qjf0vRQf0nwGLADcD1wMBUZmZWVQoY/TK3o4EjJE0m6zO/JJVfAiyayo8A2jXpcFH51BcADgRWAJ4GfhERM4uoy8ysI9RFxz9WFBH3Afel9SnAl6Yqi4jPgZ06qs6i+tRHAzOBccAwYGWyMetmZlWp2h//z6uooL5KRHwDQNIlwMMF1WNm1iGqffKLvIoK6rO6WiKith3j6M3MOkU5QnpxQX11SR+ldQG903bDJBkLFlSvmVmbtOMGaFUpKp+6k3aZ2TzFQd3MrESKGP3SFRzUzczw6Bczs1Jpx0OcVcVB3cwM96mbmZWKr9TNzEqkrupnH83HQd3MDD9RamZWKh79YmZWIr5SNzMrEV+pm5mViK/UzcxKxGkCzMxKxN0vZmYlEr5SNzMrD6cJMDMrEacJMDMrEV+pm5mVSF29+9TNzErDo1/MzErEfepmZiXiPnUzsxLxlbqZWYn4RqmZWYm4+8XMrETc/WJmViJOvWtmViIep25mViJluVLv1tUNMDOrBvVRn3tpjqSlJN0raZKkZyX9LJUPkHSXpBfT6yKpXJLOkTRZ0lOS1mrP53BQNzMju1Gad2lBLfCLiFgFWB84RNIqwK+AuyNiKHB32gYYBgxNywjggvZ8Dgd1MzM6LqhHxOsR8Vhanw48BwwCtgNGp8NGA9un9e2AyyIzHlhY0hJt/RwO6mZmQLRikTRC0sSKZURj55S0LLAmMAFYPCJeT7veABZP64OAVyveNjWVtUnV3iitnTFNXd2GaiFpRESM7Op2WHXx70XHakPMafZnL6kfcD3w84j4SJp9+ogISYXcmfWV+ryh0asAm+/596JKSepBFtCvjIgbUvGbDd0q6fWtVD4NWKri7YNTWZs4qJuZdSBll+SXAM9FxJkVu24G9krrewE3VZT/OI2CWR/4sKKbptWqtvvFzGwetRGwJ/C0pCdS2bHAKcA1kvYFXgF2TvtuA4YDk4FPgX3aU7nKku+gzNx3ao3x74U1xkHdzKxE3KduZlYiDupmZiXioN7BJIWkMyq2j5R0YgH1HDvX9r87ug4rhqQ6SU9IekbStZL6tPL9S0q6Lq2vIWl4xb7vS/pV0++2snNQ73hfAD+QNLDgeuYI6hGxYcH1Wcf5LCLWiIhVgRnAga15c0S8FhE7ps01yEZONOy7OSJO6bCW2jzHQb3j1ZI9aXb43DskLSbpekmPpGWjivK7Uka3UZJeafhSkDRW0qNp34hUdgrQO13tXZnKPk6vV0vapqLOv0raUVJ3Saelep+SdEDhPwnLYxywQsrgNzb924yXtBqApM3Sv/MTkh6X1F/SsukqvydwMrBL2r+LpL0lnSdpofR71C2dp6+kVyX1kLS8pH+k36txklbqws9vHa01SWy85Er08zGwIPAysBBwJHBi2vc3YOO0vjTZwwkA5wHHpPWtydJLDEzbA9Jrb+AZYNGGeuauN73uAIxO6z3Jckr0Jnv68LhU3guYCAzp6p/X/LhU/FvVkD2AchBwLnBCKv828ERa/zuwUVrvl96zLPBMKtsbOK/i3LO207m/ldZ3AUal9buBoWl9PeCerv6ZeOm4xQ8fFSCyPA+XAT8FPqvYtSWwSkUOiAVTfoiNyYIxEfEPSe9XvOenknZI60uRped8t5nqbwfOltSL7Avi/oj4TNJ3gdUkNfzZvlA610tt/ZzWZr0rHkoZR/b04QTghwARcY+kRSUtCDwInJn+IrshIqZW5hBpwRiyYH4vsCtwfvp92xC4tuI8vdr/kaxaOKgX50/AY8ClFWXdgPUj4vPKA5v6n1TS5mRfBBtExKeS7gMWaK7SiPg8HbcV2f/QVzecDjgsIu5o3cewAnwWEWtUFjT1OxARp0i6lazf/EFJWwGfN3rwl90M/F7SAGBt4B6gL/DB3PVbebhPvSAR8R5wDbBvRfGdwGENG5LWSKsPkh4ZTlfUi6TyhYD3U0BfiSzhfoOZKWlQY8aQPWq8CfCPVHYHcFDDeyStKKlv2z6dFWAcsAfM+jJ/J/3Ft3xEPB0RpwKPAHP3f08H+jd2woj4OL3nbOCWiKiLiI+AlyTtlOqSpNWL+EDWNRzUi3UGUDkK5qfAOulm2CRmj3o4CfiupGeAnchyLU8nC8g1kp4jyxsxvuJcI4GnGm6UzuVOYDPgnxExI5WNAiYBj6V6LsJ/qVWTE4G1JT1F9m/dkPjp5+mm6FPATLLutUr3knXpPSFpl0bOOwb4UXptsAewr6QngWfJJmmwknCagCqQ+r/rIqJW0gbABf7z2Mzawldq1WFpsuxt3cjGLe/fxe0xs3mUr9TNzErEfepmZiXioG5mViIO6mZmJeKgbnNobwbBuc7114YnWFNOm1WaOXZzSa1OSibp5Y5IntZR5zHrag7qNrdmMwhKatOIqYjYLyImNXPI5mSPr5tZOzioW3MaMghunrL53QxMairjY3o68TxJL0j6J/CVhhNJuk/SOml9a0mPSXpS0t2SliX78jg8/ZWwiZrOaLmopDuVMlqSpT+Yg6QDJZ1Wsb23pPPS+peyXs713mXTw1kN27Py4TeV3VDSTukvmycl3d/eH7pZe3icujUqXZEPY3aagbWAVSPipRQMP4yIb6YHpx6UdCewJvA1YBVgcbInWP8y13kXAy4GNk3nGhAR70m6kCx74enpuL8BZ0XEA5KWJktzsDJwAvBARJysLMVwZRqGBtcDDwFHpe1dgN+l9Z+k+noDj0i6PiKaS5BWaSRwYES8KGk94HyyjIrHA1tFxDRJC+c8l1khHNRtbo1lENwQeDgiGjI6NpXxcVPgqoioA16TdE8j51+fLHPkSzArR05jmspouSnwg/TeWzVnRktS+duSpkhaH3iRLF/Kg2l3a7NeAqDmsxs+CPxV0jXADS2dy6xIDuo2t6YyCH5SWUQjGR9VMa1aB2hVRstGXE2WJO154MaICOXLelnLnN2SDfu70UR2w4g4MF25bwM8KmntVlz9m3Uo96lbWzSV8fF+sll4uktaAvhWI+8dD2wqaUh674BUPne2waYyWt4P7J7KhjE7o+XcbiRLVLUbs9MPN5f1ssGbwFdS330vYFvIcuTTRHZDZZkUJ0TE8cDbZH8BmHUJB3Vri6YyPt5I1t0xCbiMrF97DhHxNtksTDcoyxLYkD3w78AODTdKaT6j5aaSniXrhvlfYw2MiPeB54BlIuLhVNxc1suG980kmyLuYeAusiv9Bk1lNzxN0tPpZ/Fv4MnGf2xmxXPuFzOzEvGVuplZiTiom5mViIO6mVmJOKibmZWIg7qZWYk4qJuZlYiDuplZifw/E/rEAJkaqY0AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     }
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "> **Nhận xét**:\n",
    "> * Độ chính xác của model 2 này là 86 phần trăm.\n",
    "> * Nhưng ta nên đặt câu hỏi vì sao đã qua 2 model có kiến trúc khác nhau rất nhiều nhưng độ chính xác vẫn loanh quanh 86, 87%."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "source": [
    "model_uw_2 = DeepModel.SentimentLSTM()\n",
    "model_uw_2.define(**hyperparameters_2)\n",
    "model_uw_2.model.load_weights(\"./models/lstm_model_comments_2/weights/02-0.320474.h5\")"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "source": [
    "prediction_uw_2 = model_uw_2.predict(X_test['normalize_comment'])\n",
    "prediction_uw_2"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input</th>\n",
       "      <th>output_proba</th>\n",
       "      <th>output_class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>bán hàng kỉ quấn đầy đủ chất lượn...</td>\n",
       "      <td>(0.00043627014, 0.99956375)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>đặt xl áo ghi xl nhưng bé tí tẹo ngang ...</td>\n",
       "      <td>(0.9958728, 0.004127209)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>kẹp rất rất đẹp săn được giá sale h...</td>\n",
       "      <td>(0.115075, 0.88492507)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>không nghĩ áo đẹp giao hàng nhanh không ...</td>\n",
       "      <td>(0.012311526, 0.9876885)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>hôm nay mới nhận được nhưng nhìn qua...</td>\n",
       "      <td>(0.8120184, 0.18798164)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2837</th>\n",
       "      <td>đặt dây buộc tóc nhưng chỉ nhận đưo...</td>\n",
       "      <td>(0.997141, 0.0028590155)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2838</th>\n",
       "      <td>chất vải đẹp có vẻ mát giao hàng nhanh...</td>\n",
       "      <td>(0.0035439562, 0.99645597)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2839</th>\n",
       "      <td>dây nhìn tạm không đẹp mắc sai dây pha...</td>\n",
       "      <td>(0.99527293, 0.004727039)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2840</th>\n",
       "      <td>quần đẹp chất lượng ok nhưng chữ cu...</td>\n",
       "      <td>(0.74641526, 0.2535847)</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2841</th>\n",
       "      <td>ok hơi chật tí</td>\n",
       "      <td>(0.1930916, 0.80690837)</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2842 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  input  \\\n",
       "0     bán hàng kỉ quấn đầy đủ chất lượn...   \n",
       "1     đặt xl áo ghi xl nhưng bé tí tẹo ngang ...   \n",
       "2     kẹp rất rất đẹp săn được giá sale h...   \n",
       "3     không nghĩ áo đẹp giao hàng nhanh không ...   \n",
       "4     hôm nay mới nhận được nhưng nhìn qua...   \n",
       "...                                                 ...   \n",
       "2837  đặt dây buộc tóc nhưng chỉ nhận đưo...   \n",
       "2838  chất vải đẹp có vẻ mát giao hàng nhanh...   \n",
       "2839  dây nhìn tạm không đẹp mắc sai dây pha...   \n",
       "2840  quần đẹp chất lượng ok nhưng chữ cu...   \n",
       "2841                                 ok hơi chật tí   \n",
       "\n",
       "                     output_proba  output_class  \n",
       "0     (0.00043627014, 0.99956375)             1  \n",
       "1        (0.9958728, 0.004127209)             0  \n",
       "2          (0.115075, 0.88492507)             1  \n",
       "3        (0.012311526, 0.9876885)             1  \n",
       "4         (0.8120184, 0.18798164)             0  \n",
       "...                           ...           ...  \n",
       "2837     (0.997141, 0.0028590155)             0  \n",
       "2838   (0.0035439562, 0.99645597)             1  \n",
       "2839    (0.99527293, 0.004727039)             0  \n",
       "2840      (0.74641526, 0.2535847)             0  \n",
       "2841      (0.1930916, 0.80690837)             1  \n",
       "\n",
       "[2842 rows x 3 columns]"
      ]
     },
     "metadata": {},
     "execution_count": 18
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "source": [
    "Model.confusionMatrix(y_test, prediction_uw_2['output_class'])"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "    Negative       0.86      0.89      0.88      1410\n",
      "    Positive       0.89      0.86      0.87      1432\n",
      "\n",
      "    accuracy                           0.87      2842\n",
      "   macro avg       0.88      0.87      0.87      2842\n",
      "weighted avg       0.88      0.87      0.87      2842\n",
      "\n"
     ]
    },
    {
     "output_type": "display_data",
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAEWCAYAAACZnQc8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAApCUlEQVR4nO3dd5wV1f3/8ddbUEAUEFCioGLsJfaCvXeN5adRsTc08WuPscREY5rGrtiwa1TsxhaV2DWxgAUVG0GNoCIW7Aq7fH5/zFm8wJa7u3d2L7Pv5+Mxj517Zu6cc1f83LNnznyOIgIzMyuGOdq7AWZmVjkO6mZmBeKgbmZWIA7qZmYF4qBuZlYgDupmZgXioG6tJqmbpHskfSHp1lZcZ09JD1Wybe1F0vqS3mzvdljHI89T7zgkDQaOAZYBvgJeAv4cEU+18rp7A4cD60RETWvbWe0kBbBkRIxt77aYzcw99Q5C0jHAecBfgH7AIsDFwA4VuPyiwFsdIaCXQ1Ln9m6DdWAR4a3gG9AT+BrYtZFzupAF/Q/Sdh7QJR3bCBgPHAt8DHwI7J+O/QGYAkxNdRwInAr8veTaA4EAOqfX+wHjyP5aeAfYs6T8qZL3rQM8D3yRfq5Tcuwx4I/A0+k6DwF9G/hsde3/TUn7dwS2Ad4CPgNOKjl/TeA/wOR07lBgrnTsifRZvkmfd7eS6x8PfARcX1eW3rN4qmPV9HohYBKwUXv/2/BWvM099Y5hbaArcGcj5/wWGASsDKxEFthOLjn+E7Ivh/5kgfsiSfNFxClkvf+bI2KeiLiysYZI6g5cAGwdEfOSBe6X6jmvN3BfOrcPcA5wn6Q+JacNBvYHFgDmAn7dSNU/Ifsd9Ad+D1wO7AWsBqwP/E7SYuncWuBooC/Z725T4FcAEbFBOmel9HlvLrl+b7K/WoaUVhwR/yUL+H+XNDdwNXBtRDzWSHvNWsRBvWPoA3wSjQ+P7AmcFhEfR8Qksh743iXHp6bjUyPifrJe6tItbM80YAVJ3SLiw4h4rZ5ztgXejojrI6ImIm4C3gC2Lznn6oh4KyK+A24h+0JqyFSy+wdTgeFkAfv8iPgq1T+G7MuMiBgVEc+ket8FLgM2LOMznRIRP6T2zCAiLgfGAs8CC5J9iZpVnIN6x/Ap0LeJsd6FgPdKXr+XyqZfY6YvhW+BeZrbkIj4hmzI4lDgQ0n3SVqmjPbUtal/yeuPmtGeTyOiNu3XBd2JJce/q3u/pKUk3SvpI0lfkv0l0reRawNMiojvmzjncmAF4MKI+KGJc81axEG9Y/gP8APZOHJDPiAbOqizSCpriW+AuUte/6T0YEQ8GBGbk/VY3yALdk21p65NE1rYpua4hKxdS0ZED+AkQE28p9FpZJLmIbtPcSVwahpeMqs4B/UOICK+IBtHvkjSjpLmljSnpK0l/S2ddhNwsqT5JfVN5/+9hVW+BGwgaRFJPYET6w5I6idphzS2/gPZMM60eq5xP7CUpMGSOkvaDVgOuLeFbWqOeYEvga/TXxG/nOn4ROCnzbzm+cDIiDiI7F7Bpa1upVk9HNQ7iIg4m2yO+slkMy/eB/4PuCud8idgJDAaeAV4IZW1pK4RwM3pWqOYMRDPkdrxAdmMkA2ZNWgSEZ8C25HNuPmUbObKdhHxSUva1Ey/JrsJ+xXZXxE3z3T8VOBaSZMl/aKpi0naAdiKHz/nMcCqkvasWIvNEj98ZGZWIO6pm5kViIO6mVmBOKibmRWIg7qZWYFUbeKhqZ+M8x1cm0W3hdZv7yZYFaqZMqGp5wia1JyYM2ffn7a6vrxUbVA3M2tT02qbPmc24KBuZgYQ9T0DN/txUDczA5jmoG5mVhjhnrqZWYHUFmPhLgd1MzPwjVIzs0Lx8IuZWYEU5Eapnyg1MyO7UVru1hRJV0n6WNKrJWVnSnpD0mhJd0rqVXLsREljJb0pacuS8q1S2VhJJ5TzORzUzcwg66mXuzXtGrIc+qVGACtExIrAW6TFYyQtB+wOLJ/ec7GkTpI6ARcBW5MtELNHOrdRHn4xMwOonVqxS0XEE5IGzlT2UMnLZ4Bd0v4OwPC0bu07ksYCa6ZjYyNiHICk4encMY3V7Z66mRlkN0rL3CQNkTSyZBvSzNoOAP6Z9vuTrURWZ3wqa6i8Ue6pm5lBs26URsQwYFhLqpH0W6AGuKEl72+Kg7qZGbTJlEZJ+5Gtvbtp/LiW6ARg4ZLTBqQyGilvkIdfzMyg0jdKZyFpK7IF1H8eEd+WHLob2F1SF0mLAUsCzwHPA0tKWkzSXGQ3U+9uqh731M3MgJhWuRulkm4CNgL6ShoPnEI226ULMEISwDMRcWhEvCbpFrIboDXAYRFRm67zf8CDQCfgqoh4rcm6f/wLoLp4kQyrjxfJsPpUYpGM71+4u+yY03XVn3uRDDOzquY0AWZmBeKEXmZmBeKeuplZgRQkoZeDupkZeJEMM7NCcU/dzKw40tTw2Z6DupkZuKduZlYonv1iZlYg7qmbmRWIZ7+YmRWIh1/MzArEwy9mZgXioG5mViAefjEzKxDfKDUzK5CCDL/kvkappEUlbZb2u0maN+86zcyaLaaVv1WxXIO6pIOB24DLUtEA4K486zQza5GcF55uK3kPvxwGrAk8CxARb0taIOc6zcyar8qDdbnyDuo/RMSUtHI2kjoDXlDazKpPFCM05R3UH5d0EtBN0ubAr4B7cq7TzKz5aoox+yXvG6UnAJOAV4BDgPuBk3Ou08ys+QpyozTvnvqOwHURcXnO9ZiZtU5BxtTz7qlvD7wl6XpJ26UxdTOz6hNR/lbFcg3qEbE/sARwK7AH8F9JV+RZp5lZi3hKY3kiYqqkf5LNeulGNiRzUN71mpk1S5UH63LlGtQlbQ3sBmwEPAZcAfwizzrNzFoiar3wdDn2AW4GDomIH3Kuy8ys5dxTb1pE7JHn9c3MKqbKpyqWK5cbpZKeSj+/kvRlyfaVpC/zqNPMrFWmRflbEyRdJeljSa+WlPWWNELS2+nnfKlcki6QNFbSaEmrlrxn33T+25L2Ledj5BLUI2K99HPeiOhRss0bET3yqNPMrFUqO/vlGmCrmcpOAB6OiCWBh9NrgK2BJdM2BLgEsi8B4BRgLbIcWqfUfRE0Ju8sjdeXU2Zm1u5qa8vfmhARTwCfzVS8A3Bt2r+WbCZgXfl1kXkG6CVpQWBLYEREfBYRnwMjmPWLYhZ5P3y0fOmL9PDRajnXOVs4+S/nsMG2u7PjXodOLztr6BVsv8fB7LTPLznixNP48quvpx97c+w77DnkaHbY8xB22vuX/PDDFADOv+waNt1pb9bYbKc2/wyWr8uHnc0H41/mpRcfnuXY0UcdQs2UCfTpk3XcevXqyW23XsELo0bwn6fvZfnll27r5s7+mtFTlzRE0siSbUgZNfSLiA/T/kdAv7TfH3i/5Lzxqayh8kblNaZ+oqSvgBVLx9OBicA/8qhzdrPjNptz6Tl/mqFs7TVW4c7rL+XO6y5h4ML9ueL6mwGoqanlhNP+xu+OO5x/3HAZVw89g86dOwGw0bprMfzy89u8/Za/6667hW2323OW8gEDFmLzzTbgvffGTy878fjDefnl11h1tc3Z74AjOffs09qyqcXQjDH1iBgWEauXbMOaU1VEBDllrM1rTP2vETEvcOZM4+l9IuLEPOqc3ay+8s/o2WPGRaDWXWu16cF6xeWXYeLHnwDw7+dGsdTii7HMkj8FoFfPHnTqlJ230grLMn/f3m3YcmsrTz71LJ99PnmW8rPPOpUTTvozUfK4+rLLLsWjjz4NwJtv/pdFFx3AAgv0baumFkP+Cb0mpmEV0s+PU/kEYOGS8waksobKG5V3moATJc0naU1JG9RtedZZFHfe9xDrrb0GAO+9PwFJDDn6t+y6//9x1Q23tnPrrL1sv/0WTJjwIaNHj5mhfPQrY9hpx20AWGP1lVl00QEM6L9gezRx9lXB2S8NuBuom8GyLz+OWtwN7JNmwQwCvkjDNA8CW6QYOh+wRSprVN5PlB4EHEn2DfMSMAj4D7BJA+cPIbv7y8Vn/4mD9umY09wvu/YmOnXqxHZbbAxATW0tL45+jeFXnE/Xrl046IgTWW7pJRi0+irt3FJrS926deXE4w9nq20Gz3LsjL8N5dxzTmPk8w/x6qtv8OJLr1JbkIdp2kpU8Pcl6SayJ+n7ShpPNovldOAWSQcC7/Hj0/X3A9sAY4Fvgf0BIuIzSX8Enk/nnRYRM998nUXeT5QeCawBPBMRG0taBvhLQyencalhAFM/GVfdqdByctd9I3ji6ee44oK/UrdiVL8F+rLaSiswX6+eAKy/9hqMefO/DuodzOKLD2TgwEV4YeQIAAYMWJDnn32QtdfdlokTJ3HQwcdMP3fsW88wbtx77dXU2VMF0wQ08uDlpvWcG2RLf9Z3nauAq5pTd96zX76PiO8BJHWJiDcA35ZvwFPPjOSqG2/lwjNOoVvXrtPL111zNd4e9y7fff89NTW1jHzpFRZfbJF2bKm1h1dffYOFBqzEEksNYomlBjF+/IessdaWTJw4iZ49ezDnnHMCcOABg3nyqWf5qmT2lJUh/+GXNpF3T328pF7AXcAISZ+T/dnR4R13yuk8/+JoJk/+kk133ItfHbg3V1x/M1OmTuXgo34LZDdLT/nN4fTsMS/77L4zux94JJJYf+012HCdNQE4+6IruX/Eo3z//Q9suuNe7Lz9Vhx24F7t+dGsQv5+/UVsuMHa9O3bm3fHjeQPp53F1dcMr/fcZZdZkquuOo+IYMyYNzl4yK/buLUFUJDhKkUbJXyXtCHQE3ggIqY0dX5HHX6xxnVbaP32boJVoZopE9Taa3zz+93LjjndTxve6vrykveN0tK5dq+knw7WZlZ9CpLQK+/hlxfI5ll+DgjoBXwkaSJwcESMyrl+M7PyVPlYebnyvlE6AtgmIvpGRB+yxDX3Ar8CLs65bjOzskVNbdlbNcs7qA+KiOmT5SPiIWDtlLSmS851m5mVz7NfyvKhpOOBulv2u5E9KtsJKMYAlpkVQ0HG1PPuqQ8me5r0LuBOsvH1wUAnvFapmVUT99SbFhGfAIdL6h4R38x0eGyedZuZNUdUebAuV96LZKwjaQzwenq9kiTfIDWz6lNTW/5WxfIefjmXbPWOTwEi4mXAWRrNrPp4+KU8EfF+XWKqpLq/5sysY6ryYF2uvIP6+5LWAULSnGRZG1/PuU4zs2Zrq5Qpecs7qB8KnE+2rt4E4CEaSDFpZtau3FNvWpr9Musii2Zm1cZBvWGSft/I4YiIP+ZRr5lZS0VNMR4+yqunPvOcdIDuwIFAH8BB3cyqSzFiej5BPSLOrtuXNC/ZDdL9ydIFnN3Q+8zM2ktRHj7KbUw95VI/hmxM/Vpg1Yj4PK/6zMxaxUG9YZLOBHYmW0T6ZxHhxRLNrLoVZPilySdKJe2ahlCQdLKkOySt2sTbjgUWAk4GPpD0Zdq+kvRl65ttZlZZMS3K3qpZOT3130XErZLWAzYDzgQuAdZq6A0RkXf6ATOzioqa6g7W5Son+NY91r8tMCwi7gPmyq9JZmbtYFoztipWTk99gqTLgM2BMyR1If9EYGZmbaoga2SUFZx/ATwIbBkRk4HewHF5NsrMrM0VpKfeZFCPiG+Bj4H1UlEN8HaejTIza2sxrfytmjU5/CLpFGB1YGngamBO4O/Auvk2zcys7URNe7egMsoZU98JWAV4ASAiPqib4mhmVhTV3gMvVzlBfUpEhKQAkNQ95zaZmbW5ogT1cm6U3pJmv/SSdDDwL+DyfJtlZtbGQuVvTZB0tKTXJL0q6SZJXSUtJulZSWMl3SxprnRul/R6bDo+sDUfo5wbpWcBtwG3k42r/z4iLmxNpWZm1aZSN0ol9QeOAFaPiBWATsDuwBnAuRGxBPA5WdZa0s/PU/m56bwWKyv3S0SMAEa0piIzs2oW05rugTdDZ6CbpKnA3MCHwCbA4HT8WuBUsqfzd0j7kHWgh0pStHB9vXJmv3wF1F18LrLZL99ERI+WVGhmVo2m1VYmqEfEBElnAf8DviNbxnMUMDli+hyb8WTLfJJ+vp/eWyPpC7J1Jz5pSf1NBvWImD7TRZLIvlUGtaQyM7Nq1ZwbpZKGAENKioZFxLB0bD6yOLkYMBm4FdiqUu1sSrNS76Y/B+5Kc9dPyKdJZmZtrznDLymAD2vg8GbAOxExCUDSHWTP9fSS1Dn11gcAE9L5E4CFgfGSOgM9gU9b9CEob/hl55KXc5A9iPR9Sys0M6tGLRvBrtf/gEGS5iYbftkUGAk8CuxCtgLcvsA/0vl3p9f/Sccfael4OpTXU9++ZL8GeJfsTwszs8Ko1I3SiHhW0m1kD2zWAC+S9ervA4ZL+lMquzK95Urgekljgc/IZsq0mFrxhZCrqZ+Mq86GWbvqttD67d0Eq0I1Uya0OiK/s9LmZcecxV4eUdGpMpXUYE9d0oX8OOtlFhFxRC4tMjNrBxWe0thuGht+GdlmrTAza2dRxpOis4MGg3pEXNuWDTEza09Fyf1SzuyX+YHjgeWArnXlEbFJju0yM2tT0wrSUy8nodcNwOtkE+n/QDb75fkc22Rm1uYiVPZWzcoJ6n0i4kpgakQ8HhEHkOUwMDMrjGm1KnurZuXMU5+afn4oaVvgA7J1Ss3MCqMjzH6p8ydJPYFjgQuBHsDRubbKzKyNFWVMvZyg/mxEfAF8AWycc3vMzNpFtY+Vl6ucMfWnJT0k6cCUfczMrHAiyt+qWTkrHy0FnAwsD4ySdK+kvXJvmZlZG5oWKnurZuX01ImI5yLiGGBNsoQzfjDJzApl2jSVvVWzch4+6gHsRJY5bHHgTrLgbmZWGNXeAy9XOTdKXwbuAk6LiP/k25wfLbT41m1Vlc1Gvn3rH02fZNYCRblRWk5Q/2lrErabmc0OOkxP3QHdzDqCogS6Zq1RamZWVLXTypo3UvUc1M3MgIJk3vXKR2ZmAEHxx9S98pGZdRjTCjKo7pWPzMyAaR2gpw545SMz6xiKMvzilY/MzIBaVPZWzbzykZkZ2eyXcrdq5pWPzMyo/mBdLq98ZGZGccbUy0kTcG/a9cpHZlZYVZ5Rt2zlzH65mnoeQkpj62ZmhdBhpjQC95bsdyXLrf5BPs0xM2sfte3dgAopZ/jl9tLXkm4CnsqtRWZm7WCaitFTb0lasiWBBSrdEDOz9hTN2JoiqZek2yS9Iel1SWtL6i1phKS308/50rmSdIGksZJGS1q1NZ+jyaAu6StJX9ZtwD1kT5iamRVGheepnw88EBHLACuRPcB5AvBwRCwJPJxeA2xN1lleEhgCXNKaz1HO8Mu8ranAzGx2UKnZL2kK+AbAfgARMQWYImkHYKN02rXAY2Qd5B2A69KCRM+kXv6CEfFhS+ovp6f+cDllZmazswqmCVgMmARcLelFSVdI6g70KwnUHwH90n5/4P2S949PZS3SYFCX1FVSb6CvpPnSeFBvSQNbU6GZWTWapvI3SUMkjSzZhpRcqjOwKnBJRKwCfMOPQy3A9GVCc0n229jwyyHAUcBCwCiY/vX0JTA0j8aYmbWX5qQJiIhhwLAGDo8HxkfEs+n1bWRBfWLdsIqkBYGP0/EJwMIl7x+QylqkwZ56RJwfEYsBv46In0bEYmlbKSIc1M2sUCo1+yUiPgLel7R0KtoUGAPcDeybyvYF/pH27wb2SbNgBgFftHQ8Hcp7+GiapF4RMRkgTcPZIyIubmmlZmbVpsJpAg4HbpA0FzAO2J+sE32LpAOB94BfpHPvB7YBxgLfpnNbrJygfnBEXFT3IiI+l3Qw4KBuZoVRySyNEfESsHo9hzat59wADqtU3eUE9U6SlCpGUidgrko1wMysGtQW44HSsoL6A8DNki5Lrw9JZWZmhdGR8qkfT/aU0y/T6xHA5bm1yMysHRQlqDf58FFETIuISyNil4jYhewu7oX5N83MrO1UMvdLeyqnp46kVYA9yO7WvgPckWejzMzaWuEXyZC0FFkg3wP4BLgZUER49SMzK5yiDL801lN/A3gS2C4ixgJI8tqkZlZIRVkko7Ex9Z2BD4FHJV0uaVMoyHpPZmYzaU7ul2rWWJqAuyJid2AZ4FGyPDALSLpE0hZt1D4zszZR4Xzq7aac2S/fRMSNEbE9WaKZF/EiGWZWMB1q9kudiPicLDNZQ9nJzMxmS9OqPlyXp1lB3cysqIpyo9RB3cyM6h8rL1eTY+qtIWkpSQ9LejW9XlHSyXnWaWbWEoWf/VIhlwMnAlMBImI0sHvOdZqZNds0ouytmuU9/DJ3RDwnzfDVVpNznWZmzVbdobp8eQf1TyQtTvp9SdqF7IEmM7OqUpQx9byD+mFk0x+XkTSBLBnYnjnXaWbWbLUF6avnHdTfi4jNJHUH5oiIr3Kuz8ysRYrSU8/7Ruk7koYBg4Cvc67LzKzFinKjNO+gvgzwL7JhmHckDZW0Xs51mpk1W1HSBOQa1CPi24i4JSJ2BlYBegCP51mnmVlLdJiEXq0laUNJFwOjgK5kqyeZmVWVWqLsrZrleqNU0rtkWR1vAY6LiG/yrM/MrKWqfay8XHnPflkxIr7MuY7Z2kL9f8JFl/6N+RfoQ0Rw/TW3MOzS6+g1X08uv/pcFlmkP//73wQO2u8ovpj8JYcdcSC77Lo9AJ06d2KppRdnmcXXZvLnX7TzJ7HW+t3Zw3ji2Rfp3asHdw47A4CzL7+Rx555gTnn7MzCC/bjj8cOocc83fn3qFc476rhTK2pYc7OnTn24MGstfLyAOx/3J/45LPJdJlrTgAu++sJ9OnVs90+1+yiGCE9W3O08heVfhMRf5N0IfX8riLiiKauMX/PpYvyO25Uv37z0+8n8zP65TF0n6c7Dz9+O/sMPozd99yZyZ9P5oJzL+eIow+mZ6+e/PGUs2Z47xZbbcyhh+3Hztvv206tb3sTXr6hvZuQm5GvvM7cXbvy2zMvnR7U/z1qNGuuvDydO3XinCtuAuCYg/bg9bHv0me+nizQZz7efvd9Dj3pDB6+cSiQBfVfHzyY5Zf6abt9lrY218DVW52R5ZCBu5Ydcy5799aqzQCT15j66+nnSLKx9Jk3SyZOnMTol8cA8M3X3/DWm+NYcKF+bL3Nptx8410A3HzjXWyz7WazvHfnXbbljtvubcvmWo5W/9my9Jx3nhnK1lltRTp36gTASssuwcRPPgNg2SUGskCf+QBYYtEBfP/DFKZMmdq2DS6YotwozWX4JSLuSbvfRsStpcck7ZpHnUWw8CL9+dmKyzJq5MvMP38fJk6cBGSBf/75+8xwbrduXdlks/U54bg/tkdTrR3c+eDjbLnhoFnKRzz1HMsuMZC50nALwMlnX0anOeZgs/XW5JDBOzJT/iWrRxRkACbv2S8nllkGgKQhkkZKGvn9lMn5taoKde8+N1dffwEnn/gXvv5q1vvJM/+D23LrjXnumRc8lt5BDLvxLjp16sR2m6w7Q/nYd8dz7pXDOeXIA6eXnX78r7jzsjO49uzf88Krb3DPv55q6+bOljz7pRGStga2AfpLuqDkUA8aydIYEdOXyusoY+oAnTt35urrL+C2W+7hvntGADBp0qf06zc/EydOol+/+flk0mczvGfHnbfljtvua4/mWhu766HHefy5F7ni9JNm6HF/NOlTjjrtXP5y3KEsvFC/6eX9+vYGoPvc3dhm43V45c3/8vPN12/zds9uqn1YpVx59dQ/IBtP/54Zx9LvBrbMqc7Z1nlD/8xbb47j0ouumV72wD8fYbfBOwKw2+Ad+ef9D08/Nm+PeVhnvTV4oKTMiump51/m6lvv5cJTj6Vb1y7Ty7/8+hsO+91ZHHXA7qyy/NLTy2tqa/n8iyzF0tSaGp549kWWHDigzds9O5oWUfZWzXKZ/TL94lLniGhR/vSO0lNfa9Bq3Pvgjbz26pvEtKyv8OfTzmHUyNFcce15DBiwIO+//wEH7XfU9KGW3QfvxCabrc+QA45pz6a3iyLPfvnNX4fy/OjXmfzFV/SerweH7b0LVwy/mylTp9KrR3YDdcVlluD3Rx7IZTfeyZXD72GR/j/20C/76wl069qF/Y79IzW1tUyrncagVVfguCF70alT7s8ZtqtKzH7Za9Gdy445f3/vjibrk9SJrHM7ISK2k7QYMBzoQ9bJ3TsipkjqAlwHrAZ8CuwWEe+24CNk9eY0pfGWiPiFpFeYcUqjgIiIFZu6RkcJ6tY8RQ7q1nKVCOqDF92p7Jhz43t3lhPUjwFWB3qkoH4LcEdEDJd0KfByRFwi6Vdkz/QcKml3YKeI2K2lnyOvh4+OTD+3y+n6ZmYVVcnZL5IGANsCfwaOUXYzZBNgcDrlWuBU4BJgh7QPcBswVJKihT3uXP4mi4i61Y0+Ad6PiPeALsBKZOPtZmZVpYYoeyudqZe2ITNd7jzgN/x4/7UPMLlkOHo80D/t9wfeB0jHv0jnt0jeaQKeANaXNB/wEPA8sBte/cjMqkxzeuqlM/VmJmk74OOIGCVpo4o0rhnyDuqKiG8lHQhcnFIHvJRznWZmzVbBKY3rAj+XtA1ZZtoewPlAr5LJIwOACen8CcDCwHhJnYGeZDdMWyTvW+KStDZZz7xuUnWnnOs0M2u2iCh7a+I6J0bEgIgYCOwOPBIRewKPAruk0/YF/pH2706vSccfael4OuQf1I8ie4L0zoh4TdJPyT6YmVlVaYPl7I4nu2k6lmzM/MpUfiXQJ5UfA5zQms+R6/BLRDwOPC5pHknzRMQ4oMkMjWZmbS2Px/8j4jHgsbQ/DliznnO+ByqWEyvXnrqkn0l6EXgNGCNplKTl86zTzKwlirLwdN43Si8DjomIRwHSneDLgXVyrtfMrFnyfLq+LeUd1LvXBXTI/hSR1D3nOs3Mmq0oCb3yDurjJP0OuD693gsYl3OdZmbN5nzq5TkAmB+4A7gd6JvKzMyqisfUGyGpK3AosATwCnBsRHitLTOrWrVRjAGYvIZfrgWmAk8CWwPLks1ZNzOrSkUZfskrqC8XET8DkHQl8FxO9ZiZVUS1L35RrryC+vShloio8aK3ZlbtihHS8wvqK0n6Mu0L6JZe1y2S0SOnes3MWqTab4CWK5egHhFO2mVmsxUHdTOzAvHsFzOzAvHsFzOzAnHuFzOzAvGYuplZgbinbmZWILUFydPooG5mhp8oNTMrFM9+MTMrEPfUzcwKxD11M7MCcU/dzKxAnCbAzKxAPPxiZlYg4Z66mVlxOE2AmVmBOE2AmVmBuKduZlYgtdM8pm5mVhhFmf0yR3s3wMysGkRE2VtjJC0s6VFJYyS9JunIVN5b0ghJb6ef86VySbpA0lhJoyWt2prP4aBuZkY2pl7u1oQa4NiIWA4YBBwmaTngBODhiFgSeDi9BtgaWDJtQ4BLWvM5HNTNzKhcTz0iPoyIF9L+V8DrQH9gB+DadNq1wI5pfwfgusg8A/SStGBLP4fH1M3MaN6NUklDyHrVdYZFxLB6zhsIrAI8C/SLiA/ToY+Afmm/P/B+ydvGp7IPaQEHdTMzmjelMQXwWYJ4KUnzALcDR0XEl5JK3x+Scrkz66BuZkZlHz6SNCdZQL8hIu5IxRMlLRgRH6bhlY9T+QRg4ZK3D0hlLeIxdTMzstS75W6NUdYlvxJ4PSLOKTl0N7Bv2t8X+EdJ+T5pFswg4IuSYZpmc0/dzIyKzlNfF9gbeEXSS6nsJOB04BZJBwLvAb9Ix+4HtgHGAt8C+7emcgd1MzMqt0hGRDwFqIHDm9ZzfgCHVaRyHNTNzACY5tS7ZmbF4SyNZmYF4qBuZlYgxQjpoKJ8OxWZpCH1Pa1mHZv/XVh9PE999jCk6VOsA/K/C5uFg7qZWYE4qJuZFYiD+uzB46ZWH/+7sFn4RqmZWYG4p25mViAO6mZmBeKgXmGSQtLZJa9/LenUHOo5aabX/650HZYPSbWSXpL0qqRbJc3dzPcvJOm2tL+ypG1Kjv1c0gkNv9uKzkG98n4AdpbUN+d6ZgjqEbFOzvVZ5XwXEStHxArAFODQ5rw5Ij6IiF3Sy5XJ0rbWHbs7Ik6vWEtttuOgXnk1ZLMSjp75gKT5Jd0u6fm0rVtSPkLSa5KukPRe3ZeCpLskjUrHhqSy04Fuqbd3Qyr7Ov0cLmnbkjqvkbSLpE6Szkz1jpZ0SO6/CSvHk8ASknqn/9ajJT0jaUUASRum/84vSXpR0rySBqZe/lzAacBu6fhukvaTNFRSz/TvaI50ne6S3pc0p6TFJT2Q/l09KWmZdvz8VmnNWUHbW1mrjH8N9ADeBXoCvwZOTcduBNZL+4uQrYwCMBQ4Me1vRZaGom963Tv97Aa8CvSpq2fmetPPnYBr0/5cZAvadiN7+vDkVN4FGAks1t6/r464lfy36ky2+s0vgQuBU1L5JsBLaf8eYN20P096z0Dg1VS2HzC05NrTX6drb5z2dwOuSPsPA0um/bWAR9r7d+KtcpsTeuUgskVmrwOOAL4rObQZsFzJArQ90uK065EFYyLiAUmfl7znCEk7pf2FgSWBTxup/p/A+ZK6kH1BPBER30naAlhRUt2f7T3Ttd5p6ee0FutWsiLOk2RLnz0L/D+AiHhEUh9JPYCngXPSX2R3RMT40gWMm3AzWTB/FNgduDj9e1sHuLXkOl1a/5GsWjio5+c84AXg6pKyOYBBEfF96YkN/U8qaSOyL4K1I+JbSY8BXRurNCK+T+dtSfY/9PC6ywGHR8SDzfsYloPvImLl0oKG/g1ExOmS7iMbN39a0pbA9/WePKu7gb9I6g2sBjwCdAcmz1y/FYfH1HMSEZ8BtwAHlhQ/BBxe90LSymn3adJ6halHPV8q7wl8ngL6MsCgkmtNTSuW1+dmsnUO1wceSGUPAr+se4+kpSR1b9mnsxw8CewJ07/MP0l/8S0eEa9ExBnA88DM499fAfPWd8GI+Dq953zg3oiojYgvgXck7ZrqkqSV8vhA1j4c1PN1NlA6C+YIYPV0M2wMP856+AOwhaRXgV2Bj8j+Z30A6CzpdbJFa58pudYwYHTdjdKZPARsCPwrIqaksiuAMcALqZ7L8F9q1eRUYDVJo8n+W9etOn9Uuik6GphKNrxW6lGyIb2XJO1Wz3VvBvZKP+vsCRwo6WXgNWCHyn0Ma29OE1AF0vh3bUTUSFobuMR/HptZS7inVh0WAW5J08+mAAe3c3vMbDblnrqZWYF4TN3MrEAc1M3MCsRB3cysQBzUbQatzSA407WuqXuCNeW0Wa6RczeS1OykZJLerUTytEpdx6y9OajbzBrNICipRTOmIuKgiBjTyCkbkT2+bmat4KBujanLILhRyuZ3NzCmoYyP6enEoZLelPQvYIG6C0l6TNLqaX8rSS9IelnSw5IGkn15HJ3+SlhfDWe07CPpIaWMlmTpD2Yg6VBJZ5a83k/S0LQ/S9bLmd47MD2cVfd6ej78hrIbSto1/WXzsqQnWvtLN2sNz1O3eqUe+db8mGZgVWCFiHgnBcMvImKN9ODU05IeAlYBlgaWA/qRPcF61UzXnR+4HNggXat3RHwm6VKy7IVnpfNuBM6NiKckLUKW5mBZ4BTgqYg4TVmK4dI0DHVuB/4DHJde7wb8Oe0fkOrrBjwv6faIaCxBWqlhwKER8baktYCLyTIq/h7YMiImSOpV5rXMcuGgbjOrL4PgOsBzEVGX0bGhjI8bADdFRC3wgaRH6rn+ILLMke/A9Bw59Wkoo+UGwM7pvfdpxoyWpPJJksZJGgS8TZYv5el0uLlZLwFQ49kNnwaukXQLcEdT1zLLk4O6zayhDILflBZRT8ZHlSyrVgHNymhZj+FkSdLeAO6MiFB5WS9rmHFYsu74HDSQ3TAiDk09922BUZJWa0bv36yiPKZuLdFQxscnyFbh6SRpQWDjet77DLCBpMXSe3un8pmzDTaU0fIJYHAq25ofM1rO7E6yRFV78GP64cayXtaZCCyQxu67ANtBliOfBrIbKsuk+GxE/B6YRPYXgFm7cFC3lmgo4+OdZMMdY4DryMa1ZxARk8hWYbpDWZbAuuyB9wA71d0opfGMlhtIeo1sGOZ/9TUwIj4HXgcWjYjnUnFjWS/r3jeVbIm454ARZD39Og1lNzxT0ivpd/Fv4OX6f21m+XPuFzOzAnFP3cysQBzUzcwKxEHdzKxAHNTNzArEQd3MrEAc1M3MCsRB3cysQP4/v00hp0h7HgMAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     }
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "> **Nhận xét**:\n",
    "> * Độ chính xác lúc này 87%.\n",
    "> * Nếu ta đào tạo quá 10 epoch thì càng ngàng bị overfitting, nhưng nếu ta thay đổi kiến trúc của model thì độ chính xác luôn loanh quanh 86, 87%. Như vậy đây có phải là độ chính xác tối đa mà model LSTM có thể đạt được hoặc là do chính data của ta bị variance error do có sự chồng chéo data giữa nagative và postive ở các comment 3 sao."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Bây giờ ta sẽ lưu lại các model này thành file `*.h5` và các vectorizer tương ứng thành file `*.pickle`."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "source": [
    "model_1.save(\"./models/lstm_model_comments_1/lstm_model_comments_1.h5\", \"./models/lstm_model_comments_1/lstm_tokenizer_comments_1.pickle\")\n",
    "model_uw_1.save(\"./models/lstm_model_comments_1/lstm_model_comments_uw_1.h5\", \"./models/lstm_model_comments_1/lstm_tokenizer_comments_uw_1.pickle\")"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "<keras_preprocessing.text.Tokenizer object at 0x7f4d3aa8a460> has been saved at ./models/lstm_model_comments_1/lstm_tokenizer_comments_1.pickle.\n",
      "📢 Model has been saved at ./models/lstm_model_comments_1/lstm_model_comments_1.h5 - Tokenizer has been saved at ./models/lstm_model_comments_1/lstm_tokenizer_comments_1.pickle.\n",
      "<keras_preprocessing.text.Tokenizer object at 0x7f4d3ae7a430> has been saved at ./models/lstm_model_comments_1/lstm_tokenizer_comments_uw_1.pickle.\n",
      "📢 Model has been saved at ./models/lstm_model_comments_1/lstm_model_comments_uw_1.h5 - Tokenizer has been saved at ./models/lstm_model_comments_1/lstm_tokenizer_comments_uw_1.pickle.\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "source": [
    "model_2.save(\"./models/lstm_model_comments_2/lstm_model_comments_2.h5\", \"./models/lstm_model_comments_2/lstm_tokenizer_comments_2.pickle\")\n",
    "model_uw_2.save(\"./models/lstm_model_comments_2/lstm_model_comments_uw_2.h5\", \"./models/lstm_model_comments_2/lstm_tokenizer_comments_uw_2.pickle\")"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "<keras_preprocessing.text.Tokenizer object at 0x7f4e093d01f0> has been saved at ./models/lstm_model_comments_2/lstm_tokenizer_comments_2.pickle.\n",
      "📢 Model has been saved at ./models/lstm_model_comments_2/lstm_model_comments_2.h5 - Tokenizer has been saved at ./models/lstm_model_comments_2/lstm_tokenizer_comments_2.pickle.\n",
      "<keras_preprocessing.text.Tokenizer object at 0x7f4e09444cd0> has been saved at ./models/lstm_model_comments_2/lstm_tokenizer_comments_uw_2.pickle.\n",
      "📢 Model has been saved at ./models/lstm_model_comments_2/lstm_model_comments_uw_2.h5 - Tokenizer has been saved at ./models/lstm_model_comments_2/lstm_tokenizer_comments_uw_2.pickle.\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Bây giờ chúng ta sẽ tiến hành Hyper Parameters bằng phương pháp HyperBand được đề xuất vào năm 2016. Đây là bài báo đó [https://arxiv.org/pdf/1603.06560.pdf](https://arxiv.org/pdf/1603.06560.pdf). Ý tưởng của phương pháp này là HyperBand sẽ random ra một vài hyper-params set ban đầu. Tiếp theo nó sẽ tiến hành đào tạo các model này và thực hiến đánh giá dựa trên loss value của train hoặc test do ta quy định. Tiếp theo nó sẽ tiến hành chọn lọc dựa trên các công thức để chọn ra các hyperparam set có loss thấp nhất và loại bỏ đi các hyperparams set cho hiệu suất kém. Với từng hyperparams set được chọn lọc này nó sẽ đào tạo lại và tiếp tục loại bỏ và đào tạo lại cho đến khi còn lại model tốt nhất.\n",
    "\n",
    "Bài viết gốc của tác giả đọc sẽ khó hiểu và nặng về toán học, nên ta có thể đọc tại đây để dễ hiểu hơn [https://medium.com/data-from-the-trenches/a-slightly-better-budget-allocation-for-hyperband-bbd45af14481](https://medium.com/data-from-the-trenches/a-slightly-better-budget-allocation-for-hyperband-bbd45af14481).\n",
    "\n",
    "Bây giờ ta sẽ tạo một hàm `get_params()` là một Random-Search ra một hyperparam-set. Tiếp theo hàm `try_params()` sẽ nhận vào hyperparam set phát sinh từ `get_params()` và tiến hành training. Hai hàm này sẽ được bỏ vào một object `Hyperband` như là các tham số, là object này sẽ thực hiện tính toán và trả ra hyperparams set tốt nhất."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "source": [
    "def get_params():\n",
    "    pbatch_size = np.random.choice(np.arange(5, 100, 10))\n",
    "    pnum_words = np.random.choice(np.arange(2000, 3000, 10))\n",
    "    optimizers = np.random.choice(['rmsprop', 'adam', 'adadelta'])\n",
    "    pdropout = np.random.choice(np.linspace(0.1, 0.5, 10))\n",
    "    pno_units = np.random.choice(np.arange(10, 50, 10))\n",
    "    pseq_length = np.random.choice(np.arange(50, 200, 10))\n",
    "    \n",
    "    return {\n",
    "        'pX': X_train['normalize_comment'], \n",
    "        'py': y_train, \n",
    "        'pnum_words': pnum_words, \n",
    "        'pseq_length': pseq_length,\n",
    "        'pno_units': pno_units, \n",
    "        'pdropout': pdropout, \n",
    "        'poptimizers': optimizers,\n",
    "        'pbatch_size': pbatch_size, \n",
    "        'pepochs': 10, \n",
    "        'psave_weights': './models/lstm_model_comments_3.h5'\n",
    "    }"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "source": [
    "def try_params(**kwargs):\n",
    "    model = DeepModel.SentimentLSTM()\n",
    "    model.define(**kwargs)\n",
    "    model.fit(**kwargs)\n",
    "\n",
    "    return {\"loss\": model.evaluation()}"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "source": [
    "hb = DeepModel.Hyperband(get_params, try_params)\n",
    "results = hb.run()\n",
    "\n",
    "print(results)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "\n",
      "*** 81 configurations x 1.0 iterations each\n",
      "\n",
      "1 | Fri Oct 29 19:48:53 2021 | lowest loss so far: inf (run -1)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 180).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 180).\n",
      "137/137 [==============================] - ETA: 0s - loss: 0.5406 - accuracy: 0.7995WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 180).\n",
      "137/137 [==============================] - 17s 108ms/step - loss: 0.5406 - accuracy: 0.7995 - val_loss: 0.4038 - val_accuracy: 0.8461\n",
      "Epoch 2/10\n",
      "137/137 [==============================] - 15s 106ms/step - loss: 0.3273 - accuracy: 0.8843 - val_loss: 0.3485 - val_accuracy: 0.8637\n",
      "Epoch 3/10\n",
      "137/137 [==============================] - 15s 113ms/step - loss: 0.2686 - accuracy: 0.9052 - val_loss: 0.3341 - val_accuracy: 0.8619\n",
      "Epoch 4/10\n",
      "137/137 [==============================] - 16s 113ms/step - loss: 0.2398 - accuracy: 0.9155 - val_loss: 0.3449 - val_accuracy: 0.8628\n",
      "Epoch 5/10\n",
      "137/137 [==============================] - 16s 115ms/step - loss: 0.2176 - accuracy: 0.9246 - val_loss: 0.3456 - val_accuracy: 0.8637\n",
      "Epoch 6/10\n",
      "137/137 [==============================] - 16s 113ms/step - loss: 0.2084 - accuracy: 0.9256 - val_loss: 0.3573 - val_accuracy: 0.8619\n",
      "Epoch 7/10\n",
      "137/137 [==============================] - 16s 115ms/step - loss: 0.1922 - accuracy: 0.9330 - val_loss: 0.3598 - val_accuracy: 0.8672\n",
      "Epoch 8/10\n",
      "137/137 [==============================] - 15s 107ms/step - loss: 0.1805 - accuracy: 0.9375 - val_loss: 0.3755 - val_accuracy: 0.8584\n",
      "Epoch 9/10\n",
      "137/137 [==============================] - 15s 111ms/step - loss: 0.1726 - accuracy: 0.9412 - val_loss: 0.3898 - val_accuracy: 0.8584\n",
      "Epoch 10/10\n",
      "137/137 [==============================] - 15s 107ms/step - loss: 0.1657 - accuracy: 0.9434 - val_loss: 0.4033 - val_accuracy: 0.8628\n",
      "36/36 [==============================] - 0s 12ms/step - loss: 0.1473 - accuracy: 0.9533\n",
      "\n",
      "155 seconds.\n",
      "\n",
      "2 | Fri Oct 29 19:51:28 2021 | lowest loss so far: 0.1473 (run 1)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 180).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 180).\n",
      "2046/2046 [==============================] - ETA: 0s - loss: 0.3536 - accuracy: 0.8550WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 180).\n",
      "2046/2046 [==============================] - 183s 89ms/step - loss: 0.3536 - accuracy: 0.8550 - val_loss: 0.3345 - val_accuracy: 0.8566\n",
      "Epoch 2/10\n",
      "2046/2046 [==============================] - 179s 88ms/step - loss: 0.2485 - accuracy: 0.9033 - val_loss: 0.3138 - val_accuracy: 0.8690\n",
      "Epoch 3/10\n",
      "2046/2046 [==============================] - 179s 87ms/step - loss: 0.2028 - accuracy: 0.9264 - val_loss: 0.3310 - val_accuracy: 0.8707\n",
      "Epoch 4/10\n",
      "2046/2046 [==============================] - 178s 87ms/step - loss: 0.1740 - accuracy: 0.9366 - val_loss: 0.3462 - val_accuracy: 0.8769\n",
      "Epoch 5/10\n",
      "2046/2046 [==============================] - 177s 87ms/step - loss: 0.1487 - accuracy: 0.9465 - val_loss: 0.3625 - val_accuracy: 0.8777\n",
      "Epoch 6/10\n",
      "2046/2046 [==============================] - 178s 87ms/step - loss: 0.1285 - accuracy: 0.9540 - val_loss: 0.3885 - val_accuracy: 0.8698\n",
      "Epoch 7/10\n",
      "2046/2046 [==============================] - 178s 87ms/step - loss: 0.1129 - accuracy: 0.9612 - val_loss: 0.4322 - val_accuracy: 0.8698\n",
      "Epoch 8/10\n",
      "2046/2046 [==============================] - 178s 87ms/step - loss: 0.1042 - accuracy: 0.9655 - val_loss: 0.4671 - val_accuracy: 0.8654\n",
      "Epoch 9/10\n",
      "2046/2046 [==============================] - 178s 87ms/step - loss: 0.0922 - accuracy: 0.9700 - val_loss: 0.4986 - val_accuracy: 0.8514\n",
      "Epoch 10/10\n",
      "2046/2046 [==============================] - 178s 87ms/step - loss: 0.0834 - accuracy: 0.9735 - val_loss: 0.4899 - val_accuracy: 0.8637\n",
      "36/36 [==============================] - 0s 12ms/step - loss: 0.1067 - accuracy: 0.9701\n",
      "\n",
      "1787 seconds.\n",
      "\n",
      "3 | Fri Oct 29 20:21:15 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 50).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 50).\n",
      "137/137 [==============================] - ETA: 0s - loss: 0.4957 - accuracy: 0.7936WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 50).\n",
      "137/137 [==============================] - 7s 36ms/step - loss: 0.4957 - accuracy: 0.7936 - val_loss: 0.3678 - val_accuracy: 0.8514\n",
      "Epoch 2/10\n",
      "137/137 [==============================] - 4s 29ms/step - loss: 0.3025 - accuracy: 0.8840 - val_loss: 0.3526 - val_accuracy: 0.8531\n",
      "Epoch 3/10\n",
      "137/137 [==============================] - 4s 30ms/step - loss: 0.2652 - accuracy: 0.9014 - val_loss: 0.3350 - val_accuracy: 0.8672\n",
      "Epoch 4/10\n",
      "137/137 [==============================] - 4s 29ms/step - loss: 0.2429 - accuracy: 0.9118 - val_loss: 0.3368 - val_accuracy: 0.8628\n",
      "Epoch 5/10\n",
      "137/137 [==============================] - 4s 30ms/step - loss: 0.2266 - accuracy: 0.9186 - val_loss: 0.3441 - val_accuracy: 0.8619\n",
      "Epoch 6/10\n",
      "137/137 [==============================] - 4s 30ms/step - loss: 0.2121 - accuracy: 0.9268 - val_loss: 0.3553 - val_accuracy: 0.8637\n",
      "Epoch 7/10\n",
      "137/137 [==============================] - 4s 29ms/step - loss: 0.2007 - accuracy: 0.9303 - val_loss: 0.3548 - val_accuracy: 0.8698\n",
      "Epoch 8/10\n",
      "137/137 [==============================] - 4s 28ms/step - loss: 0.1873 - accuracy: 0.9335 - val_loss: 0.3538 - val_accuracy: 0.8619\n",
      "Epoch 9/10\n",
      "137/137 [==============================] - 4s 29ms/step - loss: 0.1809 - accuracy: 0.9384 - val_loss: 0.3880 - val_accuracy: 0.8558\n",
      "Epoch 10/10\n",
      "137/137 [==============================] - 4s 29ms/step - loss: 0.1725 - accuracy: 0.9407 - val_loss: 0.3942 - val_accuracy: 0.8566\n",
      "36/36 [==============================] - 0s 4ms/step - loss: 0.1919 - accuracy: 0.9349\n",
      "\n",
      "43 seconds.\n",
      "\n",
      "4 | Fri Oct 29 20:21:58 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 150).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 150).\n",
      "108/108 [==============================] - ETA: 0s - loss: 0.4966 - accuracy: 0.8038WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 150).\n",
      "108/108 [==============================] - 13s 105ms/step - loss: 0.4966 - accuracy: 0.8038 - val_loss: 0.3646 - val_accuracy: 0.8522\n",
      "Epoch 2/10\n",
      "108/108 [==============================] - 10s 96ms/step - loss: 0.3014 - accuracy: 0.8878 - val_loss: 0.3386 - val_accuracy: 0.8637\n",
      "Epoch 3/10\n",
      "108/108 [==============================] - 10s 96ms/step - loss: 0.2562 - accuracy: 0.9053 - val_loss: 0.3253 - val_accuracy: 0.8716\n",
      "Epoch 4/10\n",
      "108/108 [==============================] - 10s 96ms/step - loss: 0.2333 - accuracy: 0.9160 - val_loss: 0.3370 - val_accuracy: 0.8610\n",
      "Epoch 5/10\n",
      "108/108 [==============================] - 10s 95ms/step - loss: 0.2142 - accuracy: 0.9231 - val_loss: 0.3426 - val_accuracy: 0.8698\n",
      "Epoch 6/10\n",
      "108/108 [==============================] - 10s 95ms/step - loss: 0.2005 - accuracy: 0.9286 - val_loss: 0.3580 - val_accuracy: 0.8672\n",
      "Epoch 7/10\n",
      "108/108 [==============================] - 10s 96ms/step - loss: 0.1881 - accuracy: 0.9343 - val_loss: 0.3771 - val_accuracy: 0.8566\n",
      "Epoch 8/10\n",
      "108/108 [==============================] - 10s 96ms/step - loss: 0.1750 - accuracy: 0.9385 - val_loss: 0.3891 - val_accuracy: 0.8619\n",
      "Epoch 9/10\n",
      "108/108 [==============================] - 10s 96ms/step - loss: 0.1658 - accuracy: 0.9433 - val_loss: 0.4074 - val_accuracy: 0.8549\n",
      "Epoch 10/10\n",
      "108/108 [==============================] - 10s 97ms/step - loss: 0.1601 - accuracy: 0.9438 - val_loss: 0.4087 - val_accuracy: 0.8566\n",
      "36/36 [==============================] - 0s 11ms/step - loss: 0.1795 - accuracy: 0.9366\n",
      "\n",
      "107 seconds.\n",
      "\n",
      "5 | Fri Oct 29 20:23:45 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 190).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 190).\n",
      "108/108 [==============================] - ETA: 0s - loss: 0.4695 - accuracy: 0.7913WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 190).\n",
      "108/108 [==============================] - 18s 150ms/step - loss: 0.4695 - accuracy: 0.7913 - val_loss: 0.3490 - val_accuracy: 0.8610\n",
      "Epoch 2/10\n",
      "108/108 [==============================] - 15s 140ms/step - loss: 0.2903 - accuracy: 0.8904 - val_loss: 0.3296 - val_accuracy: 0.8646\n",
      "Epoch 3/10\n",
      "108/108 [==============================] - 15s 140ms/step - loss: 0.2551 - accuracy: 0.9053 - val_loss: 0.3216 - val_accuracy: 0.8716\n",
      "Epoch 4/10\n",
      "108/108 [==============================] - 15s 139ms/step - loss: 0.2295 - accuracy: 0.9144 - val_loss: 0.3390 - val_accuracy: 0.8637\n",
      "Epoch 5/10\n",
      "108/108 [==============================] - 15s 140ms/step - loss: 0.2114 - accuracy: 0.9231 - val_loss: 0.3446 - val_accuracy: 0.8698\n",
      "Epoch 6/10\n",
      "108/108 [==============================] - 15s 140ms/step - loss: 0.1962 - accuracy: 0.9281 - val_loss: 0.3422 - val_accuracy: 0.8690\n",
      "Epoch 7/10\n",
      "108/108 [==============================] - 15s 140ms/step - loss: 0.1818 - accuracy: 0.9352 - val_loss: 0.3675 - val_accuracy: 0.8646\n",
      "Epoch 8/10\n",
      "108/108 [==============================] - 15s 139ms/step - loss: 0.1731 - accuracy: 0.9371 - val_loss: 0.3891 - val_accuracy: 0.8531\n",
      "Epoch 9/10\n",
      "108/108 [==============================] - 15s 140ms/step - loss: 0.1633 - accuracy: 0.9422 - val_loss: 0.3995 - val_accuracy: 0.8575\n",
      "Epoch 10/10\n",
      "108/108 [==============================] - 15s 139ms/step - loss: 0.1548 - accuracy: 0.9448 - val_loss: 0.3982 - val_accuracy: 0.8584\n",
      "36/36 [==============================] - 1s 14ms/step - loss: 0.1393 - accuracy: 0.9577\n",
      "\n",
      "155 seconds.\n",
      "\n",
      "6 | Fri Oct 29 20:26:20 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 140).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 140).\n",
      "410/410 [==============================] - ETA: 0s - loss: 0.3913 - accuracy: 0.8307WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 140).\n",
      "410/410 [==============================] - 33s 78ms/step - loss: 0.3913 - accuracy: 0.8307 - val_loss: 0.3310 - val_accuracy: 0.8681\n",
      "Epoch 2/10\n",
      "410/410 [==============================] - 31s 75ms/step - loss: 0.2629 - accuracy: 0.9013 - val_loss: 0.3148 - val_accuracy: 0.8672\n",
      "Epoch 3/10\n",
      "410/410 [==============================] - 31s 76ms/step - loss: 0.2217 - accuracy: 0.9158 - val_loss: 0.3264 - val_accuracy: 0.8742\n",
      "Epoch 4/10\n",
      "410/410 [==============================] - 31s 76ms/step - loss: 0.1940 - accuracy: 0.9266 - val_loss: 0.3266 - val_accuracy: 0.8707\n",
      "Epoch 5/10\n",
      "410/410 [==============================] - 31s 76ms/step - loss: 0.1724 - accuracy: 0.9362 - val_loss: 0.3430 - val_accuracy: 0.8734\n",
      "Epoch 6/10\n",
      "410/410 [==============================] - 31s 75ms/step - loss: 0.1536 - accuracy: 0.9435 - val_loss: 0.3740 - val_accuracy: 0.8654\n",
      "Epoch 7/10\n",
      "410/410 [==============================] - 31s 76ms/step - loss: 0.1410 - accuracy: 0.9500 - val_loss: 0.3980 - val_accuracy: 0.8558\n",
      "Epoch 8/10\n",
      "410/410 [==============================] - 31s 75ms/step - loss: 0.1266 - accuracy: 0.9538 - val_loss: 0.4528 - val_accuracy: 0.8593\n",
      "Epoch 9/10\n",
      "410/410 [==============================] - 31s 76ms/step - loss: 0.1186 - accuracy: 0.9592 - val_loss: 0.4481 - val_accuracy: 0.8566\n",
      "Epoch 10/10\n",
      "410/410 [==============================] - 31s 75ms/step - loss: 0.1096 - accuracy: 0.9618 - val_loss: 0.4887 - val_accuracy: 0.8619\n",
      "36/36 [==============================] - 0s 9ms/step - loss: 0.1126 - accuracy: 0.9630\n",
      "\n",
      "314 seconds.\n",
      "\n",
      "7 | Fri Oct 29 20:31:34 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 80).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 80).\n",
      "227/228 [============================>.] - ETA: 0s - loss: 0.4634 - accuracy: 0.8063WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 80).\n",
      "228/228 [==============================] - 12s 48ms/step - loss: 0.4630 - accuracy: 0.8065 - val_loss: 0.3638 - val_accuracy: 0.8408\n",
      "Epoch 2/10\n",
      "228/228 [==============================] - 10s 44ms/step - loss: 0.3029 - accuracy: 0.8833 - val_loss: 0.3427 - val_accuracy: 0.8619\n",
      "Epoch 3/10\n",
      "228/228 [==============================] - 10s 43ms/step - loss: 0.2639 - accuracy: 0.9020 - val_loss: 0.3364 - val_accuracy: 0.8619\n",
      "Epoch 4/10\n",
      "228/228 [==============================] - 10s 43ms/step - loss: 0.2476 - accuracy: 0.9083 - val_loss: 0.3426 - val_accuracy: 0.8646\n",
      "Epoch 5/10\n",
      "228/228 [==============================] - 10s 44ms/step - loss: 0.2294 - accuracy: 0.9165 - val_loss: 0.3479 - val_accuracy: 0.8628\n",
      "Epoch 6/10\n",
      "228/228 [==============================] - 10s 43ms/step - loss: 0.2151 - accuracy: 0.9199 - val_loss: 0.3503 - val_accuracy: 0.8637\n",
      "Epoch 7/10\n",
      "228/228 [==============================] - 10s 43ms/step - loss: 0.2064 - accuracy: 0.9245 - val_loss: 0.3591 - val_accuracy: 0.8672\n",
      "Epoch 8/10\n",
      "228/228 [==============================] - 10s 44ms/step - loss: 0.1968 - accuracy: 0.9271 - val_loss: 0.3750 - val_accuracy: 0.8619\n",
      "Epoch 9/10\n",
      "228/228 [==============================] - 10s 43ms/step - loss: 0.1893 - accuracy: 0.9295 - val_loss: 0.3792 - val_accuracy: 0.8610\n",
      "Epoch 10/10\n",
      "228/228 [==============================] - 10s 43ms/step - loss: 0.1837 - accuracy: 0.9323 - val_loss: 0.3965 - val_accuracy: 0.8522\n",
      "36/36 [==============================] - 0s 5ms/step - loss: 0.1764 - accuracy: 0.9331\n",
      "\n",
      "102 seconds.\n",
      "\n",
      "8 | Fri Oct 29 20:33:16 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 70).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 70).\n",
      "293/293 [==============================] - ETA: 0s - loss: 0.4481 - accuracy: 0.8262WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 70).\n",
      "293/293 [==============================] - 13s 41ms/step - loss: 0.4481 - accuracy: 0.8262 - val_loss: 0.3495 - val_accuracy: 0.8646\n",
      "Epoch 2/10\n",
      "293/293 [==============================] - 11s 38ms/step - loss: 0.2875 - accuracy: 0.8904 - val_loss: 0.3346 - val_accuracy: 0.8549\n",
      "Epoch 3/10\n",
      "293/293 [==============================] - 11s 38ms/step - loss: 0.2433 - accuracy: 0.9106 - val_loss: 0.3328 - val_accuracy: 0.8698\n",
      "Epoch 4/10\n",
      "293/293 [==============================] - 11s 37ms/step - loss: 0.2174 - accuracy: 0.9239 - val_loss: 0.3432 - val_accuracy: 0.8707\n",
      "Epoch 5/10\n",
      "293/293 [==============================] - 11s 37ms/step - loss: 0.1997 - accuracy: 0.9286 - val_loss: 0.3468 - val_accuracy: 0.8663\n",
      "Epoch 6/10\n",
      "293/293 [==============================] - 11s 37ms/step - loss: 0.1881 - accuracy: 0.9328 - val_loss: 0.3545 - val_accuracy: 0.8628\n",
      "Epoch 7/10\n",
      "293/293 [==============================] - 11s 38ms/step - loss: 0.1726 - accuracy: 0.9388 - val_loss: 0.3595 - val_accuracy: 0.8654\n",
      "Epoch 8/10\n",
      "293/293 [==============================] - 11s 38ms/step - loss: 0.1583 - accuracy: 0.9438 - val_loss: 0.3827 - val_accuracy: 0.8672\n",
      "Epoch 9/10\n",
      "293/293 [==============================] - 11s 38ms/step - loss: 0.1534 - accuracy: 0.9458 - val_loss: 0.4082 - val_accuracy: 0.8522\n",
      "Epoch 10/10\n",
      "293/293 [==============================] - 11s 38ms/step - loss: 0.1423 - accuracy: 0.9506 - val_loss: 0.4241 - val_accuracy: 0.8540\n",
      "36/36 [==============================] - 0s 5ms/step - loss: 0.1642 - accuracy: 0.9340\n",
      "\n",
      "114 seconds.\n",
      "\n",
      "9 | Fri Oct 29 20:35:10 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 60).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 60).\n",
      "121/121 [==============================] - ETA: 0s - loss: 0.5225 - accuracy: 0.7937WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 60).\n",
      "121/121 [==============================] - 7s 44ms/step - loss: 0.5225 - accuracy: 0.7937 - val_loss: 0.3881 - val_accuracy: 0.8408\n",
      "Epoch 2/10\n",
      "121/121 [==============================] - 4s 36ms/step - loss: 0.3127 - accuracy: 0.8822 - val_loss: 0.3363 - val_accuracy: 0.8549\n",
      "Epoch 3/10\n",
      "121/121 [==============================] - 4s 35ms/step - loss: 0.2687 - accuracy: 0.9005 - val_loss: 0.3306 - val_accuracy: 0.8628\n",
      "Epoch 4/10\n",
      "121/121 [==============================] - 4s 35ms/step - loss: 0.2442 - accuracy: 0.9097 - val_loss: 0.3327 - val_accuracy: 0.8654\n",
      "Epoch 5/10\n",
      "121/121 [==============================] - 4s 37ms/step - loss: 0.2297 - accuracy: 0.9172 - val_loss: 0.3469 - val_accuracy: 0.8575\n",
      "Epoch 6/10\n",
      "121/121 [==============================] - 4s 35ms/step - loss: 0.2149 - accuracy: 0.9230 - val_loss: 0.3451 - val_accuracy: 0.8698\n",
      "Epoch 7/10\n",
      "121/121 [==============================] - 4s 35ms/step - loss: 0.2025 - accuracy: 0.9278 - val_loss: 0.3544 - val_accuracy: 0.8637\n",
      "Epoch 8/10\n",
      "121/121 [==============================] - 4s 35ms/step - loss: 0.1927 - accuracy: 0.9314 - val_loss: 0.3569 - val_accuracy: 0.8654\n",
      "Epoch 9/10\n",
      "121/121 [==============================] - 4s 35ms/step - loss: 0.1846 - accuracy: 0.9346 - val_loss: 0.3802 - val_accuracy: 0.8619\n",
      "Epoch 10/10\n",
      "121/121 [==============================] - 4s 35ms/step - loss: 0.1763 - accuracy: 0.9390 - val_loss: 0.3790 - val_accuracy: 0.8672\n",
      "36/36 [==============================] - 0s 4ms/step - loss: 0.1651 - accuracy: 0.9445\n",
      "\n",
      "46 seconds.\n",
      "\n",
      "10 | Fri Oct 29 20:35:57 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 70).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 70).\n",
      "292/293 [============================>.] - ETA: 0s - loss: 0.4894 - accuracy: 0.7979WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 70).\n",
      "293/293 [==============================] - 13s 41ms/step - loss: 0.4893 - accuracy: 0.7980 - val_loss: 0.3789 - val_accuracy: 0.8487\n",
      "Epoch 2/10\n",
      "293/293 [==============================] - 11s 38ms/step - loss: 0.3099 - accuracy: 0.8841 - val_loss: 0.3355 - val_accuracy: 0.8690\n",
      "Epoch 3/10\n",
      "293/293 [==============================] - 11s 38ms/step - loss: 0.2687 - accuracy: 0.9014 - val_loss: 0.3441 - val_accuracy: 0.8663\n",
      "Epoch 4/10\n",
      "293/293 [==============================] - 11s 37ms/step - loss: 0.2412 - accuracy: 0.9136 - val_loss: 0.3441 - val_accuracy: 0.8628\n",
      "Epoch 5/10\n",
      "293/293 [==============================] - 11s 38ms/step - loss: 0.2220 - accuracy: 0.9211 - val_loss: 0.3556 - val_accuracy: 0.8672\n",
      "Epoch 6/10\n",
      "293/293 [==============================] - 11s 37ms/step - loss: 0.2072 - accuracy: 0.9253 - val_loss: 0.3836 - val_accuracy: 0.8575\n",
      "Epoch 7/10\n",
      "293/293 [==============================] - 11s 38ms/step - loss: 0.2015 - accuracy: 0.9268 - val_loss: 0.3759 - val_accuracy: 0.8575\n",
      "Epoch 8/10\n",
      "293/293 [==============================] - 11s 38ms/step - loss: 0.1911 - accuracy: 0.9314 - val_loss: 0.3747 - val_accuracy: 0.8628\n",
      "Epoch 9/10\n",
      "293/293 [==============================] - 11s 38ms/step - loss: 0.1837 - accuracy: 0.9337 - val_loss: 0.4213 - val_accuracy: 0.8531\n",
      "Epoch 10/10\n",
      "293/293 [==============================] - 11s 38ms/step - loss: 0.1774 - accuracy: 0.9381 - val_loss: 0.4022 - val_accuracy: 0.8549\n",
      "36/36 [==============================] - 0s 5ms/step - loss: 0.1507 - accuracy: 0.9498\n",
      "\n",
      "114 seconds.\n",
      "\n",
      "11 | Fri Oct 29 20:37:50 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "158/158 [==============================] - 11s 62ms/step - loss: 0.5130 - accuracy: 0.8028 - val_loss: 0.3776 - val_accuracy: 0.8505\n",
      "Epoch 2/10\n",
      "158/158 [==============================] - 9s 56ms/step - loss: 0.3154 - accuracy: 0.8849 - val_loss: 0.3357 - val_accuracy: 0.8672\n",
      "Epoch 3/10\n",
      "158/158 [==============================] - 9s 57ms/step - loss: 0.2642 - accuracy: 0.9058 - val_loss: 0.3311 - val_accuracy: 0.8628\n",
      "Epoch 4/10\n",
      "158/158 [==============================] - 9s 57ms/step - loss: 0.2331 - accuracy: 0.9164 - val_loss: 0.3289 - val_accuracy: 0.8751\n",
      "Epoch 5/10\n",
      "158/158 [==============================] - 9s 57ms/step - loss: 0.2114 - accuracy: 0.9239 - val_loss: 0.3373 - val_accuracy: 0.8734\n",
      "Epoch 6/10\n",
      "158/158 [==============================] - 9s 56ms/step - loss: 0.1954 - accuracy: 0.9310 - val_loss: 0.3476 - val_accuracy: 0.8663\n",
      "Epoch 7/10\n",
      "158/158 [==============================] - 9s 56ms/step - loss: 0.1819 - accuracy: 0.9351 - val_loss: 0.3665 - val_accuracy: 0.8654\n",
      "Epoch 8/10\n",
      "158/158 [==============================] - 9s 57ms/step - loss: 0.1731 - accuracy: 0.9387 - val_loss: 0.3707 - val_accuracy: 0.8654\n",
      "Epoch 9/10\n",
      "158/158 [==============================] - 9s 56ms/step - loss: 0.1595 - accuracy: 0.9434 - val_loss: 0.3987 - val_accuracy: 0.8628\n",
      "Epoch 10/10\n",
      "158/158 [==============================] - 9s 56ms/step - loss: 0.1538 - accuracy: 0.9451 - val_loss: 0.3890 - val_accuracy: 0.8672\n",
      "36/36 [==============================] - 0s 8ms/step - loss: 0.1549 - accuracy: 0.9454\n",
      "\n",
      "93 seconds.\n",
      "\n",
      "12 | Fri Oct 29 20:39:23 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 150).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 150).\n",
      "137/137 [==============================] - ETA: 0s - loss: 0.4911 - accuracy: 0.8020WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 150).\n",
      "137/137 [==============================] - 14s 95ms/step - loss: 0.4911 - accuracy: 0.8020 - val_loss: 0.3698 - val_accuracy: 0.8478\n",
      "Epoch 2/10\n",
      "137/137 [==============================] - 12s 90ms/step - loss: 0.3060 - accuracy: 0.8844 - val_loss: 0.3370 - val_accuracy: 0.8558\n",
      "Epoch 3/10\n",
      "137/137 [==============================] - 12s 89ms/step - loss: 0.2613 - accuracy: 0.9036 - val_loss: 0.3414 - val_accuracy: 0.8637\n",
      "Epoch 4/10\n",
      "137/137 [==============================] - 12s 89ms/step - loss: 0.2331 - accuracy: 0.9184 - val_loss: 0.3458 - val_accuracy: 0.8628\n",
      "Epoch 5/10\n",
      "137/137 [==============================] - 12s 90ms/step - loss: 0.2153 - accuracy: 0.9243 - val_loss: 0.3482 - val_accuracy: 0.8698\n",
      "Epoch 6/10\n",
      "137/137 [==============================] - 12s 89ms/step - loss: 0.1995 - accuracy: 0.9281 - val_loss: 0.3592 - val_accuracy: 0.8646\n",
      "Epoch 7/10\n",
      "137/137 [==============================] - 12s 89ms/step - loss: 0.1874 - accuracy: 0.9316 - val_loss: 0.3634 - val_accuracy: 0.8619\n",
      "Epoch 8/10\n",
      "137/137 [==============================] - 12s 89ms/step - loss: 0.1753 - accuracy: 0.9392 - val_loss: 0.3764 - val_accuracy: 0.8602\n",
      "Epoch 9/10\n",
      "137/137 [==============================] - 12s 87ms/step - loss: 0.1660 - accuracy: 0.9428 - val_loss: 0.3816 - val_accuracy: 0.8575\n",
      "Epoch 10/10\n",
      "137/137 [==============================] - 12s 87ms/step - loss: 0.1597 - accuracy: 0.9428 - val_loss: 0.3812 - val_accuracy: 0.8602\n",
      "36/36 [==============================] - 0s 11ms/step - loss: 0.1389 - accuracy: 0.9516\n",
      "\n",
      "125 seconds.\n",
      "\n",
      "13 | Fri Oct 29 20:41:28 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 180).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 180).\n",
      "293/293 [==============================] - ETA: 0s - loss: 0.3943 - accuracy: 0.8274WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 180).\n",
      "293/293 [==============================] - 34s 111ms/step - loss: 0.3943 - accuracy: 0.8274 - val_loss: 0.3365 - val_accuracy: 0.8619\n",
      "Epoch 2/10\n",
      "293/293 [==============================] - 31s 107ms/step - loss: 0.2779 - accuracy: 0.8919 - val_loss: 0.3218 - val_accuracy: 0.8690\n",
      "Epoch 3/10\n",
      "293/293 [==============================] - 32s 108ms/step - loss: 0.2397 - accuracy: 0.9102 - val_loss: 0.3233 - val_accuracy: 0.8725\n",
      "Epoch 4/10\n",
      "293/293 [==============================] - 31s 107ms/step - loss: 0.2112 - accuracy: 0.9210 - val_loss: 0.3292 - val_accuracy: 0.8681\n",
      "Epoch 5/10\n",
      "293/293 [==============================] - 31s 107ms/step - loss: 0.1937 - accuracy: 0.9277 - val_loss: 0.3462 - val_accuracy: 0.8637\n",
      "Epoch 6/10\n",
      "293/293 [==============================] - 31s 107ms/step - loss: 0.1733 - accuracy: 0.9373 - val_loss: 0.3893 - val_accuracy: 0.8558\n",
      "Epoch 7/10\n",
      "293/293 [==============================] - 31s 107ms/step - loss: 0.1610 - accuracy: 0.9409 - val_loss: 0.3831 - val_accuracy: 0.8593\n",
      "Epoch 8/10\n",
      "293/293 [==============================] - 31s 107ms/step - loss: 0.1563 - accuracy: 0.9427 - val_loss: 0.4044 - val_accuracy: 0.8584\n",
      "Epoch 9/10\n",
      "293/293 [==============================] - 32s 108ms/step - loss: 0.1386 - accuracy: 0.9500 - val_loss: 0.4280 - val_accuracy: 0.8593\n",
      "Epoch 10/10\n",
      "293/293 [==============================] - 32s 108ms/step - loss: 0.1275 - accuracy: 0.9531 - val_loss: 0.4260 - val_accuracy: 0.8566\n",
      "36/36 [==============================] - 1s 15ms/step - loss: 0.1372 - accuracy: 0.9516\n",
      "\n",
      "318 seconds.\n",
      "\n",
      "14 | Fri Oct 29 20:46:47 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 140).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 140).\n",
      "186/186 [==============================] - ETA: 0s - loss: 0.4911 - accuracy: 0.8059WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 140).\n",
      "186/186 [==============================] - 17s 83ms/step - loss: 0.4911 - accuracy: 0.8059 - val_loss: 0.3790 - val_accuracy: 0.8417\n",
      "Epoch 2/10\n",
      "186/186 [==============================] - 15s 79ms/step - loss: 0.3060 - accuracy: 0.8830 - val_loss: 0.3441 - val_accuracy: 0.8619\n",
      "Epoch 3/10\n",
      "186/186 [==============================] - 15s 82ms/step - loss: 0.2594 - accuracy: 0.9042 - val_loss: 0.3391 - val_accuracy: 0.8646\n",
      "Epoch 4/10\n",
      "186/186 [==============================] - 15s 81ms/step - loss: 0.2360 - accuracy: 0.9158 - val_loss: 0.3426 - val_accuracy: 0.8646\n",
      "Epoch 5/10\n",
      "186/186 [==============================] - 15s 82ms/step - loss: 0.2219 - accuracy: 0.9191 - val_loss: 0.3527 - val_accuracy: 0.8619\n",
      "Epoch 6/10\n",
      "186/186 [==============================] - 15s 81ms/step - loss: 0.2065 - accuracy: 0.9271 - val_loss: 0.3547 - val_accuracy: 0.8646\n",
      "Epoch 7/10\n",
      "186/186 [==============================] - 15s 79ms/step - loss: 0.1965 - accuracy: 0.9287 - val_loss: 0.3688 - val_accuracy: 0.8637\n",
      "Epoch 8/10\n",
      "186/186 [==============================] - 15s 81ms/step - loss: 0.1869 - accuracy: 0.9335 - val_loss: 0.3676 - val_accuracy: 0.8646\n",
      "Epoch 9/10\n",
      "186/186 [==============================] - 14s 75ms/step - loss: 0.1779 - accuracy: 0.9358 - val_loss: 0.3869 - val_accuracy: 0.8610\n",
      "Epoch 10/10\n",
      "186/186 [==============================] - 14s 76ms/step - loss: 0.1732 - accuracy: 0.9384 - val_loss: 0.3923 - val_accuracy: 0.8584\n",
      "36/36 [==============================] - 0s 10ms/step - loss: 0.1889 - accuracy: 0.9384\n",
      "\n",
      "152 seconds.\n",
      "\n",
      "15 | Fri Oct 29 20:49:18 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 90).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 90).\n",
      "228/228 [==============================] - ETA: 0s - loss: 0.4360 - accuracy: 0.8122WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 90).\n",
      "228/228 [==============================] - 13s 53ms/step - loss: 0.4360 - accuracy: 0.8122 - val_loss: 0.3466 - val_accuracy: 0.8646\n",
      "Epoch 2/10\n",
      "228/228 [==============================] - 11s 49ms/step - loss: 0.2854 - accuracy: 0.8901 - val_loss: 0.3429 - val_accuracy: 0.8584\n",
      "Epoch 3/10\n",
      "228/228 [==============================] - 11s 49ms/step - loss: 0.2514 - accuracy: 0.9036 - val_loss: 0.3227 - val_accuracy: 0.8672\n",
      "Epoch 4/10\n",
      "228/228 [==============================] - 11s 48ms/step - loss: 0.2267 - accuracy: 0.9170 - val_loss: 0.3528 - val_accuracy: 0.8646\n",
      "Epoch 5/10\n",
      "228/228 [==============================] - 11s 49ms/step - loss: 0.2078 - accuracy: 0.9235 - val_loss: 0.3502 - val_accuracy: 0.8698\n",
      "Epoch 6/10\n",
      "228/228 [==============================] - 11s 48ms/step - loss: 0.1931 - accuracy: 0.9289 - val_loss: 0.3500 - val_accuracy: 0.8637\n",
      "Epoch 7/10\n",
      "228/228 [==============================] - 11s 48ms/step - loss: 0.1799 - accuracy: 0.9328 - val_loss: 0.3631 - val_accuracy: 0.8654\n",
      "Epoch 8/10\n",
      "228/228 [==============================] - 11s 48ms/step - loss: 0.1752 - accuracy: 0.9361 - val_loss: 0.3759 - val_accuracy: 0.8646\n",
      "Epoch 9/10\n",
      "228/228 [==============================] - 11s 49ms/step - loss: 0.1588 - accuracy: 0.9405 - val_loss: 0.3921 - val_accuracy: 0.8602\n",
      "Epoch 10/10\n",
      "228/228 [==============================] - 11s 48ms/step - loss: 0.1513 - accuracy: 0.9453 - val_loss: 0.4286 - val_accuracy: 0.8681\n",
      "36/36 [==============================] - 0s 6ms/step - loss: 0.1805 - accuracy: 0.9419\n",
      "\n",
      "113 seconds.\n",
      "\n",
      "16 | Fri Oct 29 20:51:12 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 150).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 150).\n",
      "2046/2046 [==============================] - ETA: 0s - loss: 0.3639 - accuracy: 0.8554WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 150).\n",
      "2046/2046 [==============================] - 143s 69ms/step - loss: 0.3639 - accuracy: 0.8554 - val_loss: 0.3308 - val_accuracy: 0.8646\n",
      "Epoch 2/10\n",
      "2046/2046 [==============================] - 140s 68ms/step - loss: 0.2585 - accuracy: 0.9003 - val_loss: 0.3229 - val_accuracy: 0.8646\n",
      "Epoch 3/10\n",
      "2046/2046 [==============================] - 140s 68ms/step - loss: 0.2177 - accuracy: 0.9185 - val_loss: 0.3326 - val_accuracy: 0.8672\n",
      "Epoch 4/10\n",
      "2046/2046 [==============================] - 139s 68ms/step - loss: 0.1859 - accuracy: 0.9288 - val_loss: 0.3662 - val_accuracy: 0.8663\n",
      "Epoch 5/10\n",
      "2046/2046 [==============================] - 139s 68ms/step - loss: 0.1608 - accuracy: 0.9416 - val_loss: 0.3622 - val_accuracy: 0.8751\n",
      "Epoch 6/10\n",
      "2046/2046 [==============================] - 148s 72ms/step - loss: 0.1444 - accuracy: 0.9468 - val_loss: 0.3816 - val_accuracy: 0.8725\n",
      "Epoch 7/10\n",
      "2046/2046 [==============================] - 149s 73ms/step - loss: 0.1261 - accuracy: 0.9562 - val_loss: 0.3978 - val_accuracy: 0.8619\n",
      "Epoch 8/10\n",
      "2046/2046 [==============================] - 149s 73ms/step - loss: 0.1159 - accuracy: 0.9607 - val_loss: 0.4460 - val_accuracy: 0.8672\n",
      "Epoch 9/10\n",
      "2046/2046 [==============================] - 152s 74ms/step - loss: 0.1047 - accuracy: 0.9647 - val_loss: 0.4425 - val_accuracy: 0.8663\n",
      "Epoch 10/10\n",
      "2046/2046 [==============================] - 150s 73ms/step - loss: 0.0957 - accuracy: 0.9687 - val_loss: 0.4842 - val_accuracy: 0.8522\n",
      "36/36 [==============================] - 0s 10ms/step - loss: 0.1200 - accuracy: 0.9665\n",
      "\n",
      "1450 seconds.\n",
      "\n",
      "17 | Fri Oct 29 21:15:21 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 70).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 70).\n",
      "409/410 [============================>.] - ETA: 0s - loss: 0.3917 - accuracy: 0.8346WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 70).\n",
      "410/410 [==============================] - 18s 40ms/step - loss: 0.3916 - accuracy: 0.8347 - val_loss: 0.3642 - val_accuracy: 0.8391\n",
      "Epoch 2/10\n",
      "410/410 [==============================] - 16s 38ms/step - loss: 0.2763 - accuracy: 0.8928 - val_loss: 0.3269 - val_accuracy: 0.8610\n",
      "Epoch 3/10\n",
      "410/410 [==============================] - 16s 39ms/step - loss: 0.2445 - accuracy: 0.9080 - val_loss: 0.3416 - val_accuracy: 0.8672\n",
      "Epoch 4/10\n",
      "410/410 [==============================] - 16s 38ms/step - loss: 0.2213 - accuracy: 0.9182 - val_loss: 0.3615 - val_accuracy: 0.8610\n",
      "Epoch 5/10\n",
      "410/410 [==============================] - 16s 38ms/step - loss: 0.2041 - accuracy: 0.9249 - val_loss: 0.3485 - val_accuracy: 0.8672\n",
      "Epoch 6/10\n",
      "410/410 [==============================] - 16s 38ms/step - loss: 0.1854 - accuracy: 0.9325 - val_loss: 0.3625 - val_accuracy: 0.8602\n",
      "Epoch 7/10\n",
      "410/410 [==============================] - 16s 38ms/step - loss: 0.1728 - accuracy: 0.9386 - val_loss: 0.3646 - val_accuracy: 0.8646\n",
      "Epoch 8/10\n",
      "410/410 [==============================] - 16s 38ms/step - loss: 0.1609 - accuracy: 0.9419 - val_loss: 0.3851 - val_accuracy: 0.8540\n",
      "Epoch 9/10\n",
      "410/410 [==============================] - 15s 38ms/step - loss: 0.1496 - accuracy: 0.9457 - val_loss: 0.3934 - val_accuracy: 0.8584\n",
      "Epoch 10/10\n",
      "410/410 [==============================] - 16s 38ms/step - loss: 0.1488 - accuracy: 0.9456 - val_loss: 0.4327 - val_accuracy: 0.8619\n",
      "36/36 [==============================] - 0s 6ms/step - loss: 0.1461 - accuracy: 0.9507\n",
      "\n",
      "160 seconds.\n",
      "\n",
      "18 | Fri Oct 29 21:18:01 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 140).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 140).\n",
      "158/158 [==============================] - ETA: 0s - loss: 0.4333 - accuracy: 0.8271WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 140).\n",
      "158/158 [==============================] - 16s 92ms/step - loss: 0.4333 - accuracy: 0.8271 - val_loss: 0.3393 - val_accuracy: 0.8575\n",
      "Epoch 2/10\n",
      "158/158 [==============================] - 14s 87ms/step - loss: 0.2834 - accuracy: 0.8913 - val_loss: 0.3305 - val_accuracy: 0.8646\n",
      "Epoch 3/10\n",
      "158/158 [==============================] - 14s 87ms/step - loss: 0.2473 - accuracy: 0.9081 - val_loss: 0.3337 - val_accuracy: 0.8698\n",
      "Epoch 4/10\n",
      "158/158 [==============================] - 14s 85ms/step - loss: 0.2217 - accuracy: 0.9177 - val_loss: 0.3295 - val_accuracy: 0.8751\n",
      "Epoch 5/10\n",
      "158/158 [==============================] - 14s 85ms/step - loss: 0.1995 - accuracy: 0.9282 - val_loss: 0.3437 - val_accuracy: 0.8654\n",
      "Epoch 6/10\n",
      "158/158 [==============================] - 13s 85ms/step - loss: 0.1853 - accuracy: 0.9317 - val_loss: 0.3563 - val_accuracy: 0.8646\n",
      "Epoch 7/10\n",
      "158/158 [==============================] - 13s 85ms/step - loss: 0.1714 - accuracy: 0.9393 - val_loss: 0.3808 - val_accuracy: 0.8619\n",
      "Epoch 8/10\n",
      "158/158 [==============================] - 13s 85ms/step - loss: 0.1571 - accuracy: 0.9454 - val_loss: 0.3925 - val_accuracy: 0.8584\n",
      "Epoch 9/10\n",
      "158/158 [==============================] - 13s 85ms/step - loss: 0.1464 - accuracy: 0.9494 - val_loss: 0.3961 - val_accuracy: 0.8602\n",
      "Epoch 10/10\n",
      "158/158 [==============================] - 14s 86ms/step - loss: 0.1404 - accuracy: 0.9509 - val_loss: 0.4140 - val_accuracy: 0.8593\n",
      "36/36 [==============================] - 0s 10ms/step - loss: 0.1481 - accuracy: 0.9560\n",
      "\n",
      "139 seconds.\n",
      "\n",
      "19 | Fri Oct 29 21:20:20 2021 | lowest loss so far: 0.1067 (run 2)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 110).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 110).\n",
      "2046/2046 [==============================] - ETA: 0s - loss: 0.3559 - accuracy: 0.8482WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 110).\n",
      "2046/2046 [==============================] - 118s 57ms/step - loss: 0.3559 - accuracy: 0.8482 - val_loss: 0.3188 - val_accuracy: 0.8760\n",
      "Epoch 2/10\n",
      "2046/2046 [==============================] - 115s 56ms/step - loss: 0.2543 - accuracy: 0.9034 - val_loss: 0.3200 - val_accuracy: 0.8663\n",
      "Epoch 3/10\n",
      "2046/2046 [==============================] - 115s 56ms/step - loss: 0.2162 - accuracy: 0.9167 - val_loss: 0.3209 - val_accuracy: 0.8681\n",
      "Epoch 4/10\n",
      "2046/2046 [==============================] - 114s 56ms/step - loss: 0.1884 - accuracy: 0.9294 - val_loss: 0.3293 - val_accuracy: 0.8681\n",
      "Epoch 5/10\n",
      "2046/2046 [==============================] - 115s 56ms/step - loss: 0.1643 - accuracy: 0.9401 - val_loss: 0.3538 - val_accuracy: 0.8707\n",
      "Epoch 6/10\n",
      "2046/2046 [==============================] - 115s 56ms/step - loss: 0.1475 - accuracy: 0.9459 - val_loss: 0.3993 - val_accuracy: 0.8628\n",
      "Epoch 7/10\n",
      "2046/2046 [==============================] - 115s 56ms/step - loss: 0.1321 - accuracy: 0.9524 - val_loss: 0.4442 - val_accuracy: 0.8619\n",
      "Epoch 8/10\n",
      "2046/2046 [==============================] - 115s 56ms/step - loss: 0.1194 - accuracy: 0.9581 - val_loss: 0.4461 - val_accuracy: 0.8514\n",
      "Epoch 9/10\n",
      "2046/2046 [==============================] - 115s 56ms/step - loss: 0.1087 - accuracy: 0.9640 - val_loss: 0.4456 - val_accuracy: 0.8593\n",
      "Epoch 10/10\n",
      "2046/2046 [==============================] - 115s 56ms/step - loss: 0.0970 - accuracy: 0.9670 - val_loss: 0.5412 - val_accuracy: 0.8531\n",
      "36/36 [==============================] - 0s 9ms/step - loss: 0.1051 - accuracy: 0.9674\n",
      "\n",
      "1153 seconds.\n",
      "\n",
      "20 | Fri Oct 29 21:39:33 2021 | lowest loss so far: 0.1051 (run 19)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 170).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 170).\n",
      "228/228 [==============================] - ETA: 0s - loss: 0.4778 - accuracy: 0.8154WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 170).\n",
      "228/228 [==============================] - 24s 100ms/step - loss: 0.4778 - accuracy: 0.8154 - val_loss: 0.3681 - val_accuracy: 0.8531\n",
      "Epoch 2/10\n",
      "228/228 [==============================] - 22s 96ms/step - loss: 0.3015 - accuracy: 0.8870 - val_loss: 0.3429 - val_accuracy: 0.8610\n",
      "Epoch 3/10\n",
      "228/228 [==============================] - 22s 95ms/step - loss: 0.2565 - accuracy: 0.9088 - val_loss: 0.3356 - val_accuracy: 0.8637\n",
      "Epoch 4/10\n",
      "228/228 [==============================] - 22s 97ms/step - loss: 0.2289 - accuracy: 0.9164 - val_loss: 0.3421 - val_accuracy: 0.8610\n",
      "Epoch 5/10\n",
      "228/228 [==============================] - 22s 95ms/step - loss: 0.2162 - accuracy: 0.9193 - val_loss: 0.3563 - val_accuracy: 0.8584\n",
      "Epoch 6/10\n",
      "228/228 [==============================] - 22s 95ms/step - loss: 0.1988 - accuracy: 0.9264 - val_loss: 0.3623 - val_accuracy: 0.8619\n",
      "Epoch 7/10\n",
      "228/228 [==============================] - 22s 97ms/step - loss: 0.1896 - accuracy: 0.9293 - val_loss: 0.3867 - val_accuracy: 0.8628\n",
      "Epoch 8/10\n",
      "228/228 [==============================] - 22s 95ms/step - loss: 0.1748 - accuracy: 0.9365 - val_loss: 0.4037 - val_accuracy: 0.8584\n",
      "Epoch 9/10\n",
      "228/228 [==============================] - 22s 95ms/step - loss: 0.1726 - accuracy: 0.9356 - val_loss: 0.3851 - val_accuracy: 0.8646\n",
      "Epoch 10/10\n",
      "228/228 [==============================] - 22s 96ms/step - loss: 0.1622 - accuracy: 0.9397 - val_loss: 0.3945 - val_accuracy: 0.8619\n",
      "36/36 [==============================] - 0s 11ms/step - loss: 0.1534 - accuracy: 0.9481\n",
      "\n",
      "222 seconds.\n",
      "\n",
      "21 | Fri Oct 29 21:43:15 2021 | lowest loss so far: 0.1051 (run 19)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 130).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 130).\n",
      "410/410 [==============================] - ETA: 0s - loss: 0.3823 - accuracy: 0.8378WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 130).\n",
      "410/410 [==============================] - 33s 76ms/step - loss: 0.3823 - accuracy: 0.8378 - val_loss: 0.3331 - val_accuracy: 0.8628\n",
      "Epoch 2/10\n",
      "410/410 [==============================] - 30s 74ms/step - loss: 0.2686 - accuracy: 0.8959 - val_loss: 0.3288 - val_accuracy: 0.8672\n",
      "Epoch 3/10\n",
      "410/410 [==============================] - 30s 74ms/step - loss: 0.2306 - accuracy: 0.9139 - val_loss: 0.3267 - val_accuracy: 0.8663\n",
      "Epoch 4/10\n",
      "410/410 [==============================] - 30s 74ms/step - loss: 0.2063 - accuracy: 0.9235 - val_loss: 0.3392 - val_accuracy: 0.8698\n",
      "Epoch 5/10\n",
      "410/410 [==============================] - 30s 74ms/step - loss: 0.1862 - accuracy: 0.9323 - val_loss: 0.3532 - val_accuracy: 0.8619\n",
      "Epoch 6/10\n",
      "410/410 [==============================] - 30s 74ms/step - loss: 0.1645 - accuracy: 0.9394 - val_loss: 0.3831 - val_accuracy: 0.8654\n",
      "Epoch 7/10\n",
      "410/410 [==============================] - 30s 74ms/step - loss: 0.1573 - accuracy: 0.9405 - val_loss: 0.4077 - val_accuracy: 0.8584\n",
      "Epoch 8/10\n",
      "410/410 [==============================] - 30s 74ms/step - loss: 0.1415 - accuracy: 0.9500 - val_loss: 0.3938 - val_accuracy: 0.8602\n",
      "Epoch 9/10\n",
      "410/410 [==============================] - 30s 74ms/step - loss: 0.1291 - accuracy: 0.9537 - val_loss: 0.4478 - val_accuracy: 0.8584\n",
      "Epoch 10/10\n",
      "410/410 [==============================] - 30s 74ms/step - loss: 0.1294 - accuracy: 0.9545 - val_loss: 0.4257 - val_accuracy: 0.8558\n",
      "36/36 [==============================] - 0s 10ms/step - loss: 0.1157 - accuracy: 0.9648\n",
      "\n",
      "307 seconds.\n",
      "\n",
      "22 | Fri Oct 29 21:48:22 2021 | lowest loss so far: 0.1051 (run 19)\n",
      "\n",
      "Epoch 1/10\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 120).\n",
      "WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 120).\n",
      "2046/2046 [==============================] - ETA: 0s - loss: 0.3521 - accuracy: 0.8548WARNING:tensorflow:Model was constructed with shape (None, 100) for input KerasTensor(type_spec=TensorSpec(shape=(None, 100), dtype=tf.float32, name='input'), name='input', description=\"created by layer 'input'\"), but it was called on an input with incompatible shape (None, 120).\n",
      "2046/2046 [==============================] - 128s 62ms/step - loss: 0.3521 - accuracy: 0.8548 - val_loss: 0.3300 - val_accuracy: 0.8593\n",
      "Epoch 2/10\n",
      "2046/2046 [==============================] - 125s 61ms/step - loss: 0.2547 - accuracy: 0.9019 - val_loss: 0.3153 - val_accuracy: 0.8716\n",
      "Epoch 3/10\n",
      "2046/2046 [==============================] - 124s 61ms/step - loss: 0.2162 - accuracy: 0.9188 - val_loss: 0.3266 - val_accuracy: 0.8610\n",
      "Epoch 4/10\n",
      "2046/2046 [==============================] - 124s 61ms/step - loss: 0.1849 - accuracy: 0.9313 - val_loss: 0.3526 - val_accuracy: 0.8672\n",
      "Epoch 5/10\n",
      "2046/2046 [==============================] - 124s 61ms/step - loss: 0.1614 - accuracy: 0.9420 - val_loss: 0.3861 - val_accuracy: 0.8672\n",
      "Epoch 6/10\n",
      "2046/2046 [==============================] - 124s 61ms/step - loss: 0.1450 - accuracy: 0.9474 - val_loss: 0.3962 - val_accuracy: 0.8663\n",
      "Epoch 7/10\n",
      "2046/2046 [==============================] - ETA: 0s - loss: 0.1266 - accuracy: 0.9554"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "> **Nhận xét**:\n",
    "> * Ở đây nhóm đã tiến hành ngắt quá trình đào tạo lại, hình dưới đây được chụp từ file `hyperband.txt` - file này ghi lại lịch sử của quá trình optimization, ta thấy rằng:\n",
    "> ![](./images/17.png)\n",
    ">   * Dù trải qua 20 lần đào tạo lại các model dựa trên các hyperparam-set khác nhau cho validation loss thấp nhất, nhưng giá trị này vẫn lớn hơn 0.3 tương tự như `model_1` và `model_2` mà ta đào tạo ban đầu, xem ra đây có thể là hiệu suất tốt nhất mà model LSTM của ta có thể làm được. \n",
    ">   * Từ đây góp phần cho ta chắc chắn hơn rằng dataset comments của chúng ta đang thực sự bị variance error."
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "orig_nbformat": 4,
  "language_info": {
   "name": "python",
   "version": "3.8.10",
   "mimetype": "text/x-python",
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "pygments_lexer": "ipython3",
   "nbconvert_exporter": "python",
   "file_extension": ".py"
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.8.10 64-bit"
  },
  "interpreter": {
   "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}