01_AT&T_spam_detector_final.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mV2KEWKwLly7"
      },
      "source": [
        "<img src=\"https://full-stack-assets.s3.eu-west-3.amazonaws.com/M08-deep-learning/AT%26T_logo_2016.svg\" alt=\"AT&T LOGO\" width=\"50%\" />\n",
        "\n",
        "# Orange SPAM detector"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JjzRYoEwLly_"
      },
      "source": [
        "## Company's Description 📇\n",
        "\n",
        "AT&T Inc. is an American multinational telecommunications holding company headquartered at Whitacre Tower in Downtown Dallas, Texas. It is the world's largest telecommunications company by revenue and the third largest provider of mobile telephone services in the U.S. As of 2022, AT&T was ranked 13th on the Fortune 500 rankings of the largest United States corporations, with revenues of $168.8 billion! 😮"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Au-CdV4ELlzA"
      },
      "source": [
        "## Project 🚧\n",
        "\n",
        "One of the main pain point that AT&T users are facing is constant exposure to SPAM messages.\n",
        "\n",
        "AT&T has been able to manually flag spam messages for a time, but they are looking for an automated way of detecting spams to protect their users."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pHyUbBJbLlzB"
      },
      "source": [
        "## Goals 🎯\n",
        "\n",
        "Your goal is to build a spam detector, that can automatically flag spams as they come based sollely on the sms' content."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bp4LzG91LlzB"
      },
      "source": [
        "## Scope of this project 🖼️\n",
        "\n",
        "To start off, AT&T would like you to use the folowing dataset:\n",
        "\n",
        "[Dowload the Dataset](https://full-stack-bigdata-datasets.s3.eu-west-3.amazonaws.com/Deep+Learning/project/spam.csv)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IoxLMFboLlzC"
      },
      "source": [
        "## Deliverable 📬\n",
        "\n",
        "To complete this project, your team should: \n",
        "\n",
        "* Write a notebook that runs preprocessing and trains one or more deep learning models in order to predict the spam or ham nature of the sms\n",
        "* State the achieved performance clearly"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install -q -U \"tensorflow-text==2.9.*\""
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MVqcJ9C53HGt",
        "outputId": "77b03bc4-1050-4d05-8e01-215c20a96b87"
      },
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[K     |████████████████████████████████| 4.6 MB 13.0 MB/s \n",
            "\u001b[K     |████████████████████████████████| 511.8 MB 27 kB/s \n",
            "\u001b[K     |████████████████████████████████| 438 kB 53.1 MB/s \n",
            "\u001b[K     |████████████████████████████████| 1.6 MB 57.5 MB/s \n",
            "\u001b[K     |████████████████████████████████| 5.8 MB 46.3 MB/s \n",
            "\u001b[?25h"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Import Tensorflow & Pathlib librairies\n",
        "import tensorflow as tf \n",
        "import tensorflow_hub as hub\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "import pathlib \n",
        "import pandas as pd \n",
        "import os\n",
        "import io\n",
        "import warnings\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import accuracy_score,f1_score\n",
        "warnings.filterwarnings('ignore')"
      ],
      "metadata": {
        "id": "WNEYHB1DLrgZ"
      },
      "execution_count": 68,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tf.__version__"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "nW2QZRxR5OFX",
        "outputId": "051f3635-875b-45e0-d87a-b89a1d5c0657"
      },
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'2.9.2'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow_text"
      ],
      "metadata": {
        "id": "QtITtl5y37bx"
      },
      "execution_count": 71,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Import dataset with Pandas \n",
        "dataset = pd.read_csv(\"/content/spam.csv\", error_bad_lines=False, encoding=\"Latin-1\")\n",
        "dataset=dataset.iloc[:5000,:2] #we will focus only on 5000 first lines\n",
        "dataset.head()"
      ],
      "metadata": {
        "id": "M977JmBpL_0q",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "outputId": "85b18b82-d6ba-4c79-a85c-482b4255f5a7"
      },
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     v1                                                 v2\n",
              "0   ham  Go until jurong point, crazy.. Available only ...\n",
              "1   ham                      Ok lar... Joking wif u oni...\n",
              "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
              "3   ham  U dun say so early hor... U c already then say...\n",
              "4   ham  Nah I don't think he goes to usf, he lives aro..."
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-db11ae83-76b7-43d9-9c34-62eea3221314\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>v1</th>\n",
              "      <th>v2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-db11ae83-76b7-43d9-9c34-62eea3221314')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-db11ae83-76b7-43d9-9c34-62eea3221314 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-db11ae83-76b7-43d9-9c34-62eea3221314');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Preprocessing\n",
        "\n",
        "We will now go through a preprocessing phase. The goal is to clean up the character strings and encode the words so they are represented as integers."
      ],
      "metadata": {
        "id": "-GZChxiSUJS0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python -m spacy download en_core_web_sm"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AsuE8n2iUMHT",
        "outputId": "bb5b7bc7-7621-4b45-a92f-9c74cb2d9a61"
      },
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/torch/cuda/__init__.py:497: UserWarning: Can't initialize NVML\n",
            "  warnings.warn(\"Can't initialize NVML\")\n",
            "2022-12-15 17:48:39.261541: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting en-core-web-sm==3.4.1\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.4.1/en_core_web_sm-3.4.1-py3-none-any.whl (12.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 12.8 MB 27.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: spacy<3.5.0,>=3.4.0 in /usr/local/lib/python3.8/dist-packages (from en-core-web-sm==3.4.1) (3.4.3)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (1.0.3)\n",
            "Requirement already satisfied: wasabi<1.1.0,>=0.9.1 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (0.10.1)\n",
            "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (1.21.6)\n",
            "Requirement already satisfied: typer<0.8.0,>=0.3.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (0.7.0)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (2.0.7)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (1.0.9)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (2.4.5)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (3.3.0)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (2.23.0)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.10 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (3.0.10)\n",
            "Requirement already satisfied: thinc<8.2.0,>=8.1.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (8.1.5)\n",
            "Requirement already satisfied: pathy>=0.3.5 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (0.10.0)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (57.4.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (21.3)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (1.10.2)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (3.0.8)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (2.0.8)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (4.64.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (2.11.3)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging>=20.0->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (3.0.9)\n",
            "Requirement already satisfied: smart-open<6.0.0,>=5.2.1 in /usr/local/lib/python3.8/dist-packages (from pathy>=0.3.5->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (5.2.1)\n",
            "Requirement already satisfied: typing-extensions>=4.1.0 in /usr/local/lib/python3.8/dist-packages (from pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (4.4.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (2022.9.24)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (2.10)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.8/dist-packages (from thinc<8.2.0,>=8.1.0->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (0.0.3)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.8/dist-packages (from thinc<8.2.0,>=8.1.0->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (0.7.9)\n",
            "Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.8/dist-packages (from typer<0.8.0,>=0.3.0->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (7.1.2)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.8/dist-packages (from jinja2->spacy<3.5.0,>=3.4.0->en-core-web-sm==3.4.1) (2.0.1)\n",
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the package via spacy.load('en_core_web_sm')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import en_core_web_sm\n",
        "nlp = en_core_web_sm.load()"
      ],
      "metadata": {
        "id": "TTsAK65gUW5l"
      },
      "execution_count": 74,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Import Stop words \n",
        "from spacy.lang.fr.stop_words import STOP_WORDS"
      ],
      "metadata": {
        "id": "NCr2wb0Knxsv"
      },
      "execution_count": 75,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "for token in nlp(dataset['v2'][0]) : \n",
        "  print(token.lemma_) "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-95c_q9mYB42",
        "outputId": "00969f59-3c38-41d5-92fa-7ec4c055e363"
      },
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "go\n",
            "until\n",
            "jurong\n",
            "point\n",
            ",\n",
            "crazy\n",
            "..\n",
            "available\n",
            "only\n",
            "in\n",
            "bugis\n",
            "n\n",
            "great\n",
            "world\n",
            "la\n",
            "e\n",
            "buffet\n",
            "...\n",
            "Cine\n",
            "there\n",
            "get\n",
            "amore\n",
            "wat\n",
            "...\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from re import X\n",
        "# Ce bout de code va processer tout le dataframe : c'est normal s'il tourne longtemps, n'hésitez pas à continuer à coder la suite de l'exo le temps qu'il tourne\n",
        "\n",
        "dataset[\"sms_clean\"] = dataset[\"v2\"].apply(lambda x:''.join(ch for ch in x if ch.isalnum() or ch==\" \" or ch==\"'\"))\n",
        "dataset[\"sms_clean\"] = dataset[\"sms_clean\"].apply(lambda x: x.replace(\" +\",\" \").lower().strip())\n",
        "dataset[\"sms_clean\"] = dataset[\"sms_clean\"].apply(lambda x: \" \".join([token.lemma_ for token in nlp(x) if (token.lemma_ not in STOP_WORDS) and (token.text not in STOP_WORDS)]))\n",
        "dataset.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "EvqKL9kxVxsZ",
        "outputId": "09e179f0-7e05-4d35-dd08-e59287cb17d8"
      },
      "execution_count": 77,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     v1                                                 v2  \\\n",
              "0   ham  Go until jurong point, crazy.. Available only ...   \n",
              "1   ham                      Ok lar... Joking wif u oni...   \n",
              "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...   \n",
              "3   ham  U dun say so early hor... U c already then say...   \n",
              "4   ham  Nah I don't think he goes to usf, he lives aro...   \n",
              "\n",
              "                                           sms_clean  \n",
              "0  go until jurong point crazy available only in ...  \n",
              "1                            ok lar joking wif u oni  \n",
              "2  free entry in 2 wkly comp to win fa cup final ...  \n",
              "3        u dun say so early hor u c already then say  \n",
              "4  nah do not think he go to usf he live around h...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-2ff2b9af-54fe-42b9-98f8-84185b624fab\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>v1</th>\n",
              "      <th>v2</th>\n",
              "      <th>sms_clean</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "      <td>go until jurong point crazy available only in ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "      <td>ok lar joking wif u oni</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "      <td>free entry in 2 wkly comp to win fa cup final ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "      <td>u dun say so early hor u c already then say</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "      <td>nah do not think he go to usf he live around h...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2ff2b9af-54fe-42b9-98f8-84185b624fab')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-2ff2b9af-54fe-42b9-98f8-84185b624fab button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-2ff2b9af-54fe-42b9-98f8-84185b624fab');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 77
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=1000, oov_token=\"out_of_vocab\") # instanciate the tokenizer\n",
        "tokenizer.fit_on_texts(dataset.sms_clean)\n",
        "dataset[\"sms_encoded\"] = tokenizer.texts_to_sequences(dataset.sms_clean)\n",
        "#reformater le v1 en chiffres 0 et 1\n",
        "dataset[\"categ\"] = dataset[\"v1\"].apply(lambda x: 1 if x==\"spam\" else 0)\n",
        "dataset.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 320
        },
        "id": "5bFwDGTksgt9",
        "outputId": "aff4aede-5e81-4a2d-dbe3-52c9e01c1db8"
      },
      "execution_count": 78,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     v1                                                 v2  \\\n",
              "0   ham  Go until jurong point, crazy.. Available only ...   \n",
              "1   ham                      Ok lar... Joking wif u oni...   \n",
              "2  spam  Free entry in 2 a wkly comp to win FA Cup fina...   \n",
              "3   ham  U dun say so early hor... U c already then say...   \n",
              "4   ham  Nah I don't think he goes to usf, he lives aro...   \n",
              "\n",
              "                                           sms_clean  \\\n",
              "0  go until jurong point crazy available only in ...   \n",
              "1                            ok lar joking wif u oni   \n",
              "2  free entry in 2 wkly comp to win fa cup final ...   \n",
              "3        u dun say so early hor u c already then say   \n",
              "4  nah do not think he go to usf he live around h...   \n",
              "\n",
              "                                         sms_encoded  categ  \n",
              "0  [21, 380, 1, 373, 607, 561, 63, 10, 1, 87, 109...      0  \n",
              "1                            [42, 305, 1, 395, 7, 1]      0  \n",
              "2  [49, 444, 10, 24, 749, 924, 3, 83, 1, 925, 562...      1  \n",
              "3      [7, 221, 54, 26, 237, 1, 7, 142, 140, 53, 54]      0  \n",
              "4  [796, 8, 6, 66, 36, 21, 3, 750, 36, 213, 197, ...      0  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-f2a1303a-a13f-4f86-a2ca-2bd209b3adee\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>v1</th>\n",
              "      <th>v2</th>\n",
              "      <th>sms_clean</th>\n",
              "      <th>sms_encoded</th>\n",
              "      <th>categ</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "      <td>go until jurong point crazy available only in ...</td>\n",
              "      <td>[21, 380, 1, 373, 607, 561, 63, 10, 1, 87, 109...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "      <td>ok lar joking wif u oni</td>\n",
              "      <td>[42, 305, 1, 395, 7, 1]</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "      <td>free entry in 2 wkly comp to win fa cup final ...</td>\n",
              "      <td>[49, 444, 10, 24, 749, 924, 3, 83, 1, 925, 562...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "      <td>u dun say so early hor u c already then say</td>\n",
              "      <td>[7, 221, 54, 26, 237, 1, 7, 142, 140, 53, 54]</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "      <td>nah do not think he go to usf he live around h...</td>\n",
              "      <td>[796, 8, 6, 66, 36, 21, 3, 750, 36, 213, 197, ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f2a1303a-a13f-4f86-a2ca-2bd209b3adee')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-f2a1303a-a13f-4f86-a2ca-2bd209b3adee button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-f2a1303a-a13f-4f86-a2ca-2bd209b3adee');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sms_pad = tf.keras.preprocessing.sequence.pad_sequences(dataset.sms_encoded, padding=\"post\")"
      ],
      "metadata": {
        "id": "vehm3rTfwSZJ"
      },
      "execution_count": 79,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Train test split"
      ],
      "metadata": {
        "id": "yQ29Souo4dvq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Train Test Split\n",
        "# Train Test Split\n",
        "xtrain, xval, ytrain, yval = train_test_split(sms_pad,dataset.categ, test_size=0.3)"
      ],
      "metadata": {
        "id": "gKy004Lt4f-j"
      },
      "execution_count": 80,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train = tf.data.Dataset.from_tensor_slices((xtrain, ytrain))\n",
        "val = tf.data.Dataset.from_tensor_slices((xval, yval))"
      ],
      "metadata": {
        "id": "6Sk-0uwz4pnZ"
      },
      "execution_count": 81,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_batch = train.shuffle(len(train)).batch(64)\n",
        "val_batch = val.shuffle(len(val)).batch(64)"
      ],
      "metadata": {
        "id": "GtSnzMlf5JKx"
      },
      "execution_count": 82,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        " # Regardons un batch \n",
        "for sms, categ in train_batch.take(1):\n",
        "  print(sms, categ)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JMkkfWtQ5MXO",
        "outputId": "d2c6ba3f-5895-430d-dcaa-5286992ca208"
      },
      "execution_count": 83,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tf.Tensor(\n",
            "[[ 49 444  10 ...   0   0   0]\n",
            " [549   1 269 ...   0   0   0]\n",
            " [154 170 112 ...   0   0   0]\n",
            " ...\n",
            " [  1  57 278 ...   0   0   0]\n",
            " [  1 308 332 ...   0   0   0]\n",
            " [ 39   7  57 ...   0   0   0]], shape=(64, 164), dtype=int32) tf.Tensor(\n",
            "[1 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            " 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0], shape=(64,), dtype=int64)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sms.shape[1]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3SM9kj8N9mh8",
        "outputId": "d1dfa5ec-023a-47c6-f181-17e7044760b2"
      },
      "execution_count": 84,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "164"
            ]
          },
          "metadata": {},
          "execution_count": 84
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Modeling with sequential simple Model \n",
        "\n",
        "Let's create a model in order to train an embedding!"
      ],
      "metadata": {
        "id": "mpLOlVcI56ib"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Ici je vous met la version classification\n",
        "vocab_size = tokenizer.num_words\n",
        "model = tf.keras.Sequential([\n",
        "                  # Couche d'Input Word Embedding           \n",
        "                  tf.keras.layers.Embedding(vocab_size+1, 8, input_shape=[sms.shape[1]],name=\"embedding\"),\n",
        "                  # Gobal max pooling\n",
        "                  tf.keras.layers.GlobalMaxPooling1D(),\n",
        "\n",
        "                  # Couche Dense classique\n",
        "                  tf.keras.layers.Dense(16, activation='relu'),\n",
        "\n",
        "                  # Couche de sortie avec le nombre de neurones en sortie égale au nombre de classe avec fonction sigmoid\n",
        "                  tf.keras.layers.Dense(1, activation=\"sigmoid\")\n",
        "])"
      ],
      "metadata": {
        "id": "xon9DfyS5TiF"
      },
      "execution_count": 85,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V_rHD91E6KeC",
        "outputId": "a379d7de-5e53-4dee-939f-f085f259bb77"
      },
      "execution_count": 86,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential_3\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding (Embedding)       (None, 164, 8)            8008      \n",
            "                                                                 \n",
            " global_max_pooling1d_1 (Glo  (None, 8)                0         \n",
            " balMaxPooling1D)                                                \n",
            "                                                                 \n",
            " dense_6 (Dense)             (None, 16)                144       \n",
            "                                                                 \n",
            " dense_7 (Dense)             (None, 1)                 17        \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 8,169\n",
            "Trainable params: 8,169\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "optimizer= tf.keras.optimizers.Adam()\n",
        "\n",
        "model.compile(optimizer=optimizer,\n",
        "              loss=tf.keras.losses.BinaryCrossentropy(),\n",
        "              metrics=[tf.keras.metrics.BinaryAccuracy\n",
        "              ()])\n"
      ],
      "metadata": {
        "id": "tYSmyAea6X80"
      },
      "execution_count": 87,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "history = model.fit(train_batch, epochs=50, validation_data=val_batch)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yqdXrbZj8-Qt",
        "outputId": "da1c19f8-b733-499f-cf67-24da49f540c5"
      },
      "execution_count": 88,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/50\n",
            "55/55 [==============================] - 1s 7ms/step - loss: 0.6592 - binary_accuracy: 0.7963 - val_loss: 0.6028 - val_binary_accuracy: 0.8720\n",
            "Epoch 2/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.5429 - binary_accuracy: 0.8626 - val_loss: 0.4707 - val_binary_accuracy: 0.8720\n",
            "Epoch 3/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.4265 - binary_accuracy: 0.8626 - val_loss: 0.3736 - val_binary_accuracy: 0.8720\n",
            "Epoch 4/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.3503 - binary_accuracy: 0.8626 - val_loss: 0.3094 - val_binary_accuracy: 0.8720\n",
            "Epoch 5/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.2678 - binary_accuracy: 0.8671 - val_loss: 0.2333 - val_binary_accuracy: 0.8973\n",
            "Epoch 6/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.1836 - binary_accuracy: 0.9369 - val_loss: 0.1699 - val_binary_accuracy: 0.9520\n",
            "Epoch 7/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.1252 - binary_accuracy: 0.9703 - val_loss: 0.1330 - val_binary_accuracy: 0.9627\n",
            "Epoch 8/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0927 - binary_accuracy: 0.9734 - val_loss: 0.1144 - val_binary_accuracy: 0.9667\n",
            "Epoch 9/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0745 - binary_accuracy: 0.9794 - val_loss: 0.1042 - val_binary_accuracy: 0.9647\n",
            "Epoch 10/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0633 - binary_accuracy: 0.9806 - val_loss: 0.0984 - val_binary_accuracy: 0.9667\n",
            "Epoch 11/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0550 - binary_accuracy: 0.9831 - val_loss: 0.0944 - val_binary_accuracy: 0.9687\n",
            "Epoch 12/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0484 - binary_accuracy: 0.9857 - val_loss: 0.0922 - val_binary_accuracy: 0.9693\n",
            "Epoch 13/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0427 - binary_accuracy: 0.9866 - val_loss: 0.0905 - val_binary_accuracy: 0.9720\n",
            "Epoch 14/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0381 - binary_accuracy: 0.9889 - val_loss: 0.0896 - val_binary_accuracy: 0.9740\n",
            "Epoch 15/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0343 - binary_accuracy: 0.9891 - val_loss: 0.0885 - val_binary_accuracy: 0.9740\n",
            "Epoch 16/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0310 - binary_accuracy: 0.9909 - val_loss: 0.0882 - val_binary_accuracy: 0.9753\n",
            "Epoch 17/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0282 - binary_accuracy: 0.9909 - val_loss: 0.0883 - val_binary_accuracy: 0.9760\n",
            "Epoch 18/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0255 - binary_accuracy: 0.9920 - val_loss: 0.0884 - val_binary_accuracy: 0.9773\n",
            "Epoch 19/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0233 - binary_accuracy: 0.9931 - val_loss: 0.0888 - val_binary_accuracy: 0.9780\n",
            "Epoch 20/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0213 - binary_accuracy: 0.9934 - val_loss: 0.0891 - val_binary_accuracy: 0.9787\n",
            "Epoch 21/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0195 - binary_accuracy: 0.9940 - val_loss: 0.0901 - val_binary_accuracy: 0.9787\n",
            "Epoch 22/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0179 - binary_accuracy: 0.9949 - val_loss: 0.0908 - val_binary_accuracy: 0.9793\n",
            "Epoch 23/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0163 - binary_accuracy: 0.9963 - val_loss: 0.0914 - val_binary_accuracy: 0.9793\n",
            "Epoch 24/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0150 - binary_accuracy: 0.9966 - val_loss: 0.0920 - val_binary_accuracy: 0.9793\n",
            "Epoch 25/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0137 - binary_accuracy: 0.9966 - val_loss: 0.0925 - val_binary_accuracy: 0.9793\n",
            "Epoch 26/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0127 - binary_accuracy: 0.9974 - val_loss: 0.0932 - val_binary_accuracy: 0.9793\n",
            "Epoch 27/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0118 - binary_accuracy: 0.9974 - val_loss: 0.0941 - val_binary_accuracy: 0.9793\n",
            "Epoch 28/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0109 - binary_accuracy: 0.9980 - val_loss: 0.0947 - val_binary_accuracy: 0.9793\n",
            "Epoch 29/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0103 - binary_accuracy: 0.9980 - val_loss: 0.0956 - val_binary_accuracy: 0.9800\n",
            "Epoch 30/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0095 - binary_accuracy: 0.9980 - val_loss: 0.0959 - val_binary_accuracy: 0.9793\n",
            "Epoch 31/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0090 - binary_accuracy: 0.9986 - val_loss: 0.0968 - val_binary_accuracy: 0.9793\n",
            "Epoch 32/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0084 - binary_accuracy: 0.9986 - val_loss: 0.0972 - val_binary_accuracy: 0.9793\n",
            "Epoch 33/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0078 - binary_accuracy: 0.9991 - val_loss: 0.0980 - val_binary_accuracy: 0.9793\n",
            "Epoch 34/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0074 - binary_accuracy: 0.9991 - val_loss: 0.0985 - val_binary_accuracy: 0.9780\n",
            "Epoch 35/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0071 - binary_accuracy: 0.9991 - val_loss: 0.0993 - val_binary_accuracy: 0.9787\n",
            "Epoch 36/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0067 - binary_accuracy: 0.9994 - val_loss: 0.0997 - val_binary_accuracy: 0.9787\n",
            "Epoch 37/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0063 - binary_accuracy: 0.9994 - val_loss: 0.0999 - val_binary_accuracy: 0.9787\n",
            "Epoch 38/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0060 - binary_accuracy: 0.9997 - val_loss: 0.0998 - val_binary_accuracy: 0.9787\n",
            "Epoch 39/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0058 - binary_accuracy: 0.9994 - val_loss: 0.1008 - val_binary_accuracy: 0.9780\n",
            "Epoch 40/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0055 - binary_accuracy: 0.9997 - val_loss: 0.1011 - val_binary_accuracy: 0.9780\n",
            "Epoch 41/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0053 - binary_accuracy: 0.9997 - val_loss: 0.1011 - val_binary_accuracy: 0.9787\n",
            "Epoch 42/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0050 - binary_accuracy: 0.9997 - val_loss: 0.1018 - val_binary_accuracy: 0.9787\n",
            "Epoch 43/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0048 - binary_accuracy: 0.9997 - val_loss: 0.1022 - val_binary_accuracy: 0.9787\n",
            "Epoch 44/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0047 - binary_accuracy: 0.9997 - val_loss: 0.1025 - val_binary_accuracy: 0.9793\n",
            "Epoch 45/50\n",
            "55/55 [==============================] - 0s 5ms/step - loss: 0.0045 - binary_accuracy: 0.9997 - val_loss: 0.1032 - val_binary_accuracy: 0.9793\n",
            "Epoch 46/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0043 - binary_accuracy: 0.9997 - val_loss: 0.1035 - val_binary_accuracy: 0.9800\n",
            "Epoch 47/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0042 - binary_accuracy: 0.9997 - val_loss: 0.1038 - val_binary_accuracy: 0.9800\n",
            "Epoch 48/50\n",
            "55/55 [==============================] - 0s 3ms/step - loss: 0.0040 - binary_accuracy: 0.9997 - val_loss: 0.1045 - val_binary_accuracy: 0.9800\n",
            "Epoch 49/50\n",
            "55/55 [==============================] - 0s 3ms/step - loss: 0.0039 - binary_accuracy: 0.9997 - val_loss: 0.1047 - val_binary_accuracy: 0.9800\n",
            "Epoch 50/50\n",
            "55/55 [==============================] - 0s 4ms/step - loss: 0.0037 - binary_accuracy: 0.9997 - val_loss: 0.1056 - val_binary_accuracy: 0.9793\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from plotly import graph_objects as go\n",
        "\n",
        "color_chart = [\"#4B9AC7\", \"#4BE8E0\", \"#9DD4F3\", \"#97FBF6\", \"#2A7FAF\", \"#23B1AB\", \"#0E3449\", \"#015955\"]\n",
        "\n",
        "\n",
        "fig = go.Figure(data=[\n",
        "                      go.Scatter(\n",
        "                          y=history.history[\"loss\"],\n",
        "                          name=\"Training loss\",\n",
        "                          mode=\"lines\",\n",
        "                          marker=dict(\n",
        "                          color=color_chart[0]\n",
        "                          )),\n",
        "                      go.Scatter(\n",
        "                          y=history.history[\"val_loss\"],\n",
        "                          name=\"Validation loss\",\n",
        "                          mode=\"lines\",\n",
        "                          marker=dict(\n",
        "                              color=color_chart[1]\n",
        "                          ))\n",
        "])\n",
        "fig.update_layout(\n",
        "    title='Training and val loss across epochs',\n",
        "    xaxis_title='epochs',\n",
        "    yaxis_title='Cross Entropy'    \n",
        ")\n",
        "fig.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "id": "0aRmoI7G_BsV",
        "outputId": "9bde3c47-de09-4cc1-c632-200387d304f1"
      },
      "execution_count": 91,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<html>\n",
              "<head><meta charset=\"utf-8\" /></head>\n",
              "<body>\n",
              "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
              "        <script src=\"https://cdn.plot.ly/plotly-2.8.3.min.js\"></script>                <div id=\"928c4508-ec93-4d26-a4d6-6e8b0bcfa08a\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"928c4508-ec93-4d26-a4d6-6e8b0bcfa08a\")) {                    Plotly.newPlot(                        \"928c4508-ec93-4d26-a4d6-6e8b0bcfa08a\",                        [{\"marker\":{\"color\":\"#4B9AC7\"},\"mode\":\"lines\",\"name\":\"Training loss\",\"y\":[0.6592360734939575,0.5428541898727417,0.42653313279151917,0.35027047991752625,0.26782238483428955,0.1836467683315277,0.1251964271068573,0.09273720532655716,0.07445791363716125,0.06328779458999634,0.05502943694591522,0.04844886437058449,0.04266387224197388,0.038078662008047104,0.03428950160741806,0.03095153719186783,0.02822902798652649,0.02551918849349022,0.023312916979193687,0.021294616162776947,0.01950410008430481,0.01786453276872635,0.0162834320217371,0.014954930171370506,0.013666430488228798,0.01268080435693264,0.011756105348467827,0.010929387994110584,0.010278900153934956,0.009513497352600098,0.00896380189806223,0.008424763567745686,0.007840205915272236,0.00742457015439868,0.0070587703958153725,0.006721285171806812,0.006349591072648764,0.006042052060365677,0.005842477083206177,0.005492788273841143,0.005288876127451658,0.005038721952587366,0.004823390860110521,0.004663803149014711,0.004456607159227133,0.004333639517426491,0.004167974926531315,0.004024061840027571,0.0038604028522968292,0.0037151738069951534],\"type\":\"scatter\"},{\"marker\":{\"color\":\"#4BE8E0\"},\"mode\":\"lines\",\"name\":\"Validation loss\",\"y\":[0.6027665734291077,0.4707208573818207,0.37363606691360474,0.30944910645484924,0.23332826793193817,0.16992776095867157,0.13304802775382996,0.11435846239328384,0.10423056036233902,0.09836870431900024,0.09439685940742493,0.09223636984825134,0.09054248780012131,0.08957140147686005,0.08848416805267334,0.08820710331201553,0.08831673860549927,0.08844020217657089,0.08879024535417557,0.08909787237644196,0.09009549766778946,0.09075451642274857,0.09136846661567688,0.09202594310045242,0.09250194579362869,0.093178890645504,0.09406793117523193,0.094713494181633,0.09556437283754349,0.09594492614269257,0.0967988520860672,0.09722252190113068,0.09796501696109772,0.09851483255624771,0.09927971661090851,0.09969141334295273,0.09990048408508301,0.09983409941196442,0.10076002776622772,0.10106471925973892,0.10107208043336868,0.10176809877157211,0.10219351947307587,0.10253658890724182,0.10323438793420792,0.10353831946849823,0.10384508967399597,0.10448475182056427,0.10466203838586807,0.10563825815916061],\"type\":\"scatter\"}],                        {\"template\":{\"data\":{\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"choropleth\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"choropleth\"}],\"contour\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"contour\"}],\"contourcarpet\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"contourcarpet\"}],\"heatmap\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"heatmap\"}],\"heatmapgl\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"heatmapgl\"}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"histogram2d\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"histogram2d\"}],\"histogram2dcontour\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"histogram2dcontour\"}],\"mesh3d\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"mesh3d\"}],\"parcoords\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"parcoords\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}],\"scatter\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatter\"}],\"scatter3d\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatter3d\"}],\"scattercarpet\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattercarpet\"}],\"scattergeo\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattergeo\"}],\"scattergl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattergl\"}],\"scattermapbox\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattermapbox\"}],\"scatterpolar\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterpolar\"}],\"scatterpolargl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterpolargl\"}],\"scatterternary\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterternary\"}],\"surface\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"surface\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}]},\"layout\":{\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"autotypenumbers\":\"strict\",\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]],\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]},\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"geo\":{\"bgcolor\":\"white\",\"lakecolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"showlakes\":true,\"showland\":true,\"subunitcolor\":\"white\"},\"hoverlabel\":{\"align\":\"left\"},\"hovermode\":\"closest\",\"mapbox\":{\"style\":\"light\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"bgcolor\":\"#E5ECF6\",\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"ternary\":{\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"bgcolor\":\"#E5ECF6\",\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"title\":{\"x\":0.05},\"xaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"zerolinewidth\":2},\"yaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"zerolinewidth\":2}}},\"title\":{\"text\":\"Training and val loss across epochs\"},\"xaxis\":{\"title\":{\"text\":\"epochs\"}},\"yaxis\":{\"title\":{\"text\":\"Cross Entropy\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
              "                            \n",
              "var gd = document.getElementById('928c4508-ec93-4d26-a4d6-6e8b0bcfa08a');\n",
              "var x = new MutationObserver(function (mutations, observer) {{\n",
              "        var display = window.getComputedStyle(gd).display;\n",
              "        if (!display || display === 'none') {{\n",
              "            console.log([gd, 'removed!']);\n",
              "            Plotly.purge(gd);\n",
              "            observer.disconnect();\n",
              "        }}\n",
              "}});\n",
              "\n",
              "// Listen for the removal of the full notebook cells\n",
              "var notebookContainer = gd.closest('#notebook-container');\n",
              "if (notebookContainer) {{\n",
              "    x.observe(notebookContainer, {childList: true});\n",
              "}}\n",
              "\n",
              "// Listen for the clearing of the current output cell\n",
              "var outputEl = gd.closest('.output');\n",
              "if (outputEl) {{\n",
              "    x.observe(outputEl, {childList: true});\n",
              "}}\n",
              "\n",
              "                        })                };                            </script>        </div>\n",
              "</body>\n",
              "</html>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from plotly import graph_objects as go\n",
        "fig = go.Figure(data=[\n",
        "                      go.Scatter(\n",
        "                          y=history.history[\"binary_accuracy\"],\n",
        "                          name=\"Training accuracy\",\n",
        "                          mode=\"lines\",\n",
        "                          marker=dict(\n",
        "                              color=color_chart[4]\n",
        "                          )),\n",
        "                      go.Scatter(\n",
        "                          y=history.history[\"val_binary_accuracy\"],\n",
        "                          name=\"Validation accruracy\",\n",
        "                          mode=\"lines\",\n",
        "                          marker=dict(\n",
        "                              color=color_chart[5]\n",
        "                          ))\n",
        "])\n",
        "fig.update_layout(\n",
        "    title='Training and val accuracy across epochs',\n",
        "    xaxis_title='epochs',\n",
        "    yaxis_title='Accuracy'    \n",
        ")\n",
        "fig.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "id": "NISjHzqe_CrP",
        "outputId": "6b09cdd3-513c-4f2a-81df-3fb67952aac6"
      },
      "execution_count": 92,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<html>\n",
              "<head><meta charset=\"utf-8\" /></head>\n",
              "<body>\n",
              "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
              "        <script src=\"https://cdn.plot.ly/plotly-2.8.3.min.js\"></script>                <div id=\"645d28a3-7acb-4e9f-acfc-2a249277932e\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"645d28a3-7acb-4e9f-acfc-2a249277932e\")) {                    Plotly.newPlot(                        \"645d28a3-7acb-4e9f-acfc-2a249277932e\",                        [{\"marker\":{\"color\":\"#2A7FAF\"},\"mode\":\"lines\",\"name\":\"Training accuracy\",\"y\":[0.7962856888771057,0.8625714182853699,0.8625714182853699,0.8625714182853699,0.8671428561210632,0.9368571639060974,0.9702857136726379,0.9734285473823547,0.9794285893440247,0.980571448802948,0.9831428527832031,0.9857142567634583,0.9865714311599731,0.9888571500778198,0.9891428351402283,0.9908571243286133,0.9908571243286133,0.9919999837875366,0.99314284324646,0.9934285879135132,0.9940000176429749,0.994857132434845,0.9962857365608215,0.99657142162323,0.99657142162323,0.9974285960197449,0.9974285960197449,0.9980000257492065,0.9980000257492065,0.9980000257492065,0.9985714554786682,0.9985714554786682,0.9991428852081299,0.9991428852081299,0.9991428852081299,0.9994285702705383,0.9994285702705383,0.9997143149375916,0.9994285702705383,0.9997143149375916,0.9997143149375916,0.9997143149375916,0.9997143149375916,0.9997143149375916,0.9997143149375916,0.9997143149375916,0.9997143149375916,0.9997143149375916,0.9997143149375916,0.9997143149375916],\"type\":\"scatter\"},{\"marker\":{\"color\":\"#23B1AB\"},\"mode\":\"lines\",\"name\":\"Validation accruracy\",\"y\":[0.871999979019165,0.871999979019165,0.871999979019165,0.871999979019165,0.8973333239555359,0.9520000219345093,0.9626666903495789,0.9666666388511658,0.9646666646003723,0.9666666388511658,0.968666672706604,0.9693333506584167,0.972000002861023,0.9739999771118164,0.9739999771118164,0.9753333330154419,0.9760000109672546,0.9773333072662354,0.9779999852180481,0.9786666631698608,0.9786666631698608,0.9793333411216736,0.9793333411216736,0.9793333411216736,0.9793333411216736,0.9793333411216736,0.9793333411216736,0.9793333411216736,0.9800000190734863,0.9793333411216736,0.9793333411216736,0.9793333411216736,0.9793333411216736,0.9779999852180481,0.9786666631698608,0.9786666631698608,0.9786666631698608,0.9786666631698608,0.9779999852180481,0.9779999852180481,0.9786666631698608,0.9786666631698608,0.9786666631698608,0.9793333411216736,0.9793333411216736,0.9800000190734863,0.9800000190734863,0.9800000190734863,0.9800000190734863,0.9793333411216736],\"type\":\"scatter\"}],                        {\"template\":{\"data\":{\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"choropleth\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"choropleth\"}],\"contour\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"contour\"}],\"contourcarpet\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"contourcarpet\"}],\"heatmap\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"heatmap\"}],\"heatmapgl\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"heatmapgl\"}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"histogram2d\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"histogram2d\"}],\"histogram2dcontour\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"histogram2dcontour\"}],\"mesh3d\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"mesh3d\"}],\"parcoords\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"parcoords\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}],\"scatter\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatter\"}],\"scatter3d\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatter3d\"}],\"scattercarpet\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattercarpet\"}],\"scattergeo\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattergeo\"}],\"scattergl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattergl\"}],\"scattermapbox\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattermapbox\"}],\"scatterpolar\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterpolar\"}],\"scatterpolargl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterpolargl\"}],\"scatterternary\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterternary\"}],\"surface\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"surface\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}]},\"layout\":{\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"autotypenumbers\":\"strict\",\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]],\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]},\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"geo\":{\"bgcolor\":\"white\",\"lakecolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"showlakes\":true,\"showland\":true,\"subunitcolor\":\"white\"},\"hoverlabel\":{\"align\":\"left\"},\"hovermode\":\"closest\",\"mapbox\":{\"style\":\"light\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"bgcolor\":\"#E5ECF6\",\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"ternary\":{\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"bgcolor\":\"#E5ECF6\",\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"title\":{\"x\":0.05},\"xaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"zerolinewidth\":2},\"yaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"zerolinewidth\":2}}},\"title\":{\"text\":\"Training and val accuracy across epochs\"},\"xaxis\":{\"title\":{\"text\":\"epochs\"}},\"yaxis\":{\"title\":{\"text\":\"Accuracy\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
              "                            \n",
              "var gd = document.getElementById('645d28a3-7acb-4e9f-acfc-2a249277932e');\n",
              "var x = new MutationObserver(function (mutations, observer) {{\n",
              "        var display = window.getComputedStyle(gd).display;\n",
              "        if (!display || display === 'none') {{\n",
              "            console.log([gd, 'removed!']);\n",
              "            Plotly.purge(gd);\n",
              "            observer.disconnect();\n",
              "        }}\n",
              "}});\n",
              "\n",
              "// Listen for the removal of the full notebook cells\n",
              "var notebookContainer = gd.closest('#notebook-container');\n",
              "if (notebookContainer) {{\n",
              "    x.observe(notebookContainer, {childList: true});\n",
              "}}\n",
              "\n",
              "// Listen for the clearing of the current output cell\n",
              "var outputEl = gd.closest('.output');\n",
              "if (outputEl) {{\n",
              "    x.observe(outputEl, {childList: true});\n",
              "}}\n",
              "\n",
              "                        })                };                            </script>        </div>\n",
              "</body>\n",
              "</html>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Testing our Sequential Model on Detecting Spam Across New Messages"
      ],
      "metadata": {
        "id": "irF9hNT9Egwo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Use the model to predict whether a message is spam\n",
        "messages_received = ['Greg, can you call me back once you get this?',\n",
        "                'Congrats on your new iPhone! Click here to claim your prize...', \n",
        "                'Really like that new photo of you',\n",
        "                'Did you hear the news today? Terrible what has happened...',\n",
        "                'Attend this free COVID webinar today: Book your session now...']\n",
        "\n",
        "print(messages_received) \n",
        "\n",
        "# Create the sequences\n",
        "padding_type='post'\n",
        "sample_sequences = tokenizer.texts_to_sequences(messages_received)\n",
        "fakes_padded = pad_sequences(sample_sequences, padding=padding_type, maxlen=164)           \n",
        "\n",
        "fakes_prediction = model.predict(fakes_padded)\n",
        "\n",
        "# The closer the class is to 1, the more likely that the message is spam\n",
        "for x in range(len(messages_received)):\n",
        "  print(messages_received[x])\n",
        "  print(fakes_prediction[x])\n",
        "  print('\\n')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8accy52QEhwx",
        "outputId": "2f80cb36-fb76-4f3b-d2c7-f4d0a62a0e70"
      },
      "execution_count": 96,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['Greg, can you call me back once you get this?', 'Congrats on your new iPhone! Click here to claim your prize...', 'Really like that new photo of you', 'Did you hear the news today? Terrible what has happened...', 'Attend this free COVID webinar today: Book your session now...']\n",
            "1/1 [==============================] - 0s 61ms/step\n",
            "Greg, can you call me back once you get this?\n",
            "[2.9733166e-05]\n",
            "\n",
            "\n",
            "Congrats on your new iPhone! Click here to claim your prize...\n",
            "[0.9998084]\n",
            "\n",
            "\n",
            "Really like that new photo of you\n",
            "[0.00020477]\n",
            "\n",
            "\n",
            "Did you hear the news today? Terrible what has happened...\n",
            "[0.00185848]\n",
            "\n",
            "\n",
            "Attend this free COVID webinar today: Book your session now...\n",
            "[0.01462053]\n",
            "\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Transfer Learning with BERT"
      ],
      "metadata": {
        "id": "ZoHCXr3raFE0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Let's import a pretrained model "
      ],
      "metadata": {
        "id": "jABWP8z92jta"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text_test = [dataset.sms_clean[3]]\n",
        "\n",
        "preprocessor = hub.load(\"https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3\")\n",
        "bert_preprocessor = hub.KerasLayer(\"https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3\", name='preprocessing')\n",
        "bert_encoder = hub.KerasLayer('https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4')\n",
        "text_preprocessed = bert_preprocessor(text_test)\n"
      ],
      "metadata": {
        "id": "3B2_QQRMz7Wn"
      },
      "execution_count": 141,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "text_input = tf.keras.layers.Input(shape = (), dtype = tf.string, name = 'Inputs')\n",
        "preprocessed_text = bert_preprocessor(text_input)\n",
        "embeed = bert_encoder(preprocessed_text)\n",
        "dropout = tf.keras.layers.Dropout(0.1, name = 'Dropout')(embeed['pooled_output'])\n",
        "outputs = tf.keras.layers.Dense(1, activation = 'sigmoid', name = 'Dense')(dropout)"
      ],
      "metadata": {
        "id": "YVab6_30FdLP"
      },
      "execution_count": 142,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# creating final model\n",
        "model_tlbert = tf.keras.Model(inputs = [text_input], outputs = [outputs])"
      ],
      "metadata": {
        "id": "I_nERvCnFSX2"
      },
      "execution_count": 143,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sms.shape[1]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jbJ0MPnX9UCp",
        "outputId": "255eda66-6897-4275-a8e3-e12babf2df40"
      },
      "execution_count": 116,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "164"
            ]
          },
          "metadata": {},
          "execution_count": 116
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model_tlbert.summary()"
      ],
      "metadata": {
        "id": "Lvq7KF6L0RQn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bafee3a0-b84c-4104-dba7-bf18f37da634"
      },
      "execution_count": 144,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_5\"\n",
            "__________________________________________________________________________________________________\n",
            " Layer (type)                   Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            " Inputs (InputLayer)            [(None,)]            0           []                               \n",
            "                                                                                                  \n",
            " preprocessing (KerasLayer)     {'input_word_ids':   0           ['Inputs[0][0]']                 \n",
            "                                (None, 128),                                                      \n",
            "                                 'input_mask': (Non                                               \n",
            "                                e, 128),                                                          \n",
            "                                 'input_type_ids':                                                \n",
            "                                (None, 128)}                                                      \n",
            "                                                                                                  \n",
            " keras_layer_3 (KerasLayer)     {'sequence_output':  109482241   ['preprocessing[0][0]',          \n",
            "                                 (None, 128, 768),                'preprocessing[0][1]',          \n",
            "                                 'default': (None,                'preprocessing[0][2]']          \n",
            "                                768),                                                             \n",
            "                                 'pooled_output': (                                               \n",
            "                                None, 768),                                                       \n",
            "                                 'encoder_outputs':                                               \n",
            "                                 [(None, 128, 768),                                               \n",
            "                                 (None, 128, 768),                                                \n",
            "                                 (None, 128, 768),                                                \n",
            "                                 (None, 128, 768),                                                \n",
            "                                 (None, 128, 768),                                                \n",
            "                                 (None, 128, 768),                                                \n",
            "                                 (None, 128, 768),                                                \n",
            "                                 (None, 128, 768),                                                \n",
            "                                 (None, 128, 768),                                                \n",
            "                                 (None, 128, 768),                                                \n",
            "                                 (None, 128, 768),                                                \n",
            "                                 (None, 128, 768)]}                                               \n",
            "                                                                                                  \n",
            " Dropout (Dropout)              (None, 768)          0           ['keras_layer_3[0][13]']         \n",
            "                                                                                                  \n",
            " Dense (Dense)                  (None, 1)            769         ['Dropout[0][0]']                \n",
            "                                                                                                  \n",
            "==================================================================================================\n",
            "Total params: 109,483,010\n",
            "Trainable params: 769\n",
            "Non-trainable params: 109,482,241\n",
            "__________________________________________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### compile and fit model"
      ],
      "metadata": {
        "id": "AVlXf4SKcQZ0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "Metrics = [tf.keras.metrics.BinaryAccuracy(name = 'accuracy'),\n",
        "           tf.keras.metrics.Precision(name = 'precision'),\n",
        "           tf.keras.metrics.Recall(name = 'recall')\n",
        "           ]"
      ],
      "metadata": {
        "id": "kXXxd2ln6tdc"
      },
      "execution_count": 145,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_tlbert.compile(optimizer ='adam',\n",
        "               loss = 'binary_crossentropy',\n",
        "               metrics = Metrics)"
      ],
      "metadata": {
        "id": "lGSLvZaY68mW"
      },
      "execution_count": 146,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Train test split per text \n",
        "text_train, text_test, y_train, y_test = train_test_split(dataset.sms_clean, dataset.categ, test_size=0.1, random_state=1,stratify=dataset.categ)\n",
        "\n",
        "text_train, text_val, y_train, y_val = train_test_split(text_train, y_train, test_size=0.1, random_state=1,stratify=y_train)\n",
        "\n",
        "# creating tensorflow datasets slices and making batches\n",
        "\n",
        "text_train_ds = tf.data.Dataset.from_tensor_slices((text_train, y_train))\n",
        "text_test_ds = tf.data.Dataset.from_tensor_slices((text_test, y_test))\n",
        "text_val_ds = tf.data.Dataset.from_tensor_slices((text_val, y_val))\n",
        "\n",
        "text_train_ds = text_train_ds.shuffle(len(text_train_ds)).batch(64)\n",
        "text_test_ds = text_test_ds.shuffle(len(text_test_ds)).batch(64)\n",
        "text_val_ds = text_val_ds.shuffle(len(text_val_ds)).batch(64)"
      ],
      "metadata": {
        "id": "LkHtvuYaGZ1F"
      },
      "execution_count": 148,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "history_tlbert = model_tlbert.fit(text_train_ds, epochs=5, validation_data=text_val_ds)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oKQuytbC7cB3",
        "outputId": "a379b97c-7a06-413b-e364-9a4602dafddc"
      },
      "execution_count": 149,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/5\n",
            "64/64 [==============================] - 1456s 23s/step - loss: 0.4103 - accuracy: 0.8430 - precision: 0.1176 - recall: 0.0257 - val_loss: 0.3388 - val_accuracy: 0.8644 - val_precision: 0.0000e+00 - val_recall: 0.0000e+00\n",
            "Epoch 2/5\n",
            "64/64 [==============================] - 1404s 22s/step - loss: 0.3153 - accuracy: 0.8654 - precision: 0.5000 - recall: 0.0055 - val_loss: 0.2892 - val_accuracy: 0.8644 - val_precision: 0.0000e+00 - val_recall: 0.0000e+00\n",
            "Epoch 3/5\n",
            "64/64 [==============================] - 1398s 22s/step - loss: 0.2750 - accuracy: 0.8743 - precision: 0.8462 - recall: 0.0807 - val_loss: 0.2472 - val_accuracy: 0.8800 - val_precision: 1.0000 - val_recall: 0.1148\n",
            "Epoch 4/5\n",
            "64/64 [==============================] - 1399s 22s/step - loss: 0.2423 - accuracy: 0.8909 - precision: 0.9055 - recall: 0.2110 - val_loss: 0.2193 - val_accuracy: 0.8933 - val_precision: 1.0000 - val_recall: 0.2131\n",
            "Epoch 5/5\n",
            "64/64 [==============================] - 1390s 22s/step - loss: 0.2185 - accuracy: 0.9089 - precision: 0.9037 - recall: 0.3615 - val_loss: 0.2189 - val_accuracy: 0.8778 - val_precision: 1.0000 - val_recall: 0.0984\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from plotly import graph_objects as go\n",
        "fig = go.Figure(data=[\n",
        "                      go.Scatter(\n",
        "                          y=history_tlbert.history[\"accuracy\"],\n",
        "                          name=\"Training accuracy\",\n",
        "                          mode=\"lines\",\n",
        "                          marker=dict(\n",
        "                              color=color_chart[4]\n",
        "                          )),\n",
        "                      go.Scatter(\n",
        "                          y=history_tlbert.history[\"val_accuracy\"],\n",
        "                          name=\"Validation accuracy\",\n",
        "                          mode=\"lines\",\n",
        "                          marker=dict(\n",
        "                              color=color_chart[5]\n",
        "                          ))\n",
        "])\n",
        "fig.update_layout(\n",
        "    title='Training and val accuracy across epochs',\n",
        "    xaxis_title='epochs',\n",
        "    yaxis_title='Accuracy'    \n",
        ")\n",
        "fig.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 542
        },
        "id": "2UkDLC-MkoKg",
        "outputId": "af364307-d062-44c5-a0ec-af2e7f52c48d"
      },
      "execution_count": 153,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<html>\n",
              "<head><meta charset=\"utf-8\" /></head>\n",
              "<body>\n",
              "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax) {MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
              "        <script src=\"https://cdn.plot.ly/plotly-2.8.3.min.js\"></script>                <div id=\"ad23546b-bc19-4b4a-9c64-e3253db5ac64\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"ad23546b-bc19-4b4a-9c64-e3253db5ac64\")) {                    Plotly.newPlot(                        \"ad23546b-bc19-4b4a-9c64-e3253db5ac64\",                        [{\"marker\":{\"color\":\"#2A7FAF\"},\"mode\":\"lines\",\"name\":\"Training accuracy\",\"y\":[0.8429629802703857,0.86543208360672,0.8743209838867188,0.8908641934394836,0.9088888764381409],\"type\":\"scatter\"},{\"marker\":{\"color\":\"#23B1AB\"},\"mode\":\"lines\",\"name\":\"Validation accuracy\",\"y\":[0.8644444346427917,0.8644444346427917,0.8799999952316284,0.8933333158493042,0.8777777552604675],\"type\":\"scatter\"}],                        {\"template\":{\"data\":{\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"choropleth\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"choropleth\"}],\"contour\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"contour\"}],\"contourcarpet\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"contourcarpet\"}],\"heatmap\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"heatmap\"}],\"heatmapgl\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"heatmapgl\"}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"histogram2d\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"histogram2d\"}],\"histogram2dcontour\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"histogram2dcontour\"}],\"mesh3d\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"type\":\"mesh3d\"}],\"parcoords\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"parcoords\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}],\"scatter\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatter\"}],\"scatter3d\":[{\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatter3d\"}],\"scattercarpet\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattercarpet\"}],\"scattergeo\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattergeo\"}],\"scattergl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattergl\"}],\"scattermapbox\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scattermapbox\"}],\"scatterpolar\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterpolar\"}],\"scatterpolargl\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterpolargl\"}],\"scatterternary\":[{\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"type\":\"scatterternary\"}],\"surface\":[{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"type\":\"surface\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}]},\"layout\":{\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"autotypenumbers\":\"strict\",\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]],\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]},\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"geo\":{\"bgcolor\":\"white\",\"lakecolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"showlakes\":true,\"showland\":true,\"subunitcolor\":\"white\"},\"hoverlabel\":{\"align\":\"left\"},\"hovermode\":\"closest\",\"mapbox\":{\"style\":\"light\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"bgcolor\":\"#E5ECF6\",\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"gridwidth\":2,\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\"}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"ternary\":{\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"bgcolor\":\"#E5ECF6\",\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"title\":{\"x\":0.05},\"xaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"zerolinewidth\":2},\"yaxis\":{\"automargin\":true,\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"zerolinewidth\":2}}},\"title\":{\"text\":\"Training and val accuracy across epochs\"},\"xaxis\":{\"title\":{\"text\":\"epochs\"}},\"yaxis\":{\"title\":{\"text\":\"Accuracy\"}}},                        {\"responsive\": true}                    ).then(function(){\n",
              "                            \n",
              "var gd = document.getElementById('ad23546b-bc19-4b4a-9c64-e3253db5ac64');\n",
              "var x = new MutationObserver(function (mutations, observer) {{\n",
              "        var display = window.getComputedStyle(gd).display;\n",
              "        if (!display || display === 'none') {{\n",
              "            console.log([gd, 'removed!']);\n",
              "            Plotly.purge(gd);\n",
              "            observer.disconnect();\n",
              "        }}\n",
              "}});\n",
              "\n",
              "// Listen for the removal of the full notebook cells\n",
              "var notebookContainer = gd.closest('#notebook-container');\n",
              "if (notebookContainer) {{\n",
              "    x.observe(notebookContainer, {childList: true});\n",
              "}}\n",
              "\n",
              "// Listen for the clearing of the current output cell\n",
              "var outputEl = gd.closest('.output');\n",
              "if (outputEl) {{\n",
              "    x.observe(outputEl, {childList: true});\n",
              "}}\n",
              "\n",
              "                        })                };                            </script>        </div>\n",
              "</body>\n",
              "</html>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Compare performance between our models"
      ],
      "metadata": {
        "id": "Cdheum-Vkcd4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Let's see how percentage is accuracy and f1-score of each model\n"
      ],
      "metadata": {
        "id": "ZjimT4QwsiHt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def performance_model(model, X, y):\n",
        "\n",
        "    y_pred = np.round(model.predict(X))\n",
        "    accuracy = accuracy_score(y, y_pred)\n",
        "    f1 = f1_score(y, y_pred)\n",
        "  \n",
        "    model_performance = {'accuracy': accuracy,\n",
        "                          'f1-score': f1}\n",
        "  \n",
        "    return model_performance"
      ],
      "metadata": {
        "id": "Wo3xjqCCsfWj"
      },
      "execution_count": 154,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "Basic_model = performance_model(model,xval, yval)\n",
        "BERT_model = performance_model(model_tlbert,text_val, y_val)\n",
        "  \n",
        "data_results = pd.DataFrame({'Sequential Simple Model':Basic_model,\n",
        "                              'BERT-Transfer learning Model':BERT_model}).transpose()\n",
        "                             \n",
        "data_results.iloc[:,:3]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 148
        },
        "id": "9H8QFHVrjcag",
        "outputId": "39a634cf-c742-47b1-8a9d-41b02bc1f4ff"
      },
      "execution_count": 158,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "47/47 [==============================] - 0s 2ms/step\n",
            "15/15 [==============================] - 144s 9s/step\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                              accuracy  f1-score\n",
              "Sequential Simple Model       0.979333  0.916442\n",
              "BERT-Transfer learning Model  0.877778  0.179104"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-13d9ee37-d3d3-424a-b6e5-e96c38d7aeca\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>accuracy</th>\n",
              "      <th>f1-score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Sequential Simple Model</th>\n",
              "      <td>0.979333</td>\n",
              "      <td>0.916442</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>BERT-Transfer learning Model</th>\n",
              "      <td>0.877778</td>\n",
              "      <td>0.179104</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-13d9ee37-d3d3-424a-b6e5-e96c38d7aeca')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-13d9ee37-d3d3-424a-b6e5-e96c38d7aeca button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-13d9ee37-d3d3-424a-b6e5-e96c38d7aeca');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 158
        }
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3.9.12 ('base')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.12"
    },
    "vscode": {
      "interpreter": {
        "hash": "16196ea7eff63910081d4e10ae1bdb1eb18fd83cb470bb8efbb9fa6b0c724af5"
      }
    },
    "colab": {
      "provenance": [],
      "toc_visible": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}