English_To_Telugu_Encoder_Decoder.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xLj_6a1Eh8RW"
   },
   "source": [
    "# Neural Machine Translation\n",
    "\n",
    "Machine translation is a challenging task that traditionally involves large statistical models developed using highly sophisticated linguistic knowledge.\n",
    "\n",
    "Neural machine translation is the use of deep neural networks for the problem of machine translation.\n",
    "\n",
    "In this Notebook, you will discover how to develop a neural machine translation system for translating English phrases to Telugu.\n",
    "\n",
    "Steps:\n",
    "\n",
    "* Prepare Data\n",
    "* Preprocess Data\n",
    "* Text to Sequence Conversion\n",
    "* Build Encoder-Decoder Model\n",
    "* Train Model\n",
    "* Translate English to Telugu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7jfOypdih_OW"
   },
   "source": [
    "In this Notebook, The [English-Telugu](https://github.com/scionoftech/English_Telugu_Bilingual-Sentence-Pairs) Bilingual-Sentence-Pairs data is used for Neural Machine Translation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QdzTBuL2adyL"
   },
   "outputs": [],
   "source": [
    "project_path = \"machine_translation/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 63
    },
    "colab_type": "code",
    "id": "Rgvg66I4apty",
    "outputId": "37cff2db-ed57-48d3-e3b6-c71e24952638"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p style=\"color: red;\">\n",
       "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n",
       "We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n",
       "or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n",
       "<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import string\n",
    "from string import digits\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import re\n",
    "from sklearn.utils import shuffle\n",
    "from sklearn.model_selection import train_test_split\n",
    "from tensorflow.keras.layers import Input, LSTM, Embedding, Dense\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint,EarlyStopping,ReduceLROnPlateau"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eL-N73JViCJf"
   },
   "source": [
    "## Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iv8sRE556QfV"
   },
   "outputs": [],
   "source": [
    "# read phrases from english_telugu_data.txt file\n",
    "english_sentances = []\n",
    "telugu_sentances = []\n",
    "with open(project_path+\"english_telugu_data.txt\", mode='rt', encoding='utf-8') as fp:\n",
    "    for line in fp.readlines():\n",
    "        eng_tel = line.split(\"++++$++++\")\n",
    "        english_sentances.append(eng_tel[0])\n",
    "        telugu_sentances.append(eng_tel[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DyHnelVf6SPZ"
   },
   "outputs": [],
   "source": [
    "data = pd.DataFrame({\"english_sentances\":english_sentances,\"telugu_sentances\":telugu_sentances})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "m-Psopq3bHMh",
    "outputId": "d78f125b-ef16-4b9b-8469-f1752c4b9545"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>english_sentances</th>\n",
       "      <th>telugu_sentances</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>His legs are long.</td>\n",
       "      <td>అతని కాళ్ళు పొడవుగా ఉన్నాయి.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Who taught Tom how to speak French?</td>\n",
       "      <td>టామ్ ఫ్రెంచ్ మాట్లాడటం ఎలా నేర్పించారు?\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>I swim in the sea every day.</td>\n",
       "      <td>నేను ప్రతి రోజు సముద్రంలో ఈత కొడతాను.\\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Tom popped into the supermarket on his way hom...</td>\n",
       "      <td>టామ్ కొంచెం పాలు కొనడానికి ఇంటికి వెళ్ళేటప్పుడ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Smoke filled the room.</td>\n",
       "      <td>పొగ గదిని నింపింది.\\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   english_sentances                                   telugu_sentances\n",
       "0                                 His legs are long.                     అతని కాళ్ళు పొడవుగా ఉన్నాయి.\\n\n",
       "1                Who taught Tom how to speak French?          టామ్ ఫ్రెంచ్ మాట్లాడటం ఎలా నేర్పించారు?\\n\n",
       "2                       I swim in the sea every day.            నేను ప్రతి రోజు సముద్రంలో ఈత కొడతాను.\\n\n",
       "3  Tom popped into the supermarket on his way hom...  టామ్ కొంచెం పాలు కొనడానికి ఇంటికి వెళ్ళేటప్పుడ...\n",
       "4                             Smoke filled the room.                              పొగ గదిని నింపింది.\\n"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "_91YZHuCbI9J",
    "outputId": "9622c7b8-7c36-4500-f6ad-c6821938f30d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(155798, 2)"
      ]
     },
     "execution_count": 7,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AdriXixpbjuD"
   },
   "source": [
    "## Text Pre-Processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gOyqb9FKcjbc"
   },
   "outputs": [],
   "source": [
    "contraction_mapping = {\"ain't\": \"is not\", \"aren't\": \"are not\",\"can't\": \"cannot\", \"'cause\": \"because\", \"could've\": \"could have\", \"couldn't\": \"could not\",\n",
    "                           \"didn't\": \"did not\",  \"doesn't\": \"does not\", \"don't\": \"do not\", \"hadn't\": \"had not\", \"hasn't\": \"has not\", \"haven't\": \"have not\",\n",
    "                           \"he'd\": \"he would\",\"he'll\": \"he will\", \"he's\": \"he is\", \"how'd\": \"how did\", \"how'd'y\": \"how do you\", \"how'll\": \"how will\", \"how's\": \"how is\",\n",
    "                           \"I'd\": \"I would\", \"I'd've\": \"I would have\", \"I'll\": \"I will\", \"I'll've\": \"I will have\",\"I'm\": \"I am\", \"I've\": \"I have\", \"i'd\": \"i would\",\n",
    "                           \"i'd've\": \"i would have\", \"i'll\": \"i will\",  \"i'll've\": \"i will have\",\"i'm\": \"i am\", \"i've\": \"i have\", \"isn't\": \"is not\", \"it'd\": \"it would\",\n",
    "                           \"it'd've\": \"it would have\", \"it'll\": \"it will\", \"it'll've\": \"it will have\",\"it's\": \"it is\", \"let's\": \"let us\", \"ma'am\": \"madam\",\n",
    "                           \"mayn't\": \"may not\", \"might've\": \"might have\",\"mightn't\": \"might not\",\"mightn't've\": \"might not have\", \"must've\": \"must have\",\n",
    "                           \"mustn't\": \"must not\", \"mustn't've\": \"must not have\", \"needn't\": \"need not\", \"needn't've\": \"need not have\",\"o'clock\": \"of the clock\",\n",
    "                           \"oughtn't\": \"ought not\", \"oughtn't've\": \"ought not have\", \"shan't\": \"shall not\", \"sha'n't\": \"shall not\", \"shan't've\": \"shall not have\",\n",
    "                           \"she'd\": \"she would\", \"she'd've\": \"she would have\", \"she'll\": \"she will\", \"she'll've\": \"she will have\", \"she's\": \"she is\",\n",
    "                           \"should've\": \"should have\", \"shouldn't\": \"should not\", \"shouldn't've\": \"should not have\", \"so've\": \"so have\",\"so's\": \"so as\",\n",
    "                           \"this's\": \"this is\",\"that'd\": \"that would\", \"that'd've\": \"that would have\", \"that's\": \"that is\", \"there'd\": \"there would\",\n",
    "                           \"there'd've\": \"there would have\", \"there's\": \"there is\", \"here's\": \"here is\",\"they'd\": \"they would\", \"they'd've\": \"they would have\",\n",
    "                           \"they'll\": \"they will\", \"they'll've\": \"they will have\", \"they're\": \"they are\", \"they've\": \"they have\", \"to've\": \"to have\",\n",
    "                           \"wasn't\": \"was not\", \"we'd\": \"we would\", \"we'd've\": \"we would have\", \"we'll\": \"we will\", \"we'll've\": \"we will have\", \"we're\": \"we are\",\n",
    "                           \"we've\": \"we have\", \"weren't\": \"were not\", \"what'll\": \"what will\", \"what'll've\": \"what will have\", \"what're\": \"what are\",\n",
    "                           \"what's\": \"what is\", \"what've\": \"what have\", \"when's\": \"when is\", \"when've\": \"when have\", \"where'd\": \"where did\", \"where's\": \"where is\",\n",
    "                           \"where've\": \"where have\", \"who'll\": \"who will\", \"who'll've\": \"who will have\", \"who's\": \"who is\", \"who've\": \"who have\",\n",
    "                           \"why's\": \"why is\", \"why've\": \"why have\", \"will've\": \"will have\", \"won't\": \"will not\", \"won't've\": \"will not have\",\n",
    "                           \"would've\": \"would have\", \"wouldn't\": \"would not\", \"wouldn't've\": \"would not have\", \"y'all\": \"you all\",\n",
    "                           \"y'all'd\": \"you all would\",\"y'all'd've\": \"you all would have\",\"y'all're\": \"you all are\",\"y'all've\": \"you all have\",\n",
    "                           \"you'd\": \"you would\", \"you'd've\": \"you would have\", \"you'll\": \"you will\", \"you'll've\": \"you will have\",\n",
    "                           \"you're\": \"you are\", \"you've\": \"you have\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "i3xbjiJtbQeK"
   },
   "outputs": [],
   "source": [
    "# clean english sentances\n",
    "def clean_eng(text):\n",
    "    # Lowercase all characters\n",
    "    text = text.lower()\n",
    "    # map contractions\n",
    "    text = ' '.join([contraction_mapping[w] if w in contraction_mapping else w for w in text.split(\" \")])\n",
    "    # Remove quotes\n",
    "    text = re.sub(\"'\", '', text)\n",
    "    # Remove all the special characters\n",
    "    exclude = set(string.punctuation) # Set of all special characters\n",
    "    text = ''.join([c for c in text if c not in exclude])\n",
    "    # Remove all numbers from text\n",
    "    remove_digits = str.maketrans('', '', digits)\n",
    "    text = text.translate(remove_digits)\n",
    "    # Remove extra spaces\n",
    "    text= text.strip()\n",
    "\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Fbhk3FuMe-3J"
   },
   "outputs": [],
   "source": [
    "# clean telugu sentances\n",
    "def clean_tel(text):\n",
    "    # Lowercase all characters\n",
    "    text = text.lower()\n",
    "    # Remove quotes\n",
    "    text = re.sub(\"'\", '', text)\n",
    "    # Remove all the special characters\n",
    "    exclude = set(string.punctuation) # Set of all special characters\n",
    "    text = ''.join([c for c in text if c not in exclude])\n",
    "    # Remove all numbers from text\n",
    "    remove_digits = str.maketrans('', '', digits)\n",
    "    text = text.translate(remove_digits)\n",
    "    # Remove Telugu numbers from text\n",
    "    text = re.sub(\"[౦౧౨౩౪౫౬౭౮౯]\", '', text)\n",
    "    # Remove extra spaces\n",
    "    text= text.strip()\n",
    "    text = 'START_ '+ text + ' _END'\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LLj0iaLEheFS"
   },
   "outputs": [],
   "source": [
    "# clean text\n",
    "data_df = data.copy()\n",
    "data_df[\"english_sentances\"] = data_df[\"english_sentances\"] .apply(lambda x: clean_eng(x))\n",
    "data_df[\"telugu_sentances\"] = data_df[\"telugu_sentances\"] .apply(lambda x: clean_tel(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MIU-GmMHq3rV"
   },
   "outputs": [],
   "source": [
    "# Vocabulary of English\n",
    "all_eng_words=set()\n",
    "for eng in data_df.english_sentances:\n",
    "    for word in eng.split():\n",
    "        if word not in all_eng_words:\n",
    "            all_eng_words.add(word)\n",
    "\n",
    "# Vocabulary of French \n",
    "all_telugu_words=set()\n",
    "for tel in data_df.telugu_sentances:\n",
    "    for word in tel.split():\n",
    "        if word not in all_telugu_words:\n",
    "            all_telugu_words.add(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "xTWqCwNJ5hne",
    "outputId": "0a66807e-1bde-4472-84f8-6276753d8e5d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "101"
      ]
     },
     "execution_count": 13,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Max Length of source sequence\n",
    "lenght_list=[]\n",
    "for l in data_df.english_sentances:\n",
    "    lenght_list.append(len(l.split(' ')))\n",
    "max_length_src = np.max(lenght_list)\n",
    "max_length_src"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "SOfsJRLK5mSV",
    "outputId": "936918b0-821d-4808-f90e-2ee18c773437"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": 14,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Max Length of target sequence\n",
    "lenght_list=[]\n",
    "for l in data_df.telugu_sentances:\n",
    "    lenght_list.append(len(l.split(' ')))\n",
    "max_length_tar = np.max(lenght_list)\n",
    "max_length_tar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "S7enCaIZ5sgd",
    "outputId": "4ddd4a02-c6aa-4b0e-98c3-920122116656"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(13909, 38724)"
      ]
     },
     "execution_count": 15,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_words = sorted(list(all_eng_words))\n",
    "target_words = sorted(list(all_telugu_words))\n",
    "num_encoder_tokens = len(all_eng_words)\n",
    "num_decoder_tokens = len(all_telugu_words)\n",
    "num_encoder_tokens, num_decoder_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "HtXR5yOx5sdl",
    "outputId": "5ad58477-aec3-4ff6-fe71-cfd9567aa2ec"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "38725"
      ]
     },
     "execution_count": 16,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_decoder_tokens += 1 # For zero padding\n",
    "num_decoder_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BG1F6rP_5sa1"
   },
   "outputs": [],
   "source": [
    "input_token_index = dict([(word, i+1) for i, word in enumerate(input_words)])\n",
    "target_token_index = dict([(word, i+1) for i, word in enumerate(target_words)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "O9E1MWLu5-70"
   },
   "outputs": [],
   "source": [
    "reverse_input_char_index = dict((i, word) for word, i in input_token_index.items())\n",
    "reverse_target_char_index = dict((i, word) for word, i in target_token_index.items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 359
    },
    "colab_type": "code",
    "id": "F5aO68Of5-5P",
    "outputId": "24d069e6-cb40-4432-aaa4-fb80a6442841"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>english_sentances</th>\n",
       "      <th>telugu_sentances</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>125402</th>\n",
       "      <td>we are not cowards</td>\n",
       "      <td>START_ మేము పిరికివాళ్ళు కాదు _END</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>133209</th>\n",
       "      <td>she decided to marry him even though her paren...</td>\n",
       "      <td>START_ ఆమె తల్లిదండ్రులు ఆమెను కోరుకోనప్పటికీ ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>153771</th>\n",
       "      <td>this is so cool</td>\n",
       "      <td>START_ ఇది చాలా బాగుంది _END</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148623</th>\n",
       "      <td>she is not a good person</td>\n",
       "      <td>START_ ఆమె మంచి వ్యక్తి కాదు _END</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54287</th>\n",
       "      <td>tom will not catch us</td>\n",
       "      <td>START_ టామ్ మమ్మల్ని పట్టుకోడు _END</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18169</th>\n",
       "      <td>give me the coordinates</td>\n",
       "      <td>START_ నాకు కోఆర్డినేట్లు ఇవ్వండి _END</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119299</th>\n",
       "      <td>your plan failed</td>\n",
       "      <td>START_ మీ ప్రణాళిక విఫలమైంది _END</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61905</th>\n",
       "      <td>there were candles everywhere</td>\n",
       "      <td>START_ ప్రతిచోటా కొవ్వొత్తులు ఉన్నాయి _END</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74122</th>\n",
       "      <td>i assure you tom will be perfectly safe</td>\n",
       "      <td>START_ టామ్ సంపూర్ణంగా సురక్షితంగా ఉంటాడని నేన...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60761</th>\n",
       "      <td>it looks like tom is having fun</td>\n",
       "      <td>START_ టామ్ సరదాగా ఉన్నట్లు కనిపిస్తోంది _END</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        english_sentances                                   telugu_sentances\n",
       "125402                                 we are not cowards                 START_ మేము పిరికివాళ్ళు కాదు _END\n",
       "133209  she decided to marry him even though her paren...  START_ ఆమె తల్లిదండ్రులు ఆమెను కోరుకోనప్పటికీ ...\n",
       "153771                                    this is so cool                       START_ ఇది చాలా బాగుంది _END\n",
       "148623                           she is not a good person                  START_ ఆమె మంచి వ్యక్తి కాదు _END\n",
       "54287                               tom will not catch us                START_ టామ్ మమ్మల్ని పట్టుకోడు _END\n",
       "18169                             give me the coordinates             START_ నాకు కోఆర్డినేట్లు ఇవ్వండి _END\n",
       "119299                                   your plan failed                  START_ మీ ప్రణాళిక విఫలమైంది _END\n",
       "61905                       there were candles everywhere         START_ ప్రతిచోటా కొవ్వొత్తులు ఉన్నాయి _END\n",
       "74122             i assure you tom will be perfectly safe  START_ టామ్ సంపూర్ణంగా సురక్షితంగా ఉంటాడని నేన...\n",
       "60761                     it looks like tom is having fun      START_ టామ్ సరదాగా ఉన్నట్లు కనిపిస్తోంది _END"
      ]
     },
     "execution_count": 19,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_df = shuffle(data_df)\n",
    "data_df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "2BSw8mbF5-2W",
    "outputId": "b6eb617c-7732-4b9a-9d4e-0d5f69717ac0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((140218,), (15580,))"
      ]
     },
     "execution_count": 20,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train - Test Split\n",
    "X, y = data_df.english_sentances, data_df.telugu_sentances\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.1)\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qobghoFY6P2q"
   },
   "source": [
    "Save the train and test dataframes for reproducing the results later, as they are shuffled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qpx-Y5OD5-yF"
   },
   "outputs": [],
   "source": [
    "X_train.to_pickle(project_path+'X_train.pkl')\n",
    "X_test.to_pickle(project_path+'X_test.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ABn5DINz6M1G"
   },
   "outputs": [],
   "source": [
    "def generate_batch(X = X_train, y = y_train, batch_size = 128):\n",
    "    ''' Generate a batch of data '''\n",
    "    while True:\n",
    "        for j in range(0, len(X), batch_size):\n",
    "            encoder_input_data = np.zeros((batch_size, max_length_src),dtype='float32')\n",
    "            decoder_input_data = np.zeros((batch_size, max_length_tar),dtype='float32')\n",
    "            decoder_target_data = np.zeros((batch_size, max_length_tar, num_decoder_tokens),dtype='float32')\n",
    "            for i, (input_text, target_text) in enumerate(zip(X[j:j+batch_size], y[j:j+batch_size])):\n",
    "                for t, word in enumerate(input_text.split()):\n",
    "                    encoder_input_data[i, t] = input_token_index[word] # encoder input seq\n",
    "                for t, word in enumerate(target_text.split()):\n",
    "                    if t<len(target_text.split())-1:\n",
    "                        decoder_input_data[i, t] = target_token_index[word] # decoder input seq\n",
    "                    if t>0:\n",
    "                        # decoder target sequence (one hot encoded)\n",
    "                        # does not include the START_ token\n",
    "                        # Offset by one timestep\n",
    "                        decoder_target_data[i, t - 1, target_token_index[word]] = 1.\n",
    "            yield([encoder_input_data, decoder_input_data], decoder_target_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VZWlQNc86h0c"
   },
   "source": [
    "## Encoder - Decoder Model Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![encoder_decoder.png](encoder_decoder.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wybGr8xg6Mys"
   },
   "outputs": [],
   "source": [
    "latent_dim = 50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 190
    },
    "colab_type": "code",
    "id": "EulUVpRU6MwF",
    "outputId": "11df514f-476b-460b-a837-ac5c428d5897"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/backend.py:3994: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n"
     ]
    }
   ],
   "source": [
    "# Encoder\n",
    "encoder_inputs = Input(shape=(None,))\n",
    "enc_emb =  Embedding(num_encoder_tokens, latent_dim, mask_zero = True)(encoder_inputs)\n",
    "encoder_lstm = LSTM(latent_dim, return_state=True)\n",
    "encoder_outputs, state_h, state_c = encoder_lstm(enc_emb)\n",
    "# We discard `encoder_outputs` and only keep the states.\n",
    "encoder_states = [state_h, state_c]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BcmuXUZR6ops"
   },
   "outputs": [],
   "source": [
    "# Set up the decoder, using `encoder_states` as initial state.\n",
    "decoder_inputs = Input(shape=(None,))\n",
    "dec_emb_layer = Embedding(num_decoder_tokens, latent_dim, mask_zero = True)\n",
    "dec_emb = dec_emb_layer(decoder_inputs)\n",
    "# We set up our decoder to return full output sequences,\n",
    "# and to return internal states as well. We don't use the\n",
    "# return states in the training model, but we will use them in inference.\n",
    "decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)\n",
    "decoder_outputs, _, _ = decoder_lstm(dec_emb,\n",
    "                                     initial_state=encoder_states)\n",
    "decoder_dense = Dense(num_decoder_tokens, activation='softmax')\n",
    "decoder_outputs = decoder_dense(decoder_outputs)\n",
    "\n",
    "# Define the model that will turn\n",
    "# `encoder_input_data` & `decoder_input_data` into `decoder_target_data`\n",
    "model = Model([encoder_inputs, decoder_inputs], decoder_outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "A3oNMBCG6onE"
   },
   "outputs": [],
   "source": [
    "# compile the model\n",
    "model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dGVnwIIK6okU"
   },
   "outputs": [],
   "source": [
    "# from IPython.display import Image\n",
    "# Image(retina=True, filename='train_model.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UCnb0Xn36wfF"
   },
   "outputs": [],
   "source": [
    "batch_size = 128\n",
    "epochs = 30\n",
    "train_samples_steps = len(X_train) // batch_size\n",
    "val_samples_steps = len(X_test) // batch_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n-Ogd3Mb9u33"
   },
   "outputs": [],
   "source": [
    "# generate train and test datra\n",
    "train_gen = generate_batch(X_train, y_train, batch_size = batch_size)\n",
    "test_gen = generate_batch(X_test, y_test, batch_size = batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4mfdmxuxDSWT"
   },
   "outputs": [],
   "source": [
    "# Defining a helper function to save the model after each epoch \n",
    "# in which the loss decreases \n",
    "filepath = project_path+'NMT_model_enc_dec.h5'\n",
    "checkpoint = ModelCheckpoint(filepath, monitor='val_loss', verbose=1, save_best_only=True, mode='min')\n",
    "# Defining a helper function to reduce the learning rate each time \n",
    "# the learning plateaus \n",
    "reduce_alpha = ReduceLROnPlateau(monitor ='val_loss', factor = 0.2,patience = 1, min_lr = 0.001)\n",
    "# stop traning if there increase in loss\n",
    "es = EarlyStopping(monitor='val_loss', mode='min', verbose=1,patience=2)\n",
    "callbacks = [checkpoint, es, reduce_alpha] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "yjJCH_Pc6ymU",
    "outputId": "7633e4ca-6887-4c41-c069-859d8c7611cb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 1.1126 - acc: 0.2418Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:10 - loss: 0.9905 - acc: 0.2983\n",
      "Epoch 00001: val_loss improved from inf to 0.99052, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 692s 632ms/step - loss: 1.1125 - acc: 0.2418 - val_loss: 0.9905 - val_acc: 0.2983\n",
      "Epoch 2/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.9380 - acc: 0.3343Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:07 - loss: 0.8931 - acc: 0.3642\n",
      "Epoch 00002: val_loss improved from 0.99052 to 0.89315, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 690s 630ms/step - loss: 0.9379 - acc: 0.3343 - val_loss: 0.8931 - val_acc: 0.3642\n",
      "Epoch 3/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.8593 - acc: 0.3819Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:04 - loss: 0.8322 - acc: 0.4001\n",
      "Epoch 00003: val_loss improved from 0.89315 to 0.83222, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 688s 628ms/step - loss: 0.8593 - acc: 0.3819 - val_loss: 0.8322 - val_acc: 0.4001\n",
      "Epoch 4/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.8009 - acc: 0.4194Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:06 - loss: 0.7822 - acc: 0.4357\n",
      "Epoch 00004: val_loss improved from 0.83222 to 0.78216, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 687s 628ms/step - loss: 0.8009 - acc: 0.4194 - val_loss: 0.7822 - val_acc: 0.4357\n",
      "Epoch 5/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.7560 - acc: 0.4530Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:05 - loss: 0.7484 - acc: 0.4635\n",
      "Epoch 00005: val_loss improved from 0.78216 to 0.74836, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 687s 628ms/step - loss: 0.7559 - acc: 0.4531 - val_loss: 0.7484 - val_acc: 0.4635\n",
      "Epoch 6/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.7213 - acc: 0.4789Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:06 - loss: 0.7199 - acc: 0.4848\n",
      "Epoch 00006: val_loss improved from 0.74836 to 0.71990, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 686s 627ms/step - loss: 0.7213 - acc: 0.4789 - val_loss: 0.7199 - val_acc: 0.4848\n",
      "Epoch 7/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.6923 - acc: 0.5006Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:05 - loss: 0.6995 - acc: 0.5014\n",
      "Epoch 00007: val_loss improved from 0.71990 to 0.69951, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 686s 627ms/step - loss: 0.6923 - acc: 0.5006 - val_loss: 0.6995 - val_acc: 0.5014\n",
      "Epoch 8/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.6684 - acc: 0.5186Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:06 - loss: 0.6822 - acc: 0.5168\n",
      "Epoch 00008: val_loss improved from 0.69951 to 0.68217, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 688s 629ms/step - loss: 0.6685 - acc: 0.5186 - val_loss: 0.6822 - val_acc: 0.5168\n",
      "Epoch 9/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.6532 - acc: 0.5346Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:07 - loss: 0.6732 - acc: 0.5287\n",
      "Epoch 00009: val_loss improved from 0.68217 to 0.67316, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 688s 628ms/step - loss: 0.6533 - acc: 0.5346 - val_loss: 0.6732 - val_acc: 0.5287\n",
      "Epoch 10/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.6405 - acc: 0.5471Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:05 - loss: 0.6625 - acc: 0.5357\n",
      "Epoch 00010: val_loss improved from 0.67316 to 0.66246, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 687s 627ms/step - loss: 0.6404 - acc: 0.5471 - val_loss: 0.6625 - val_acc: 0.5357\n",
      "Epoch 11/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.6277 - acc: 0.5576Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:04 - loss: 0.6532 - acc: 0.5444\n",
      "Epoch 00011: val_loss improved from 0.66246 to 0.65321, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 685s 626ms/step - loss: 0.6277 - acc: 0.5577 - val_loss: 0.6532 - val_acc: 0.5444\n",
      "Epoch 12/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.6159 - acc: 0.5671Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:05 - loss: 0.6451 - acc: 0.5494\n",
      "Epoch 00012: val_loss improved from 0.65321 to 0.64514, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 686s 627ms/step - loss: 0.6158 - acc: 0.5671 - val_loss: 0.6451 - val_acc: 0.5494\n",
      "Epoch 13/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.6033 - acc: 0.5761Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:05 - loss: 0.6372 - acc: 0.5560\n",
      "Epoch 00013: val_loss improved from 0.64514 to 0.63725, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 686s 627ms/step - loss: 0.6033 - acc: 0.5761 - val_loss: 0.6372 - val_acc: 0.5560\n",
      "Epoch 14/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5951 - acc: 0.5842Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:08 - loss: 0.6322 - acc: 0.5604\n",
      "Epoch 00014: val_loss improved from 0.63725 to 0.63222, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 687s 628ms/step - loss: 0.5952 - acc: 0.5842 - val_loss: 0.6322 - val_acc: 0.5604\n",
      "Epoch 15/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5886 - acc: 0.5907Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:06 - loss: 0.6284 - acc: 0.5642\n",
      "Epoch 00015: val_loss improved from 0.63222 to 0.62835, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 689s 629ms/step - loss: 0.5885 - acc: 0.5907 - val_loss: 0.6284 - val_acc: 0.5642\n",
      "Epoch 16/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5817 - acc: 0.5966Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:06 - loss: 0.6244 - acc: 0.5676\n",
      "Epoch 00016: val_loss improved from 0.62835 to 0.62440, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 689s 629ms/step - loss: 0.5818 - acc: 0.5966 - val_loss: 0.6244 - val_acc: 0.5676\n",
      "Epoch 17/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5751 - acc: 0.6018Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:06 - loss: 0.6207 - acc: 0.5694\n",
      "Epoch 00017: val_loss improved from 0.62440 to 0.62072, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 689s 629ms/step - loss: 0.5751 - acc: 0.6018 - val_loss: 0.6207 - val_acc: 0.5694\n",
      "Epoch 18/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5692 - acc: 0.6063Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:06 - loss: 0.6190 - acc: 0.5722\n",
      "Epoch 00018: val_loss improved from 0.62072 to 0.61898, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 689s 629ms/step - loss: 0.5692 - acc: 0.6063 - val_loss: 0.6190 - val_acc: 0.5722\n",
      "Epoch 19/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5639 - acc: 0.6103Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:07 - loss: 0.6153 - acc: 0.5732\n",
      "Epoch 00019: val_loss improved from 0.61898 to 0.61529, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 689s 630ms/step - loss: 0.5640 - acc: 0.6102 - val_loss: 0.6153 - val_acc: 0.5732\n",
      "Epoch 20/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5594 - acc: 0.6140Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:09 - loss: 0.6133 - acc: 0.5753\n",
      "Epoch 00020: val_loss improved from 0.61529 to 0.61334, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 694s 634ms/step - loss: 0.5594 - acc: 0.6139 - val_loss: 0.6133 - val_acc: 0.5753\n",
      "Epoch 21/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5555 - acc: 0.6168Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:07 - loss: 0.6105 - acc: 0.5772\n",
      "Epoch 00021: val_loss improved from 0.61334 to 0.61053, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 693s 632ms/step - loss: 0.5555 - acc: 0.6168 - val_loss: 0.6105 - val_acc: 0.5772\n",
      "Epoch 22/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5521 - acc: 0.6196Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:06 - loss: 0.6096 - acc: 0.5776\n",
      "Epoch 00022: val_loss improved from 0.61053 to 0.60960, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 690s 630ms/step - loss: 0.5521 - acc: 0.6196 - val_loss: 0.6096 - val_acc: 0.5776\n",
      "Epoch 23/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5489 - acc: 0.6221Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:10 - loss: 0.6084 - acc: 0.5777\n",
      "Epoch 00023: val_loss improved from 0.60960 to 0.60837, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 692s 632ms/step - loss: 0.5489 - acc: 0.6221 - val_loss: 0.6084 - val_acc: 0.5777\n",
      "Epoch 24/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5462 - acc: 0.6242Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:08 - loss: 0.6071 - acc: 0.5798\n",
      "Epoch 00024: val_loss improved from 0.60837 to 0.60707, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 695s 634ms/step - loss: 0.5461 - acc: 0.6242 - val_loss: 0.6071 - val_acc: 0.5798\n",
      "Epoch 25/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5435 - acc: 0.6263Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:07 - loss: 0.6065 - acc: 0.5805\n",
      "Epoch 00025: val_loss improved from 0.60707 to 0.60655, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 692s 632ms/step - loss: 0.5435 - acc: 0.6263 - val_loss: 0.6065 - val_acc: 0.5805\n",
      "Epoch 26/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5408 - acc: 0.6283Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:06 - loss: 0.6058 - acc: 0.5812\n",
      "Epoch 00026: val_loss improved from 0.60655 to 0.60578, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 689s 629ms/step - loss: 0.5409 - acc: 0.6283 - val_loss: 0.6058 - val_acc: 0.5812\n",
      "Epoch 27/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5384 - acc: 0.6302Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:05 - loss: 0.6039 - acc: 0.5808\n",
      "Epoch 00027: val_loss improved from 0.60578 to 0.60393, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 687s 628ms/step - loss: 0.5384 - acc: 0.6302 - val_loss: 0.6039 - val_acc: 0.5808\n",
      "Epoch 28/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5360 - acc: 0.6319Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:05 - loss: 0.6029 - acc: 0.5807\n",
      "Epoch 00028: val_loss improved from 0.60393 to 0.60294, saving model to /content/drive/My Drive/DLCP/openwork/machine_translation/NMT_model_enc_dec.h5\n",
      "1095/1095 [==============================] - 688s 628ms/step - loss: 0.5361 - acc: 0.6319 - val_loss: 0.6029 - val_acc: 0.5807\n",
      "Epoch 29/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5339 - acc: 0.6334Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:10 - loss: 0.6035 - acc: 0.5806\n",
      "Epoch 00029: val_loss did not improve from 0.60294\n",
      "1095/1095 [==============================] - 687s 627ms/step - loss: 0.5338 - acc: 0.6334 - val_loss: 0.6035 - val_acc: 0.5806\n",
      "Epoch 30/30\n",
      "1094/1095 [============================>.] - ETA: 0s - loss: 0.5319 - acc: 0.6350Epoch 1/30\n",
      " 121/1095 [==>...........................] - ETA: 6:06 - loss: 0.6030 - acc: 0.5809\n",
      "Epoch 00030: val_loss did not improve from 0.60294\n",
      "1095/1095 [==============================] - 687s 628ms/step - loss: 0.5319 - acc: 0.6351 - val_loss: 0.6030 - val_acc: 0.5809\n",
      "Epoch 00030: early stopping\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f1d0cf76f98>"
      ]
     },
     "execution_count": 31,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train the model\n",
    "model.fit_generator(generator = train_gen,\n",
    "                    steps_per_epoch = train_samples_steps,\n",
    "                    epochs=epochs,\n",
    "                    validation_data = test_gen,\n",
    "                    validation_steps = val_samples_steps,callbacks = callbacks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o8tqLkCp6_NN"
   },
   "source": [
    "## Inference Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1RCtJwpx6yZM"
   },
   "outputs": [],
   "source": [
    "# Encode the input sequence to get the \"thought vectors\"\n",
    "encoder_model = Model(encoder_inputs, encoder_states)\n",
    "\n",
    "# Decoder setup\n",
    "# Below tensors will hold the states of the previous time step\n",
    "decoder_state_input_h = Input(shape=(latent_dim,))\n",
    "decoder_state_input_c = Input(shape=(latent_dim,))\n",
    "decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]\n",
    "\n",
    "dec_emb2= dec_emb_layer(decoder_inputs) # Get the embeddings of the decoder sequence\n",
    "\n",
    "# To predict the next word in the sequence, set the initial states to the states from the previous time step\n",
    "decoder_outputs2, state_h2, state_c2 = decoder_lstm(dec_emb2, initial_state=decoder_states_inputs)\n",
    "decoder_states2 = [state_h2, state_c2]\n",
    "decoder_outputs2 = decoder_dense(decoder_outputs2) # A dense softmax layer to generate prob dist. over the target vocabulary\n",
    "\n",
    "# Final decoder model\n",
    "decoder_model = Model(\n",
    "    [decoder_inputs] + decoder_states_inputs,\n",
    "    [decoder_outputs2] + decoder_states2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZwrLbY6P7Hqd"
   },
   "source": [
    "## Decode sample sequeces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GOpcegyx7CWt"
   },
   "outputs": [],
   "source": [
    "def decode_sequence(input_seq):\n",
    "    # Encode the input as state vectors.\n",
    "    states_value = encoder_model.predict(input_seq)\n",
    "    # Generate empty target sequence of length 1.\n",
    "    target_seq = np.zeros((1,1))\n",
    "    # Populate the first character of target sequence with the start character.\n",
    "    target_seq[0, 0] = target_token_index['START_']\n",
    "\n",
    "    # Sampling loop for a batch of sequences\n",
    "    # (to simplify, here we assume a batch of size 1).\n",
    "    stop_condition = False\n",
    "    decoded_sentence = ''\n",
    "    while not stop_condition:\n",
    "        output_tokens, h, c = decoder_model.predict([target_seq] + states_value)\n",
    "\n",
    "        # Sample a token\n",
    "        sampled_token_index = np.argmax(output_tokens[0, -1, :])\n",
    "        sampled_char = reverse_target_char_index[sampled_token_index]\n",
    "        decoded_sentence += ' '+sampled_char\n",
    "\n",
    "        # Exit condition: either hit max length\n",
    "        # or find stop character.\n",
    "        if (sampled_char == '_END' or\n",
    "           len(decoded_sentence) > 50):\n",
    "            stop_condition = True\n",
    "\n",
    "        # Update the target sequence (of length 1).\n",
    "        target_seq = np.zeros((1,1))\n",
    "        target_seq[0, 0] = sampled_token_index\n",
    "\n",
    "        # Update states\n",
    "        states_value = [h, c]\n",
    "\n",
    "    return decoded_sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kybjUWhh7NAk"
   },
   "source": [
    "### Evaluation on Train Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 867
    },
    "colab_type": "code",
    "id": "LYA7d35PcRCR",
    "outputId": "2b73ce72-8abd-4595-e2c1-0502f1c44322"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input English sentence: count to thirty\n",
      "Actual Telugu Translation:  ముప్పైకి లెక్కించండి \n",
      "Predicted Telugu Translation:  ముప్పై సంవత్సరాలు \n",
      "\n",
      "\n",
      "Input English sentence: tom is trying to confuse you\n",
      "Actual Telugu Translation:  టామ్ మిమ్మల్ని కలవరపెట్టడానికి ప్రయత్నిస్తున్నాడు \n",
      "Predicted Telugu Translation:  టామ్ మిమ్మల్ని మీకు ఒక నిమిషాలు ఉన్నాడు \n",
      "\n",
      "\n",
      "Input English sentence: i would like to send a telegram\n",
      "Actual Telugu Translation:  నేను టెలిగ్రామ్ పంపాలనుకుంటున్నాను \n",
      "Predicted Telugu Translation:  నేను ఒక విధంగా మేరీ నాకు ఇష్టమైన నుండి ఒక రెండు \n",
      "\n",
      "\n",
      "Input English sentence: i cannot remember my password\n",
      "Actual Telugu Translation:  నా పాస్‌వర్డ్ నాకు గుర్తులేదు \n",
      "Predicted Telugu Translation:  నా పేరు నాకు గుర్తులేదు \n",
      "\n",
      "\n",
      "Input English sentence: i do only what i want to do\n",
      "Actual Telugu Translation:  నేను చేయాలనుకున్నది మాత్రమే చేస్తాను \n",
      "Predicted Telugu Translation:  నేను ఏమి చేయాలో నాకు ఇష్టం \n",
      "\n",
      "\n",
      "Input English sentence: tom might cry\n",
      "Actual Telugu Translation:  టామ్ ఏడుపు ఉండవచ్చు \n",
      "Predicted Telugu Translation:  టామ్ యొక్క సమయం ఉండాలని మీరు ఒక చేస్తాడు \n",
      "\n",
      "\n",
      "Input English sentence: tom may not be able to come\n",
      "Actual Telugu Translation:  టామ్ రాకపోవచ్చు \n",
      "Predicted Telugu Translation:  టామ్ దాన్ని అడిగాడు \n",
      "\n",
      "\n",
      "Input English sentence: we have to persuade tom to stay here\n",
      "Actual Telugu Translation:  మేము ఇక్కడ ఉండటానికి టామ్‌ను ఒప్పించాలి \n",
      "Predicted Telugu Translation:  మేము టామ్‌ను ఇక్కడ చేయడానికి ఒక చేస్తాము \n",
      "\n",
      "\n",
      "Input English sentence: tom would not let me leave\n",
      "Actual Telugu Translation:  టామ్ నన్ను వదిలి వెళ్ళనివ్వడు \n",
      "Predicted Telugu Translation:  టామ్ నన్ను వచ్చే మాట చాలా రాలేదు \n",
      "\n",
      "\n",
      "Input English sentence: tom is worried about you\n",
      "Actual Telugu Translation:  టామ్ మీ గురించి బాధపడుతున్నాడు \n",
      "Predicted Telugu Translation:  టామ్ మీ గురించి ఆందోళన చెందలేదు \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_gen = generate_batch(X_test, y_test, batch_size = 1)\n",
    "for k in range(10):\n",
    "    (input_seq, actual_output), _ = next(test_gen)\n",
    "    decoded_sentence = decode_sequence(input_seq)\n",
    "    print('Input English sentence:', X_test[k:k+1].values[0])\n",
    "    print('Actual Telugu Translation:', y_test[k:k+1].values[0][6:-4])\n",
    "    print('Predicted Telugu Translation:', decoded_sentence[:-4])\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YJXCUSIZ-yVy"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "machine_shape": "hm",
   "name": "English_To_Telugu_Encoder_Decoder.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}