neural_machine_translation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NEURAL MACHINE TRANSLATION\n",
    "\n",
    "In this notebook we will implement a Machine Translator using a **Sequence to Sequence (Encoder-Decoder) LSTM Network with Attention**.\n",
    "\n",
    "Specifically we will tackle:\n",
    "    1. English to German text translation\n",
    "    \n",
    "    2. French to English text translation\n",
    "    \n",
    "### Dataset\n",
    "\n",
    "We are using the Open Subtitles Dataset (http://opus.lingfil.uu.se/OpenSubtitles.php) which is a collection of documents from http://www.opensubtitles.org/. So basically the data comprises of movie subtitles in different languages.\n",
    "We have chosen data for English-> German task and French-> English translation task. \n",
    "\n",
    "Note - Some language data is in a TMX format, which needs to be converted in a useful format before we train our model. I have written a TMX convertor which accomplishes this task, and can be used on any language dataset for creating a machine translator.\n",
    "\n",
    "Data preprocessing includes:\n",
    "    1. Tokenizing \n",
    "    2. Anonymizing rare words <ukn> \n",
    "    3. Adding start (<go>) and end (<eos>) tokens to the sequences\n",
    "    4. Padding sequences <pad>\n",
    "    5. One hot encoding sequences\n",
    "    \n",
    "These tasks are handled in the helper script data_utils.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# dependencies\n",
    "%matplotlib inline\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "import time\n",
    "import data_utils\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def softmax(x):\n",
    "    n = np.max(x)\n",
    "    e_x = np.exp(x - n)\n",
    "    return e_x / e_x.sum()\n",
    "\n",
    "# feed data into placeholders\n",
    "def feed_dict(x, y, batch_size = 64):\n",
    "    feed = {}\n",
    "    \n",
    "    idxes = np.random.choice(len(x), size = batch_size, replace = False)\n",
    "    \n",
    "    for i in range(input_seq_len):\n",
    "        feed[encoder_inputs[i].name] = np.array([x[j][i] for j in idxes], dtype = np.int32)\n",
    "        \n",
    "    for i in range(output_seq_len):\n",
    "        feed[decoder_inputs[i].name] = np.array([y[j][i] for j in idxes], dtype = np.int32)\n",
    "        \n",
    "    feed[targets[len(targets)-1].name] = np.full(shape = [batch_size], fill_value = de_word2idx['<pad>'], dtype = np.int32)\n",
    "    \n",
    "    for i in range(output_seq_len-1):\n",
    "        batch_weights = np.ones(batch_size, dtype = np.float32)\n",
    "        target = feed[decoder_inputs[i+1].name]\n",
    "        for j in range(batch_size):\n",
    "            if target[j] == de_word2idx['<pad>']:\n",
    "                batch_weights[j] = 0.0\n",
    "        feed[target_weights[i].name] = batch_weights\n",
    "        \n",
    "    feed[target_weights[output_seq_len-1].name] = np.zeros(batch_size, dtype = np.float32)\n",
    "    \n",
    "    return feed\n",
    "\n",
    "# decode output sequence\n",
    "def decode_output(output_seq):\n",
    "    words = []\n",
    "    for i in range(output_seq_len):\n",
    "        smax = softmax(output_seq[i])\n",
    "        idx = np.argmax(smax)\n",
    "        words.append(de_idx2word[idx])\n",
    "    return words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ENGLISH TO GERMAN TRANSLATOR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# read dataset\n",
    "X, Y, en_word2idx, en_idx2word, en_vocab, de_word2idx, de_idx2word, de_vocab = data_utils.read_dataset('data.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence in English - encoded: [108, 5, 867, 93, 38, 25, 2583]\n",
      "Sentence in German - encoded: [166, 262, 8, 474, 268, 324, 67, 15, 130]\n",
      "Decoded:\n",
      "------------------------\n",
      "Well I suppose it' il be alright \n",
      "\n",
      "Na schön ich denke dieses Mal wird es gehen\n"
     ]
    }
   ],
   "source": [
    "# inspect data\n",
    "print 'Sentence in English - encoded:', X[0]\n",
    "print 'Sentence in German - encoded:', Y[0]\n",
    "print 'Decoded:\\n------------------------'\n",
    "\n",
    "for i in range(len(X[0])):\n",
    "    print en_idx2word[X[0][i]],\n",
    "    \n",
    "print '\\n'\n",
    "\n",
    "for i in range(len(Y[0])):\n",
    "    print de_idx2word[Y[0][i]],"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# data processing\n",
    "\n",
    "# data padding\n",
    "def data_padding(x, y, length = 15):\n",
    "    for i in range(len(x)):\n",
    "        x[i] = x[i] + (length - len(x[i])) * [en_word2idx['<pad>']]\n",
    "        y[i] = [de_word2idx['<go>']] + y[i] + [de_word2idx['<eos>']] + (length-len(y[i])) * [de_word2idx['<pad>']]\n",
    "\n",
    "data_padding(X, Y)\n",
    "\n",
    "# data splitting\n",
    "X_train,  X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.1)\n",
    "\n",
    "del X\n",
    "del Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# build a model\n",
    "\n",
    "input_seq_len = 15\n",
    "output_seq_len = 17\n",
    "en_vocab_size = len(en_vocab) + 2 # + <pad>, <ukn>\n",
    "de_vocab_size = len(de_vocab) + 4 # + <pad>, <ukn>, <eos>, <go>\n",
    "\n",
    "# placeholders\n",
    "encoder_inputs = [tf.placeholder(dtype = tf.int32, shape = [None], name = 'encoder{}'.format(i)) for i in range(input_seq_len)]\n",
    "decoder_inputs = [tf.placeholder(dtype = tf.int32, shape = [None], name = 'decoder{}'.format(i)) for i in range(output_seq_len)]\n",
    "\n",
    "targets = [decoder_inputs[i+1] for i in range(output_seq_len-1)]\n",
    "# add one more target\n",
    "targets.append(tf.placeholder(dtype = tf.int32, shape = [None], name = 'last_target'))\n",
    "target_weights = [tf.placeholder(dtype = tf.float32, shape = [None], name = 'target_w{}'.format(i)) for i in range(output_seq_len)]\n",
    "\n",
    "# output projection\n",
    "size = 512\n",
    "w_t = tf.get_variable('proj_w', [de_vocab_size, size], tf.float32)\n",
    "b = tf.get_variable('proj_b', [de_vocab_size], tf.float32)\n",
    "w = tf.transpose(w_t)\n",
    "output_projection = (w, b)\n",
    "\n",
    "outputs, states = tf.contrib.legacy_seq2seq.embedding_attention_seq2seq(\n",
    "                                            encoder_inputs,\n",
    "                                            decoder_inputs,\n",
    "                                            tf.contrib.rnn.BasicLSTMCell(size),\n",
    "                                            num_encoder_symbols = en_vocab_size,\n",
    "                                            num_decoder_symbols = de_vocab_size,\n",
    "                                            embedding_size = 100,\n",
    "                                            feed_previous = False,\n",
    "                                            output_projection = output_projection,\n",
    "                                            dtype = tf.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# define our loss function\n",
    "\n",
    "# sampled softmax loss - returns: A batch_size 1-D tensor of per-example sampled softmax losses\n",
    "def sampled_loss(labels, logits):\n",
    "    return tf.nn.sampled_softmax_loss(\n",
    "                        weights = w_t,\n",
    "                        biases = b,\n",
    "                        labels = tf.reshape(labels, [-1, 1]),\n",
    "                        inputs = logits,\n",
    "                        num_sampled = 512,\n",
    "                        num_classes = de_vocab_size)\n",
    "\n",
    "# Weighted cross-entropy loss for a sequence of logits\n",
    "loss = tf.contrib.legacy_seq2seq.sequence_loss(outputs, targets, target_weights, softmax_loss_function = sampled_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# ops and hyperparameters\n",
    "learning_rate = 5e-3\n",
    "batch_size = 64\n",
    "steps = 1000\n",
    "\n",
    "# ops for projecting outputs\n",
    "outputs_proj = [tf.matmul(outputs[i], output_projection[0]) + output_projection[1] for i in range(output_seq_len)]\n",
    "\n",
    "# training op\n",
    "optimizer = tf.train.RMSPropOptimizer(learning_rate).minimize(loss)\n",
    "\n",
    "# init op\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "# forward step\n",
    "def forward_step(sess, feed):\n",
    "    output_sequences = sess.run(outputs_proj, feed_dict = feed)\n",
    "    return output_sequences\n",
    "\n",
    "# training step\n",
    "def backward_step(sess, feed):\n",
    "    sess.run(optimizer, feed_dict = feed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------TRAINING------------------\n",
      "step: 0, loss: 9.15700340271\n",
      "step: 4, loss: 9.16397285461\n",
      "step: 9, loss: 9.00219345093\n",
      "step: 14, loss: 9.09013843536\n",
      "step: 19, loss: 9.02696609497\n",
      "Checkpoint is saved\n",
      "step: 24, loss: 8.9703540802\n",
      "step: 29, loss: 8.98925685883\n",
      "step: 34, loss: 8.76154136658\n",
      "step: 39, loss: 8.15525436401\n",
      "Checkpoint is saved\n",
      "step: 44, loss: 7.24500846863\n",
      "step: 49, loss: 7.15777587891\n",
      "step: 54, loss: 7.11880111694\n",
      "step: 59, loss: 6.68689155579\n",
      "Checkpoint is saved\n",
      "step: 64, loss: 6.33931493759\n",
      "step: 69, loss: 6.3931517601\n",
      "step: 74, loss: 5.88977241516\n",
      "step: 79, loss: 5.81943750381\n",
      "Checkpoint is saved\n",
      "step: 84, loss: 5.61211776733\n",
      "step: 89, loss: 5.38785266876\n",
      "step: 94, loss: 5.37372303009\n",
      "step: 99, loss: 5.86443328857\n",
      "Checkpoint is saved\n",
      "step: 104, loss: 5.84311008453\n",
      "step: 109, loss: 5.39813232422\n",
      "step: 114, loss: 5.33500385284\n",
      "step: 119, loss: 5.07542133331\n",
      "Checkpoint is saved\n",
      "step: 124, loss: 5.9093875885\n",
      "step: 129, loss: 5.23268318176\n",
      "step: 134, loss: 5.03547716141\n",
      "step: 139, loss: 4.85352993011\n",
      "Checkpoint is saved\n",
      "step: 144, loss: 5.19811439514\n",
      "step: 149, loss: 5.10133123398\n",
      "step: 154, loss: 5.80060482025\n",
      "step: 159, loss: 4.87752103806\n",
      "Checkpoint is saved\n",
      "step: 164, loss: 4.69631671906\n",
      "step: 169, loss: 4.75822257996\n",
      "step: 174, loss: 4.51942920685\n",
      "step: 179, loss: 5.3478679657\n",
      "Checkpoint is saved\n",
      "step: 184, loss: 4.5581908226\n",
      "step: 189, loss: 4.94794464111\n",
      "step: 194, loss: 4.24569416046\n",
      "step: 199, loss: 4.5277132988\n",
      "Checkpoint is saved\n",
      "step: 204, loss: 4.67749834061\n",
      "step: 209, loss: 5.12088108063\n",
      "step: 214, loss: 4.03745555878\n",
      "step: 219, loss: 4.26204490662\n",
      "Checkpoint is saved\n",
      "step: 224, loss: 4.45562553406\n",
      "step: 229, loss: 4.53939533234\n",
      "step: 234, loss: 3.81297540665\n",
      "step: 239, loss: 4.04966640472\n",
      "Checkpoint is saved\n",
      "step: 244, loss: 3.91259813309\n",
      "step: 249, loss: 3.93588852882\n",
      "step: 254, loss: 3.72767567635\n",
      "step: 259, loss: 3.58555603027\n",
      "Checkpoint is saved\n",
      "step: 264, loss: 3.48668122292\n",
      "step: 269, loss: 3.7715485096\n",
      "step: 274, loss: 3.82588505745\n",
      "step: 279, loss: 3.81011795998\n",
      "Checkpoint is saved\n",
      "step: 284, loss: 3.50599241257\n",
      "step: 289, loss: 3.57776784897\n",
      "step: 294, loss: 3.28960394859\n",
      "step: 299, loss: 3.17676305771\n",
      "Checkpoint is saved\n",
      "step: 304, loss: 3.45937538147\n",
      "step: 309, loss: 3.45889377594\n",
      "step: 314, loss: 3.30122423172\n",
      "step: 319, loss: 2.92198801041\n",
      "Checkpoint is saved\n",
      "step: 324, loss: 3.1335272789\n",
      "step: 329, loss: 3.06132173538\n",
      "step: 334, loss: 3.12045907974\n",
      "step: 339, loss: 3.10897731781\n",
      "Checkpoint is saved\n",
      "step: 344, loss: 3.12083172798\n",
      "step: 349, loss: 2.98700928688\n",
      "step: 354, loss: 3.09267830849\n",
      "step: 359, loss: 2.72253131866\n",
      "Checkpoint is saved\n",
      "step: 364, loss: 3.09008741379\n",
      "step: 369, loss: 3.11639308929\n",
      "step: 374, loss: 2.75074076653\n",
      "step: 379, loss: 2.96363019943\n",
      "Checkpoint is saved\n",
      "step: 384, loss: 2.75268793106\n",
      "step: 389, loss: 3.26041889191\n",
      "step: 394, loss: 2.81868982315\n",
      "step: 399, loss: 2.62101173401\n",
      "Checkpoint is saved\n",
      "step: 404, loss: 2.54101467133\n",
      "step: 409, loss: 2.57273674011\n",
      "step: 414, loss: 2.6184129715\n",
      "step: 419, loss: 2.59512686729\n",
      "Checkpoint is saved\n",
      "step: 424, loss: 2.59006547928\n",
      "step: 429, loss: 2.5291800499\n",
      "step: 434, loss: 2.35099744797\n",
      "step: 439, loss: 2.5148293972\n",
      "Checkpoint is saved\n",
      "step: 444, loss: 2.23212528229\n",
      "step: 449, loss: 2.79554772377\n",
      "step: 454, loss: 2.53582048416\n",
      "step: 459, loss: 2.5102725029\n",
      "Checkpoint is saved\n",
      "step: 464, loss: 2.43915081024\n",
      "step: 469, loss: 2.41429114342\n",
      "step: 474, loss: 2.60880255699\n",
      "step: 479, loss: 2.13307762146\n",
      "Checkpoint is saved\n",
      "step: 484, loss: 2.20084500313\n",
      "step: 489, loss: 1.92101216316\n",
      "step: 494, loss: 2.55321574211\n",
      "step: 499, loss: 2.19230365753\n",
      "Checkpoint is saved\n",
      "step: 504, loss: 2.5924539566\n",
      "step: 509, loss: 1.97204601765\n",
      "step: 514, loss: 2.26565432549\n",
      "step: 519, loss: 1.94829809666\n",
      "Checkpoint is saved\n",
      "step: 524, loss: 2.25480651855\n",
      "step: 529, loss: 2.06318306923\n",
      "step: 534, loss: 2.33809757233\n",
      "step: 539, loss: 1.91838788986\n",
      "Checkpoint is saved\n",
      "step: 544, loss: 2.01863408089\n",
      "step: 549, loss: 2.01448082924\n",
      "step: 554, loss: 1.90342307091\n",
      "step: 559, loss: 2.11137914658\n",
      "Checkpoint is saved\n",
      "step: 564, loss: 2.05398869514\n",
      "step: 569, loss: 1.861328125\n",
      "step: 574, loss: 1.91523396969\n",
      "step: 579, loss: 1.65993237495\n",
      "Checkpoint is saved\n",
      "step: 584, loss: 1.80678081512\n",
      "step: 589, loss: 1.96383666992\n",
      "step: 594, loss: 2.05678653717\n",
      "step: 599, loss: 1.96713924408\n",
      "Checkpoint is saved\n",
      "step: 604, loss: 1.81870746613\n",
      "step: 609, loss: 1.90742623806\n",
      "step: 614, loss: 1.75151729584\n",
      "step: 619, loss: 1.54424977303\n",
      "Checkpoint is saved\n",
      "step: 624, loss: 1.74893736839\n",
      "step: 629, loss: 1.80970931053\n",
      "step: 634, loss: 1.89435684681\n",
      "step: 639, loss: 1.63867139816\n",
      "Checkpoint is saved\n",
      "step: 644, loss: 1.63359189034\n",
      "step: 649, loss: 1.57063937187\n",
      "step: 654, loss: 1.81840777397\n",
      "step: 659, loss: 1.98436224461\n",
      "Checkpoint is saved\n",
      "step: 664, loss: 1.7420283556\n",
      "step: 669, loss: 1.89861190319\n",
      "step: 674, loss: 1.77726912498\n",
      "step: 679, loss: 1.56204771996\n",
      "Checkpoint is saved\n",
      "step: 684, loss: 1.61685800552\n",
      "step: 689, loss: 1.53837919235\n",
      "step: 694, loss: 1.50878977776\n",
      "step: 699, loss: 1.56032395363\n",
      "Checkpoint is saved\n",
      "step: 704, loss: 1.48484826088\n",
      "step: 709, loss: 1.81972098351\n",
      "step: 714, loss: 1.78298652172\n",
      "step: 719, loss: 1.29245901108\n",
      "Checkpoint is saved\n",
      "step: 724, loss: 1.42026042938\n",
      "step: 729, loss: 1.2513718605\n",
      "step: 734, loss: 1.61335992813\n",
      "step: 739, loss: 1.45178604126\n",
      "Checkpoint is saved\n",
      "step: 744, loss: 1.22602343559\n",
      "step: 749, loss: 1.19310927391\n",
      "step: 754, loss: 1.48023760319\n",
      "step: 759, loss: 1.25379800797\n",
      "Checkpoint is saved\n",
      "step: 764, loss: 1.3165820837\n",
      "step: 769, loss: 1.37616968155\n",
      "step: 774, loss: 1.38497114182\n",
      "step: 779, loss: 1.35652017593\n",
      "Checkpoint is saved\n",
      "step: 784, loss: 1.47648286819\n",
      "step: 789, loss: 1.44657492638\n",
      "step: 794, loss: 1.33593058586\n",
      "step: 799, loss: 1.51317489147\n",
      "Checkpoint is saved\n",
      "step: 804, loss: 1.79135489464\n",
      "step: 809, loss: 1.33778095245\n",
      "step: 814, loss: 1.55585002899\n",
      "step: 819, loss: 1.54594826698\n",
      "Checkpoint is saved\n",
      "step: 824, loss: 1.37593984604\n",
      "step: 829, loss: 1.55142068863\n",
      "step: 834, loss: 1.21050071716\n",
      "step: 839, loss: 1.30560541153\n",
      "Checkpoint is saved\n",
      "step: 844, loss: 1.67816090584\n",
      "step: 849, loss: 1.41701698303\n",
      "step: 854, loss: 1.35460484028\n",
      "step: 859, loss: 1.36788010597\n",
      "Checkpoint is saved\n",
      "step: 864, loss: 1.18451976776\n",
      "step: 869, loss: 1.56292009354\n",
      "step: 874, loss: 1.3371078968\n",
      "step: 879, loss: 1.26361906528\n",
      "Checkpoint is saved\n",
      "step: 884, loss: 1.20032143593\n",
      "step: 889, loss: 1.22065603733\n",
      "step: 894, loss: 1.38776421547\n",
      "step: 899, loss: 1.15384173393\n",
      "Checkpoint is saved\n",
      "step: 904, loss: 1.19300925732\n",
      "step: 909, loss: 1.27860713005\n",
      "step: 914, loss: 1.1585021019\n",
      "step: 919, loss: 1.63725423813\n",
      "Checkpoint is saved\n",
      "step: 924, loss: 1.1042406559\n",
      "step: 929, loss: 1.69578313828\n",
      "step: 934, loss: 1.29086256027\n",
      "step: 939, loss: 1.11724925041\n",
      "Checkpoint is saved\n",
      "step: 944, loss: 1.03709959984\n",
      "step: 949, loss: 1.33006501198\n",
      "step: 954, loss: 1.62902784348\n",
      "step: 959, loss: 1.09637749195\n",
      "Checkpoint is saved\n",
      "step: 964, loss: 0.912763834\n",
      "step: 969, loss: 1.29105257988\n",
      "step: 974, loss: 1.13126969337\n",
      "step: 979, loss: 0.932434380054\n",
      "Checkpoint is saved\n",
      "step: 984, loss: 1.28647613525\n",
      "step: 989, loss: 1.13877940178\n",
      "step: 994, loss: 1.05656659603\n",
      "step: 999, loss: 1.31152820587\n",
      "Checkpoint is saved\n",
      "Training time for 1000 steps: 4078.50049996s\n"
     ]
    }
   ],
   "source": [
    "# let's train the model\n",
    "\n",
    "# we will use this list to plot losses through steps\n",
    "losses = []\n",
    "\n",
    "# save a checkpoint so we can restore the model later \n",
    "saver = tf.train.Saver()\n",
    "\n",
    "print '------------------TRAINING------------------'\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    \n",
    "    t = time.time()\n",
    "    for step in range(steps):\n",
    "        feed = feed_dict(X_train, Y_train)\n",
    "            \n",
    "        backward_step(sess, feed)\n",
    "        \n",
    "        if step % 5 == 4 or step == 0:\n",
    "            loss_value = sess.run(loss, feed_dict = feed)\n",
    "            print 'step: {}, loss: {}'.format(step, loss_value)\n",
    "            losses.append(loss_value)\n",
    "        \n",
    "        if step % 20 == 19:\n",
    "            saver.save(sess, 'checkpoints/', global_step=step)\n",
    "            print 'Checkpoint is saved'\n",
    "            \n",
    "    print 'Training time for {} steps: {}s'.format(steps, time.time() - t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbIAAAEiCAYAAACcFVdfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VOXZPvDrLLNPkslOEhKWrOw7hMWIVFFQW3BDq4it\nS93FpdbttdX+6tYK1qq1b9HWvS2ir1qtIgVBNlmUnUBYspCE7JPMZNYzc35/DISEJBAgmclMru9f\nZGY4ued8JrnyPOc+zyNYrVYVREREYUoMdQFERETngkFGRERhjUFGRERhjUFGRERhjUFGRERhjUFG\nRERhLWhBdvfddyM7OxtTpkxpeezJJ5/ExIkTMW3aNMyfPx9NTU3BKoeIiCJE0ILs+uuvx7Jly9o8\nNmPGDGzcuBFr165FZmYmFi9eHKxyiIgoQgQtyCZPngyLxdLmsenTp0MUAyWMHz8e5eXlwSqHiIgi\nRK+5Rvbuu+/ioosuCnUZREQUZnpFkP3hD3+ARqPB1VdfHepSiIgozMihLuC9997D119/jU8//TTU\npRARURgKapCpatv1iVesWIE//elP+OKLL6DT6YJZChERRYigTS3ecsstuPjii3HgwAEMHz4c7777\nLh5++GHY7XbMmTMHBQUFePDBB4NVDnVRUVFRqEvok3jeQ4PnPTwFbUS2ZMmSdo/dcMMNwfr2REQU\noXpFswcREdHZYpAREVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFY\nY5AREVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFYY5AR\nEVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFY\nY5AREVFYY5AREVFYY5AREVFYC1qQ3X333cjOzsaUKVNaHrNarZg7dy7Gjx+PK664Ao2NjcEqh4iI\nIkTQguz666/HsmXL2jy2ePFiTJ8+HVu2bEFBQQEWL14crHKIiChCBC3IJk+eDIvF0uaxL774Atdd\ndx0A4LrrrsPnn38erHKIiChChPQaWU1NDZKSkgAAycnJqKmpCWU5REQUhnpVs4cgCKEugYiIwowc\nym+elJSE6upqJCUloaqqComJiaf9P0VFRUGojFrjOQ8NnvfQ4HkPruzs7HM+RlCDTFXVNl/PmjUL\n77//PhYuXIgPPvgAs2fPPu0xuuNNU9cVFRXxnIcAz3to8LyHp6BNLd5yyy24+OKLceDAAQwfPhzv\nvvsu7r//fqxatQrjx4/H6tWrsXDhwmCVQ0REESJoI7IlS5Z0+Pgnn3wSrBKIiCgC9apmDyIiojPF\nICMiorDGICMiorDGICMiorDGICMiorDGICMiorDGICMiorDGICMiorDGICMiorDGICMiorDGICMi\norAWdkGm+NXTv4iIiPqMsAuyv+yxh7oEIiLqRcIuyJYfcaHEpqDO5cOvNzdiQ5U71CUREVEIhXSH\n6LNxQ7YJf9huQ53Lj/GJWvxppw3Ly1y4Z3gU4vRhl8tERHSOwi7I5g4yYFWFC1cNNuCKwUa4FDPe\n2t+Mn39Th8sGGFDt9CPJIOKWIeZQl0pEREEQdkMYWRTw2nlxuGKwEQCglwX8YqgZf5hsgcenYkSc\nBp8UO1Hr8oW4UiIiCoawG5F1JitGg6wYDQDgQJOCf5e4cFOuKcRVERFRTwu7EVlX/GSgAf8ucbJV\nn4ioD4jIIBscLSPVKGHdUXY0EhFFuogMMgCYma7Ht5UMMiKiSBexQZYZLaPUzoYPIqJIF7FBlm6W\nUGr3QVV5nYyIKJJFbJCZNSJMsoAalz/UpRARUQ+K2CADTozKiIgockV0kGWYZZTalVCXQUREPSiy\ngyxKQqmNIzIiokgW2UFmljgiIyKKcBEeZGzBJyKKdBEdZEkGEXavHw6FnYtERJEqooNMFASkmTgq\nIyKKZL0iyBYtWoT8/HxMmTIFt956KzweT7cdO8MsoYxBRkQUsUIeZKWlpXjrrbewZs0arF+/Hoqi\nYNmyZd12/P5mCUfY8EFEFLFCHmTR0dHQarVwOBxQFAUOhwMpKSnddvxEvYRaru5BRBSxQh5kFosF\nd999N4YPH44hQ4YgJiYG06dP77bjJ+hFBhkRUQQTrFZrSFfVLS4uxrx58/Dll18iOjoaN954I+bM\nmYOrr766w9cXFRWd0fFLnCL+XmnErwfbu6NcIiLqRtnZ2ed8DLkb6jgnP/zwAyZNmoTY2FgAwOWX\nX45NmzZ1GmRn+qbjXX78qaKuW05WX1RUVMRzFwI876HB8x6eQj61mJWVhS1btsDlckFVVaxevRo5\nOTnddnyLTkCzV4XHx+1ciIgiUciDbMSIEbj22msxffp0TJ06Faqq4qabbuq244uCgDi9iDo3r5MR\nEUWikE8tAsC9996Le++9t8eOH68XUefyI8Uo9dj3ICKi0Aj5iCwYAi34vCmaiCgS9YkgYws+EVHk\n6hNBFq8XUetkkBERRaI+EWSJepFTi0REEapPBFk8l6kiIopYfSLIEo51LRIRUeTpM0FW4/JDVXlT\nNBFRpOkTQWbSiBAAOBQGGRFRpOkTQQawBZ+IKFL1nSAzMMiIiCJRnwmydJOML8ucUPycXiQiiiR9\nJshuH2ZCvcuPp7c2we7lyIyIKFL0mSAzyiKenWRBlEbAtSvqsHiHDV6OzoiIwl6fCTIA0EoCfjk6\nGm9dEIftdR7safCGuiQiIjpHfSrIjovXSxiXqMWeegYZEVG465NBBgBDYzXY06CEugwiIjpHfTzI\nvFztg4gozPXZIOtnEOEHUMN7y4iIwlqfDTJBEDA0VmbDBxFRmOtykK1ZswbFxcUAgKNHj+L222/H\nnXfeiaqqqp6qrccNjdWw4YOIKMx1OcgeeughSJIEAHjiiSegKApEUcR9993XY8X1NDZ8EBGFP7mr\nL6ysrER6ejoURcF///tf7Ny5E1qtFnl5eT1ZX4/Ktcg40KTA7VOhk4RQl0NERGehyyOyqKgoVFdX\nY926dcjLy4PZbAYAKEr4jmiMsojMaAm7OL1IRBS2ujwiu+222zBjxgx4PB48++yzAICNGzciOzu7\nx4oLhvGJWmyp8WBcojbUpRAR0VnocpAtXLgQl112GSRJwqBBgwAAqampePnll3usuGAYn6jFy7vs\n+EWoCyEiorNyRu33AwYMQGVlJT766CMAQEpKCgYOHNgTdQXNkFgNKpp9sLp5PxkRUTjqcpDt3r0b\n48aNw3333Yd77rkHALBu3TrcfffdPVZcMMiigNEJGmyt8QAAV/ogIgozXQ6yBx98EI899hg2b94M\nWQ7MSE6dOhUbN27sseKCZXyiFsuPuPDYd1Y8uMEa6nKIiOgMdDnI9u7di3nz5gEIrIoBACaTCU6n\ns2cqC6JJSTrst3qRa9Fgb4MCH0dlRERho8tBlpGRgW3btrV5bOvWrRg8eHC3FxVsqSYJH1+SiAW5\nJlh0AiqafaEuiYiIuqjLXYuPP/44rrnmGvzsZz+D1+vFokWL8Oabb4Z91+LJsqI1ONCoIN3c5VND\nREQh1OUR2SWXXIIPP/wQdXV1mDp1KsrKyvDuu+9ixowZ51xEY2MjFixYgIkTJyI/Px9btmw552Oe\nrayYwGofREQUHs5o2DFq1Ci8+OKL3V7EI488gosuughvvfUWFEWBw+Ho9u/RVVkxMj4tDv/rfkRE\nfUWXR2SvvPIKduzYAQDYvHkzhg8fjpEjR2LTpk3nVEBTUxM2bNiAG264AQAgyzKio6PP6ZjnIita\nxoFGjsiIiMJFl4Psz3/+MwYMGAAAeOqpp3DnnXfioYcewqOPPnpOBZSUlCA+Ph533nknCgoKcN99\n94W0EzLJIMLjV1HPDTeJiMKCYLVau9Rrnp6ejrKyMthsNowYMQIHDx6EJEnIyMhAaWnpWRewbds2\nXHjhhfj6668xZswYPPLII4iOjsZjjz3W4euLiorO+nt11R9KTJgV78YwM0dmREQ9qTvW6+3yNbK0\ntDR89913KCwsxJQpUyBJEpqamlr2KDtbqampSEtLw5gxYwAAP/nJT/DSSy91+vpgLFI80m2DUxeD\n7GzTOR+rxKbAKAtINJzbeQqVoqKisF8YOhzxvIcGz3t46nKQPf3001iwYAE0Gg3eeecdAMBXX32F\ncePGnVMBSUlJSEtLw4EDB5CVlYXVq1eHfI+zEXEavL6nGZOSdRgc3fkp+uSwA3F6CSPiNLDoOp6l\nff+AAylGCTflnnsoEhFRe10OspkzZ6KwsLDNY3PmzMGcOXPOuYjnn38et956K7xeLwYOHIhXX331\nnI95LgpS9XAoKu5f34BfjY7GlH66dq+xuv34855mjIzXYPEOGz6cGQ9RaL85Z43Th64uFKL4VTR6\n/IjXh+fojYgoFLocZIWFhYiLi0NSUhLsdjtefvlliKKIe++9FxqN5pyKGDFiBFatWnVOx+hul2QY\nkGCQ8MedNuQna9uF1IFGBbkWGS/kW3DdilqU2X0YENX+dFY7/fB0caGQdUfd+HeJC7+fbOmOt0BE\n1Cd0uWvx5ptvRmNjIwDgf/7nf7B+/Xps2bIFCxcu7LHiQm1cggZGWcCGKk+75/Y3epEdEwiuPIsG\nhdb2u0yrqopqpw/lzV1rGjnS7EO1k8tjERGdiS6PyMrKypCdnQ1VVfHZZ5/hu+++g16vx6hRo3qy\nvpASBAHzMo34xwEHpp40vVjUqGBycmBX6aGxgcWGL05v+/8bPSo0ogCHT4VD8cMon/rvhopmH2rZ\n9k9EdEa6PCLT6XSw2WzYunUr+vfvj/j4eOh0Orjd7p6sL+QKUnSodvqwu77tiOtAo4KsmMCUap5F\n7nBEVuPyIdkgItUoobwLCxFXOnxoVlQ4Fa6+T0TUVV0ekV111VX48Y9/DLvdjltvvRUAsH37dmRk\nZPRYcb2BLAq4arARHx12YFhcDADAofhR4/IhwxxoysiO0aDYpsDtU6GTTlxLq3b6kWSQIItAebMP\n2TGnvpZY3uyDRgTqXD7056LFRERd0uXfls8++yxWrlwJWZZRUFAAABBFEc8880yPFddbXJKux1v7\nm1Hn8iFeL+Fgo4KBUTJkMRBaellAf5OMg00KhsaeCKsapw+JBhEGWTjtiMzrV1Hv9iMrRkad24/+\n5h59S0REEeOM/uyfMWMGysrKsGnTJqSkpLTcxBzporQiLkjV4fMSF27MNaGoUWlp9DhuSKyMwgZv\nmyA7PiKL1orY38HUY2tVDh8S9CL6GSReJyMiOgNdvkZ29OhRzJ49G+PGjcP8+fMxduxYzJ49G5WV\nlT1ZX68xZ6ARn5Y4ofjVY0HWdppwSKwGexrahlW104ckg4g00+mvkVU4fEg1SojXi6hjkBERdVmX\ng+yBBx7A8OHDcfjwYezbtw/FxcUYMWIE7r///p6sr9fIjJGRFS3jiq9qsarCjZyTRmTjErTYWuvF\n/+6xw+MLNGvUuAIjsi4FWbMPqSYJCXoRta5Tv/Z0zxMR9SVdnlrcuHEj9u3b13Lzs8lkwtNPP40h\nQ4b0WHG9zbOTYmD1qKhy+pB7UpAlGyW8cX4cXtzRhGd+aMJvxseg2ulDol5EkkGE1eNv1wzSWkXz\n8RGZhP2n2Ubmlm/q8XpBHPoZuQIIEVGXR2QWi6XdElVFRUWIiYnp9qJ6K0EQEKsTkWfRQOhgOao4\nvYjHx0Zjc7UHDW4/al1+JBokSIKAfgYJlY7OR1IVjhMjslNNLToUP6we9ZTHIiLqS7o8Irvvvvsw\nZ84czJ8/v2VLl/feew+PP/54T9YXdoyyiMnJWnx02AGTLLSMwDKiJOy3ejHwpGWsAtfRpJapRZ0k\nnLLZo8YZeK6KQUZEBOAMRmQLFizAm2++ibq6Onz55Zeoq6vDkiVLUF5e3pP1haWL0vX46JATSa22\nbrk0w4B/HXRCbbWC8BG7gnlf1+HfJU5UOPxIMR4fkfnavK6140tYVTvZEEJEBJxh+/3555+P888/\nv+Vrt9uNuXPnclR2knEJWmglAYmGE38nTE7W4o3CZnxX7UF+cmC5qy01HoxJ0ODv+5qhlQCzJvB6\nURDQrKgwa9pPX1Y7/ZAE4GgX12T8qsyJQVEycizntrAzEVFvdc7LR3Q2cujLZFHARf118LU6NYIg\n4KfZRry734FJSVoIgoDNNR5cmmFAVoyM76pPLEx8vAX/eLC1Vu30ISdG7vLU4seHnfhRmp5BRkQR\nq8tTi53pqOmBgJvzzLg5r+1mmtNTdbB6/Pih1gvFr2JbrRfjErUYECXjmkxjy+sCLfgdTx3WuPwY\nHqfp0tSix6fiQKOCBjenIYkocp12RLZ69epOn/N6T71aRV8WaPJoG/KSIGBBrglvFNpx21Az0kxS\nhztLx+s6v5es2unD5QMM+KT4xPW2ow4/UkztW/EPNClQVMDqYZARUeQ6bZDdc889p3y+f//+3VZM\nXzAjTYf3iprxlz12jE/UdviaBH3ny1RVO/3IMMvQy0JLG/6Tmxvx4cyEdq8tbPAiXidyREZEEe20\nQbZjx45g1NFnSIKAm/PM+J/Njbh1SMcrAycZROzr4Kbo4xt1JhkCazJWOXzYXutBrcsPl6JCL7cd\nAe5p8GJyPy0OnuYGayKicHbO18jozE3rp8Vdw8wYHtdxA8YFaXpsOOrG0ZMaOmxeFZIgwKQJrBZy\n1OnD9mP7pHV0g3ShVcGUZB3qOSIjogjGIAsBQRBwdaYRGrHjRplYnYgfDzTgrX3NUFUV31a6UeP0\ntYzGgMCSWEcdPuyq9yInRm4XZE0eP+rdfoyK18Dq8bO7lIgiFndv7KXmZRlxw3/r8OAGH/Y2KLgk\nQ48JidqWm6yTDRLWHXUjTidiWJym3aLEhdZAwJk0IgQATp8Ko8wOUyKKPByR9VJRGhG3DzVjdIIW\nfz0/Fv894sKR5lYjMoOI7XVejIrXINXYfh3HTdWelr3RLGz4IKIIxhFZLzYrw9Dy71yLBh8fdrQ8\nlnxs5fuR8VoYJAFba07cUH2oScHXR1x4Y3ocgMBUpdWtIq3tbW1ERBGBI7IwcfkAPSodfiTpj4/I\nAkE2Kl6DVJOEimMjMr+q4sXtTfh5nhkJ+sBrYrWdj8g8Pl47I6LwxiALE1P66RCrE1tGYhatgP8Z\nF40kg4SUY40fflXFynI3VASC7ziLTmx3U7SqqvhboR03rKwL5tsgIup2nFoME7Io4LVpsUgyBv72\nEAQBP0oLhJVBFmDWBJa1+vqIC1cOMkJstXRYrE5s04Kvqipe3GFDUaMCq9sPh+KHUebfNEQUnvjb\nK4ykmAKbdHYk1Shhv1XBrvrATdCtBa6RnQiyrbVebK/zYvEUy7HRHBtBiCh8McgiRIpJxNJDDoxN\n0LYbXZ18jeyd/c2Yn22CURaR0kHHIxFROGGQRYhUo4TtdV5MT9O1e651+/32Og9qnH7MOPa6fgwy\nIgpzDLIIkWqSoBUDG3ieLLZVs8c7+5txfbYR8rFVRTgiI6JwxyCLECPitLgp19Rh00bssRFZsU3B\n4SYfZqaf6GhMNUnt1nQkIgonvSbI/H4/CgoKcO2114a6lLCUapLw0+yO73iO1gqwe1UsO+TAZQP0\nbdZ47GcUUdHMICOi8NVrguzPf/4z8vLyQl1GRJIEAdEaAV8fceHygYY2zwWmFrmoMBGFr14RZOXl\n5fj6668xf/78UJcSsSw6EZOTdS2rfRxn1ojQiECjR4XdG5h+JCIKJ70iyB577DE8/fTTEDq5R4rO\n3ZgELa7JNHb43PGGj/eKHLh3XQPqXJxqJKLwEfKVPb766iskJSVh5MiR+Pbbb087xVVUVBSkyiLL\nLD2AWqCotv1zUX4jNh9swr+r9BhiUvDU+qO4u78DXhWQBZ7zUOF5Dw2e9+DKzs4+52MIVqs1pBdH\nnn76afzrX/+CJElwuVyw2+247LLL8Je//CWUZfUpr++24/taD0waAS/kW3DHmgb4VRUldh9mx7vw\n4JSMUJfY5xQVFXXLDzidGZ738BTyqcUnn3wSu3btwvbt2/HGG2/gvPPOY4gFWYpRxP5GBZdlGKAR\nBTyfH4N7RkTh9YJYfNOghUthIwgR9V4hn1qk0EsxSYjWCpiWEljtI14vIf5YU0imwYevypz4yaD2\n19ccih/v7nfgmwoX0kwyZmXoMSNN3+51REQ9KeQjstamTZuGf/zjH6Euo88Zk6DFoskW6KT2zTYz\n49341yEnfCddu7R5/Fiwsh51bj+emhCDmel6vLbbzjZ+Igq6XhVkFBoaUUBWjKbD57IMPsRoBaw4\n4mrz+NqjbuRZNHh0TDSyYzS4qL8eBlnAvka27xNRcDHI6JQEAbhvRBRe321v05a/usKN6altFyie\n1k+HtZXuYJdIRH0cg4xOK9eiwaUDDFi0wwZVVWHz+rGzg33PpvbTYd1RBhkRBReDjLrkxhwTjjr8\neLfIgXWVboxJ0LRboHhIrAyrR8WOOg8e2tCAT4qdIaqWiPoSBhl1iVYS8EJ+DL4sc+F/9zbj/NT2\n3YmSIGBKshYPrLfCJIv4spRBRkQ9j+331GXxegmLJluweKcNUzrY9wwArs824ZJ0PYbEanDF8loc\ndfjQz3hifcd39zejwuFDnE7Ez/JMkLgsGRGdI47I6IwkGyU8N8kCk6bjj06qScKIeC1kUcC0fjp8\n26r5o8Sm4KPDTgyN1WBLjQf/LnF1eAwiojPBIKMec36KDqsrTgTZ8U7HywYY8NCoKPyt0A6r2x/C\nCokoEjDIqMeMTdSixK6gxhlo2/+mwtXSsp8Vo8GMND3eKLSHskQiigAMMuoxGlHARf31eHW3HSU2\nBU1eFcPjTtx4fWOOCf8td0PxB2c1EFVV8da+5narlBBReGOQUY/6xVAzapw+PLGpEeen6CC2au6w\n6ESkGCXsD9JqILUuP/62rxklNu63RhRJGGTUo3SSgN9OsEAUgAv7t2/ZHx2vwbZaDwDAp6rw+Hpu\ntHSwKRCYexq8PfY9iCj42H5PPS5OL+LvF8R1uAP46AQtPitx4qfZwDv7HfhPqRNPTYhBlEbAa7vt\nqHP5kWSQYPf6YfOqeGxMNAZFy/CpKsqbfcgwBz7Cbp+KleUubKr2YFqKDj/qYBX+A40KojQCCq1e\nXDbA0OPvm4iCgyMyCoqOQgwARsRpsLveC4fix/8dduCKQUY8stGKO75twIg4Le4ZHoULUnW4NsuI\nH6Xp8f++b4LHp+JPO+34xeqGlutrnxQ78fFhJyw6Ef/p5EbsA00KZqbrsbeBCxsTRRKOyCikLDoR\nyQYRr+yyY0isBvOyjJiRFuhsTDQEbqQehkCDyIREFbvqPbhrbQN8fhWJBhH7rAqGxWnwfY0H12cb\nMTZRi2uW18HjU6E9aVuag40KfpoVjc9LnHAo/nZLbBFReOJPMoXc6AQtvih14ZrMwOadiQapJcRa\nEwQBD42KRopRwvP5FkxM0uKHWg8Uv4qd9V6MTtAiSiMiI0rCXmvgOtj7Rc0osSlwKiqqXT4MjpYx\nKFpGEbebIYoYDDIKuUlJWgyJlTE6vuM90Vqz6EQ8PSEGiQYJYxMCQbbPqqCfUUKMNvBxHpugxfc1\nHlQ6fFiytxnv7G/GoSYFA8wyZFHAUIsGe9nwQRQxOLVIITcpWYdxidpOr6N1ZmS8Br/dqmBTtRtj\nE06E4NgEDd7Z74BTUTErQ481lW70M0rIigl83PNiNW2WziKi8MYRGfUKsnjmiwebj00jfnzYiTEJ\nJxYxHh6nxf5GBV+WuXBjjgkXp+vxz4MOZEYHgmxorIxd9d5uvRG7zuVDqZ3TlUShwCCjsDYmXgu7\nV8XIVtOSBllAToyMcYlaJBslXDnICJ8fLSOyNJOMAVESVpZ336LFn5W48OJ2W7cdj4i6jkFGYW1S\nshbD4jQwn7Qa/+3DzLh9qBkAkGKS8EK+BcNiT4Td9dkmvFfkgL+blqsqtinYXufFUQdXDSEKNgYZ\nhbUxCVq8NNXS7vGhsRokt9oHbXySts305bgEDQyygLXddK2s2KZgWKyMFUe4NQ1RsDHIKOydzeac\ngiDghmwTXt/TjMrmE6OonXUePPadFQ+sb8Db+5qxvMyFDVXuUy6d5fUHVhm5Oc+M5UdcUDsZ5bl9\nKt7e39zp8yc73KT06JJdRJGCQUZ91tR+Wlw12IC71jbgzUI7bltdj2d+aEJ+sg5XDTbCoajYVO3G\ne0UO3Lq6HrvqO27ZL2/2IdkgYUyCBoof2NnJ674sdeLNwmZUOzveg21vgxeF1hP/98nNjdhc4zn3\nN0oU4dh+T32WIAi4YrARaWYJ31V5cPtQM0bGa1qmIKf0C6wwoqoqVle68dgmKxZPjkVmjIxSu4JP\ni524e3gUim0KBkZJgVFejhGPbWpEskHC/BwjpqcG1nxU/CreP+BAskHE/kalzbTncZ+XOuFQVDw5\nLgZNioCyZh/K7AoAXdDOCVE4YpBRnzcpSYdJSZ2HhSAImJ6qR43TjyWFdjwzMQZ/2mnHlhoPrh5s\nxOEmBQOjAj9KszMMmNlfj+11Xry4vQnbar24IceIzdUepJkkDIvTYH+jF+eltP9+ZXYfSm0KVFXF\nAYcEAUCpnc0jRKfDqUWiLvrxQAMONSlYUtiMo04fZvbXY3WlG8U2HwZFnfibUBYFjEvU4i/nx8Hj\nV/HzVfVYtMOG+Tkm5MRosN/a8f1mR+w+eFXgsM2HIqeM8YlalDHIiE6LQUbURTpJwE25gbb9u4aZ\n8aP+OnxT4To2tdh+ciNKI+Lh0dH4+JIEvHVBPMYkaJFrkbHf6m3X8OFQ/GhW/JieqsPWGg+KHBJm\nZ+iPTS2ens3jh9Xd8bU3okjHICM6AzPT9fjthBjkJ+swNkGLI80+VDh8SDe3v+Z1nCQISDUFnk/U\ni1AR2K26tTK7D2kmGeMStFhf5UalW8KUfjp4/IGQOp2/7LXjjUL7Ob03onDFICM6A5IgtFzfkkUB\n0/rp0M8otdsypjOCICDHosH+k1bfP2IPhOGYBC1+qPUiQ++DThLQ3ySh1O6Dx6fit1sbW9rxG9z+\nln3XvH4dPhQoAAAZYUlEQVQVayrc2N1JtyRRpAt5kJWXl+Pyyy9Hfn4+pkyZgtdffz3UJRF12aUZ\nBkxPPbOuwpyYwPTi4SalZfHismYf+pskWHQiMqNlZBkDQZdhllBmV/BDrQf/LXdje12gHX95mQu/\n32bDEbuCrTUepJgkVDr8sHs5vUh9T8i7FmVZxu9+9zuMHDkSdrsd06dPx4wZM5CTkxPq0ohOa1ic\nBsPiTr/9TGs5MTJe3W3Hx8WBEdXk5AQcsSuYkBRY+HhBjhH++gYAQLpZRlmzDzvrvUg2iFhf5cGE\nJB3WHXVjaJzm2DJbwMz+gVX+CxsUjE/Sdvq9iSJRyEdkycnJGDlyJADAbDYjJycHlZWVIa6KqOeM\niNci16LBq9Ni0d8k4ftaD0rtPvQ3Bf6uLEjVI00XGFmlmyUU2xSsO+rGfSOisKHKDavbj4NNCp4a\nH411VW6sPerG9FQdhsZqsOsU+6w1efz47dZG7LNyCpIiS8iDrLWSkhLs3LkT48aNC3UpRD0m9tjm\noOlmGRek6rGq3I0jzR03jKQfu1k7ySBhcrIWqgp8cMCBcYlaxOsl/GSgAbkWGfF6CcPjNKe8TvZp\nsRMVDh8e+a6xy40hDsWPV3fZUO3kbQDUe/WaILPb7ViwYAGee+45mM3mUJdDFBTnp+qwqsINjQhE\na9v/OPY3yVBU4LwUHQRBwORkHT485MDUfoHpw5tyTfjthBgAgYWS9zR40ez145cbrLjm61rc/E09\n9jZ44fGp+OiwE78cFY2/XxCHfxc7cbjp1K39zV4/Ht7QiL0NCu5b14Dy5p7Zb82pqHAovLZHZy/k\n18gAQFEULFiwAPPmzcOll156ytcWFRUFqSo6jue8Z/XXmqCi/Xk+/nWGzoyBnqMoKvIjwydDVY1I\nspejqKj9gsImwYzbVlZhkN6HB9JcKHLIeHSDB+dZPEiRJfiqDqMawAUxOvz5ezduTXN2WJNfBV4s\nNSFV68N1/VxYa9XgntUKnhxsg6nzOw3OystlRpglFT9P7biWYOPnPbiys7PP+Ri9Isjuuusu5Obm\n4o477jjta7vjTVPXFRUV8Zz3sB9rnDjUpCA7O7Xlsdbn/e1Wp3+QX0VaqgdjkpM7PNYkRxPsXhVP\njIuGJAjIB6A76MBru+14cbIF2YmBkVzaID+uW1EHR1wSvj7iwgCzjKszjS3H+aTYCa3OhSenWSAK\nAnIB2HbY8LXLiF+Ojsb2Og/2Nii4NsvYYR1AYI3KDVUeTDxpC53WNle7UVNiwyGXipSBqTBrRKw7\n6kauRUaCvpsTswv4eQ9PIZ9a3LhxI5YuXYo1a9bgvPPOQ0FBAVasWBHqsoiC5vIBeiwcGdWl18qi\ngEnJnbf7LxwZhV+Pj2mztc3Vgw3441QLxiac6K40yiKuHGTEg+ut8KvAPw864Du22kidy4c3C+14\nYFQUxFbHuXWICZuqPXiz0I5fb27EPw86Ot0RQPGr+P12Gx7b1IhN1R2v4K/4Vby62467hkVhfKIW\n/y13o8Sm4KktjXhycyO8fm5hQ10T8hFZfn4+6uvrQ10GUcgIZ7GfWmc62ptNEASMim/fkv/TbCMu\nHaBHvF7CL9bU4/uaQGv/a7vtmJ1hwODotr8eTBoRC0dG4YVtTXhukgUVzT68tMOG1wti24y4qp0+\nvLCtCZIg4K5hZnxZ5mrZSaC1FUdciNGKmNZPC50ELNnbjNUVLtw21Iwfaj14dZe9ywF/3LPfN6HB\n48flAwyY1k/bref2bOyo82BkB+eeulfIR2REFBqyKCD+2PTdxf31+KrMhc3Vbuxu8OLGHFOH/2dq\nPx0+ujgBQ2I1mJGmQ5RWwKclJ65trSp34dbV9RgRp8XvJsbgkgw9ttZ40NTBMlv/LnXh6sFGCEJg\nkWWrx49Gj4o5Aw14dEw0NlS5saOu6/uxuX0qvj3qRkGKDn/cacOehp5pTumqEpuCe9dZUedix2dP\nY5AREWak6bGhyoPFO2y4b0QUDHLnI5njoz5BEHDHUDM+KHLA61fhUlS8vNOG5ydZsCDXBFkUEKUR\nMTFJi1XlrjbHKLUrKLcryE/Wthzz/pFReHxsNGRRgFkjYl6WER8e6noDyI46DwZHy7hsgAH5yVrs\nawzt/XIrj73nfZ3sdnA6oZxaVVUV31S4Tv/CXoJBRkSw6ESMTdAiK0aDyae4BneyHIsG6WYJq8pd\n+LLMiaFxGuTFtl3p5OJ0PT4tcWFDlRuHjrX8f1nqwkXp+jZTkpOTdW2mMy9J12NbrQeVjq6NaDZV\nBxpLACA7WsaBxtCNyFRVxaoKN8YmaNrs+t1VLkXFVctrcbSL7707lNgUKMfCs9blx2+2NJ32/kGn\norb8n1BikBERAOBXYwIjojM1L8uIfxxw4F8HHbguq/2U5PhELYbHafB/h5345QYrFu+wYfkRF2al\nG055XKMs4pIMPf510IH3i5px57f1KDrFKKt1kGXFaFDUSZAVWr14dZftDN7hmTvU5IPbp2LuICMK\nz2JEtqvei0aPiv+WB29U9PimRmytCUzlHt/QtbNmnuN+930jviwL/ciNQUZEAACzRoSui6v4tzYx\nUQsVQJwusLrIyWQxMG34fL4Ff58RB49PRbpZwqDo0/eazR1kxCfFTuxu8GJmfz0e2mDFf0qdcPtU\nWN1+vLbbhjcK7ShvVtDo8SMnJnDMzGgZpXalw+m5leUuLD3kbPml3RNWVbgwPVWPIbEyCo/tP1fp\n8OH/bW3Eyztt+KH2xPfeUBXo1mzthzoPxiZosLzM1W7vutY+LXa2LDx9LuxeP440+1pGzCV2BRrx\n1EHmUPz4rtrT6UaxwRTyrkUiCm+CIODh0dHQdnKvWGtRGhG/GtP1UV+KUcKymQmI1QX+5h4Sq8Fr\nu+14ZZcdkhC4tne4ScHnJVZMSNK23C6glwWkGAPrVGbHtA3X72u8uCbTgD/tsmHJ+XHt7nFzKH4Y\n5a79je9XVfhUQNPqGIpfxYpyF54aH4MEvQStKOCow48PDjggCEC8XsTTWxrx9o/iARX43fdNuDCt\n7S0YP9R6cHOeGX/Y3oSiRgU5lo4Xpv681AmvTz3nDs3j07AHjwVZqc2Haf102HmKINtU7YFBFnDI\nxiAjoggwJPbMdgA4E8dDDAByLRr8cWosrG4/PH4VSQYJPlXFe0UOjIpvW0NWtIyixrZB1ujxo9Lh\nw21DYnGw0YrPSpyYO+jETd2qCtz8TT3uGGpGQar+lHUpfhWPfGfFngYFk5K0+FmeCRlmGV+WuZBq\nlFpGh3kWGZtqPFhZ4cJbF8QhXi+hotmH9/YHgm1QlNzmXjuH4sehJh+Gx2lwUX89lh9xdRhkdq8f\npTYf4vUidtZ7z6nNf3+jgqGxMg7bAlOKJXYFVw824umtTZ0G+5pKN+ZlGvF+kQOqqob0VgdOLRJR\n2LHoRCQZArcOSIKAG3NM7e6Vyz52naywwYv5K+vQ6PHjh1oPRsRpIIsCbh5ixtKDDvhbTd0VuyTU\nOP346HD7bskqhw+7673Y2+CFT1XxRmEz/Crw9ow4ZMXIeHijFZUOH/6+rxm3DTG3/GLPs2jwRqEd\n+Unaltsdfp5nwhelTnxe4sST46Lh8qkta1nurPMi1yJDJwmY2V+P5WWuDqcPd9Z7MSRWxtxBBnzc\nQb3HKX61ZcqwM/utXlzUX48ye6Dho8TmQ1aMjJwYGXvq2/9fj0/Fd1UezEo3QC8JqHKGdq1MjsiI\nKCJlx8hYfsSF76rciNKKeHt/MxQ/MCYhEHhDLDLMGhGbqj3IP9apudWmwVWDjVh+xIXDTQoGRcuo\nd/nx8q7Ada1UkwSXoqLO7YdBEvCXgjhYdCKuzzbB41Nxyzf1GJeobdO5mWfRoMmj4qrBJ0Z+8XoJ\nC3JN8PpVJBokTEjUYnO1B2mDZPxQ622psb9Zxv+bGIPnt9nwXlEzmjwqRsRp8OjYaGyr9WJ0vBYz\n0/X4275mVDt9LeHe2keHnXizsBnLZsbDpOl47FLUGFhuLNkgYU+DF05FRaJexIh4DXbWe9rtcfd9\nrQeZ0TLi9CIGR8s41KSgnzH4S4odxyAjooiUFSPjYJOCnww04KZcExasqoMsCHg+P7BbgCAImDvI\ngP877ER+sg6qquL7JhnPDNdBKwXWmxwSK+P13XZckmHAoxcltDTDHHX4oJMEWFpNe96Ua4IgCLio\nf9vbF4bFaXDHUHO72xKubBVsE5K0WFXhwqwMAzZUuXF/q+tlI+O1eOP8OOyq9yJeL+KR76zYXe/F\ntjoP7hpmRpRGxHVZRty3rgFPT4hpM5VqdfvxXlEzBkVJWFHuxk8GnugUdSoqRAHwqSqqnD4MjJIx\nKFrG6go30s0SBEHAiDgN/nXQ0e7crql047yUwPscHC3jsE1ps3rLD7UejI7XtJludCkqGtx+pHT3\nqtPg1CIRRahorYhfjY7CncPMiNWJmJdphKKqyGzVLTkjTY+9Vi/KmxUcODb9lnXspurPSpxYetCJ\n5/Mt+MVQc5uOzn5Gqc21OyAQjDflmpBmajs+MMgC5p1icWUgcIvCtlovHt9kRVaMjJEnXe/TywLG\nJ2kxKFrG/BwTXt1tQ6nNh7xj186uzzbh1iFmPLTBig8PnZgufWt/M2ak6XFznhmfFjtbOiArHT7c\ntroej2+yoqhRwcAoGbIoYHC0jG8q3BgQFQibkfEaHG5S2mz5o/hVrD/qxrSWIJNamkSAwBJl96+3\nYlVF2+nQ/yt24O61DbB5T0xDWt3dMyXJERkRRaxZGSdGINdkGjGpVWcjAOgkAddkGnHHmgYk6EWM\nj/ZCEAQkGSS8XhCLQcd+wfc0i05EullCtFbEo2Oi29R4skvS9Xi/qBlDYmVoW4XrjDQ9smNkPP+D\nDV+VueD1q2jyqHhzehyitQKcih97GhQ0K368sM2GeZlGrK9y4w/bbBh9bEHpzGgZdW4/BpgD0WCU\nRfw024T/3WvHs5MsAAIt+UkGCSnHphIHR8v44MCJUdv2Oi/STRL+vNuO/GRtS6PI1hovTBoB/7vH\njgdHBTpXV5a7cMXgU4d8VzDIiKhP0IgCsmLad/9dn23CRf31WFvpRqqroeXxk9v2e9ofJltgkIUO\nF35uTRYFPDgq0CBysnSzjD9Os2BjlQcJehGZ0SeC+LIBBty/vgH9TRLuGxGF81J0uLC/Hretrm8Z\n2Q06NhIbEHUiGn4y0IBlhx3YXufBqHgtvq08MRoDgAFmGRXNPnj9KjSigG21HswdZMCeBi/eK3Lg\n1iFmuH0qdtV78eYFcbhnbQN+qPVgTIIWXx1hkBERdYskg4QrBhs73Kw0WMydNGJ0ZFxi5632kiBg\nage7DVw52IixiVrkxsgt165idSLemB4H47G1NVNNEvSSgAzzietYWknArXlm/H6bDS/kW/DtUTde\nyLe0eb71PXvb6ry4crARBak6/GxVPa4YZECp3YeBUYFR3K9GR+E3WxpxQ7YJdS5OLRIRURfpJKFl\n5NVatPZEgIqCgL8UxLYJMgD4UX89Gj1+3LamHrE6EQNOen5qPx0+LXbixhwRNq8fA6MkiIKAH6Xp\n8eEhJyThRPhOSNLhmYkWPLbJitkZp16mrKsYZERE1KL1tGJrVww2It0swam0v/l5XpYR81fWIdEg\nYXT8ieuQ12YZcdvqesTpRdw34kQn5rA4Df5+QTz0Z7EkWkfYtUhERF0yIUnX4YonMVoRcwca8Pd9\nzW1WWOlnlDA5WYdKhw/DTrr9IFYnnnK7oDPBERkREZ2zqzKN+KzE1e763Y25RgyKltp0WHY3BhkR\nEZ2zKI2If10U3+52hTSTjGuzejZqOLVIRETdIhj33HWEQUZERGGNQUZERGGNQUZERGGNQUZERGGN\nQUZERGGNQUZERGGNQUZERGGNQUZERGGNQUZERGGNQUZERGGNQUZERGGtVwTZihUrMGHCBIwbNw4v\nvfRSqMshIqIwEvIg8/v9+OUvf4lly5Zh48aN+PDDD7F///5Ql0VERGEi5EG2detWZGZmIiMjAxqN\nBldeeSW++OKLUJdFRERhIuRBVlFRgbS0tJavU1NTUVFREcKKiIgonIQ8yKh3y87ODnUJfRLPe2jw\nvIenkAdZamoqjhw50vJ1RUUFUlNTQ1gRERGFk5AH2dixY3Ho0CGUlpbC4/Fg2bJlmDVrVqjLIiKi\nMCGHugBJkvD73/8eV1xxBfx+P+bPn4/c3NxQl0VERGFCsFqtaqiLICIiOlshn1o8Hd4sHTwjRozA\n1KlTcd5552HGjBkAAKvVirlz52L8+PG44oor0NjYGOIqw9/dd9+N7OxsTJkypeWxU53nRYsWYezY\nsZg4cSJWrlwZipIjQkfn/bnnnsPQoUNRUFCAgoICrFixouU5nvfuUV5ejssvvxz5+fmYMmUKXn/9\ndQDd+5nv1UHGm6WDSxRFfP755/j2229bPjyLFy/G9OnTsWXLFhQUFGDx4sUhrjL8XX/99Vi2bFmb\nxzo7z4WFhfj444+xadMmLF26FA8++CBUlZMoZ6Oj8w4Ad955J9asWYM1a9bgwgsvBADs27eP572b\nyLKM3/3ud9i4cSOWL1+OJUuWYP/+/d36me/VQcabpYNLVVX4/f42j33xxRe47rrrAADXXXcdPv/8\n81CUFlEmT54Mi8XS5rHOzvN//vMfXHnllZBlGQMGDEBmZia2bt0a9JojQUfnHUCHvyS/+OILnvdu\nkpycjJEjRwIAzGYzcnJyUFFR0a2f+V4dZLxZOrgEQcCcOXNwwQUX4O233wYAVFdXIykpCUDgA1lT\nUxPKEiNWTU1Nh+f55J+BlJQU/gx0s7/+9a+YNm0a7rnnnpbpLZ73nlFSUoKdO3di/Pjxnf5uOZtz\n36uDjILrq6++wpo1a7B06VL89a9/xfr16yEIQpvXnPw19Qye5+C45ZZbsH37dqxduxbJycl44okn\nQl1SxLLb7ViwYAGee+45mM3mbv3d0quDjDdLB1e/fv0AAAkJCbj00kuxdetWJCUlobq6GgBQVVWF\nxMTEUJYYsTo7z6mpqSgvL295HX8GuldCQkLLL9Abb7yxZQqL5717KYqCBQsWYN68ebj00ksBdO9n\nvlcHGW+WDh6HwwG73Q4AaG5uxqpVqzBs2DDMmjUL77//PgDggw8+wOzZs0NZZsQ4+bpMZ+d51qxZ\nWLZsGTweD4qLi3Ho0CGMGzcu6PVGipPPe1VVVcu/P/vsMwwdOhQAz3t3u+uuu5Cbm4s77rij5bHu\n/MyH/IboU+HN0sFTXV2NG264AYIgwOfz4eqrr8aMGTMwZswY3HTTTXj33XeRnp6Ov/3tb6EuNezd\ncsstWLt2Lerr6zF8+HA88sgjuP/++7FgwYJ25zkvLw9z587FpEmToNFo8OKLL3La8Sx1dN6//fZb\n7Ny5E6IoIiMjo+UWH5737rNx40YsXboUQ4cOxXnnnQdBEPDkk09i4cKFHf5uOZtzzxuiiYgorPXq\nqUUiIqLTYZAREVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFYY5AREVFYY5AR9ZANGzbg4osvRkZG\nBgYPHoxZs2Zh27ZteP/997lCDVE36tUrexCFK5vNhmuvvRYvvfQS5syZA4/Hgw0bNkCr1Ya6NKKI\nwxEZUQ84ePAgBEHA3LlzIQgCdDodpk+fDlmW8cADD2Dz5s3o378/Bg4cCADweDx44oknMHz4cOTm\n5uLBBx+E2+0GAKxduxbDhg3DokWLkJmZiVGjRmHp0qUt32v58uXIz89Heno6hg0bhldeeSUUb5ko\nZBhkRD0gMzMTkiThjjvuwIoVK2C1WgEAOTk5WLRoESZMmIAjR46guLgYAPDrX/8ahw4dwrp16/D9\n99+joqICL7zwQsvxqqqq0NDQgMLCQrz22mtYuHAhDh48CAC499578cc//hFlZWVYv349CgoKgv5+\niUKJQUbUA6KiovDll19CFEUsXLgQWVlZ+OlPf9rpxqRvv/02nnnmGcTExMBkMuH+++/Hhx9+2PK8\nIAh4/PHHodFoMHXqVMycORMff/wxAECj0aCwsBA2mw0xMTEtu/ES9RUMMqIekp2djVdffRW7du3C\nxo0bUVlZiUcffbTd62pra+FwODB9+nQMHDgQAwcOxFVXXYWGhoaW11gsFuj1+pav09PTcfToUQDA\nO++8g6+++gojRozAZZddhs2bN/f8myPqRRhkREFwfES2d+/edltSxMfHw2g0YuPGjSguLkZxcTFK\nS0tRWlra8hqr1Qqn09ny9ZEjR1o2Qh09ejTef/99HDx4ELNnz8bPfvaz4Lwpol6CQUbUA4qKivDK\nK6+goqICQCB4li1bhokTJyIpKQkVFRXwer0AAtOGN954Ix599FHU1tYCCOyKu3LlypbjqaqKZ599\nFl6vF+vXr8fy5csxd+5ceL1eLF26FE1NTZAkCWazGaLIH2vqW9h+T9QDzGYztm7ditdeew1NTU2I\niYnBJZdcgqeeego6nQ55eXnIycmBJEk4cOAAfvOb3+CFF17AhRdeiPr6eqSmpuLnP/85ZsyYAQDo\n168fLBYL8vLyYDQasXjxYmRmZsLr9eKf//wnHn74Yfh8PmRnZ2PJkiUhfvdEwcWNNYl6ubVr1+L2\n22/Hrl27Ql0KUa/EOQgiIgprDDIiIgprnFokIqKwxhEZERGFNQYZERGFNQYZERGFNQYZERGFNQYZ\nERGFNQYZERGFtf8Pj7mC4c1Z8SYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11fc81190>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot losses\n",
    "\n",
    "with plt.style.context('fivethirtyeight'):\n",
    "    plt.plot(losses, linewidth = 1)\n",
    "    plt.xlabel('Steps')\n",
    "    plt.ylabel('Losses')\n",
    "    plt.ylim((0, 12))\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing Our Trained NMT (English => German)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.\n",
      "--------------------------------\n",
      "What' s your name\n",
      "Was ist denn Name \n",
      "--------------------------------\n",
      "2.\n",
      "--------------------------------\n",
      "My name is\n",
      "Mein Name ist \n",
      "--------------------------------\n",
      "3.\n",
      "--------------------------------\n",
      "What are you doing\n",
      "Was machst du gemacht \n",
      "--------------------------------\n",
      "4.\n",
      "--------------------------------\n",
      "I am reading a book\n",
      "Ich bin ein Buch Buch \n",
      "--------------------------------\n",
      "5.\n",
      "--------------------------------\n",
      "How are you\n",
      "Wie geht' s \n",
      "--------------------------------\n",
      "6.\n",
      "--------------------------------\n",
      "I am good\n",
      "Ich bin gut \n",
      "--------------------------------\n",
      "7.\n",
      "--------------------------------\n",
      "Do you speak English\n",
      "Mach du Ihnen an \n",
      "--------------------------------\n",
      "8.\n",
      "--------------------------------\n",
      "What time is it\n",
      "Was ist es \n",
      "--------------------------------\n",
      "9.\n",
      "--------------------------------\n",
      "Hi\n",
      "Hallo \n",
      "--------------------------------\n",
      "10.\n",
      "--------------------------------\n",
      "Goodbye\n",
      "Wiedersehen \n",
      "--------------------------------\n",
      "11.\n",
      "--------------------------------\n",
      "Yes\n",
      "Ja \n",
      "--------------------------------\n",
      "12.\n",
      "--------------------------------\n",
      "No\n",
      "Nein \n",
      "--------------------------------\n"
     ]
    }
   ],
   "source": [
    "# let's test the model\n",
    "\n",
    "with tf.Graph().as_default():\n",
    "    \n",
    "    # placeholders\n",
    "    encoder_inputs = [tf.placeholder(dtype = tf.int32, shape = [None], name = 'encoder{}'.format(i)) for i in range(input_seq_len)]\n",
    "    decoder_inputs = [tf.placeholder(dtype = tf.int32, shape = [None], name = 'decoder{}'.format(i)) for i in range(output_seq_len)]\n",
    "\n",
    "    # output projection\n",
    "    size = 512\n",
    "    w_t = tf.get_variable('proj_w', [de_vocab_size, size], tf.float32)\n",
    "    b = tf.get_variable('proj_b', [de_vocab_size], tf.float32)\n",
    "    w = tf.transpose(w_t)\n",
    "    output_projection = (w, b)\n",
    "    \n",
    "    # change the model so that output at time t can be fed as input at time t+1\n",
    "    outputs, states = tf.contrib.legacy_seq2seq.embedding_attention_seq2seq(\n",
    "                                                encoder_inputs,\n",
    "                                                decoder_inputs,\n",
    "                                                tf.contrib.rnn.BasicLSTMCell(size),\n",
    "                                                num_encoder_symbols = en_vocab_size,\n",
    "                                                num_decoder_symbols = de_vocab_size,\n",
    "                                                embedding_size = 100,\n",
    "                                                feed_previous = True, # <-----this is changed----->\n",
    "                                                output_projection = output_projection,\n",
    "                                                dtype = tf.float32)\n",
    "    \n",
    "    # ops for projecting outputs\n",
    "    outputs_proj = [tf.matmul(outputs[i], output_projection[0]) + output_projection[1] for i in range(output_seq_len)]\n",
    "\n",
    "    # let's translate these sentences     \n",
    "    en_sentences = [\"What' s your name\", 'My name is', 'What are you doing', 'I am reading a book',\\\n",
    "                    'How are you', 'I am good', 'Do you speak English', 'What time is it', 'Hi', 'Goodbye', 'Yes', 'No']\n",
    "    en_sentences_encoded = [[en_word2idx.get(word, 0) for word in en_sentence.split()] for en_sentence in en_sentences]\n",
    "    \n",
    "    # padding to fit encoder input\n",
    "    for i in range(len(en_sentences_encoded)):\n",
    "        en_sentences_encoded[i] += (15 - len(en_sentences_encoded[i])) * [en_word2idx['<pad>']]\n",
    "    \n",
    "    # restore all variables - use the last checkpoint saved\n",
    "    saver = tf.train.Saver()\n",
    "    path = tf.train.latest_checkpoint('checkpoints')\n",
    "    \n",
    "    with tf.Session() as sess:\n",
    "        # restore\n",
    "        saver.restore(sess, path)\n",
    "        \n",
    "        # feed data into placeholders\n",
    "        feed = {}\n",
    "        for i in range(input_seq_len):\n",
    "            feed[encoder_inputs[i].name] = np.array([en_sentences_encoded[j][i] for j in range(len(en_sentences_encoded))], dtype = np.int32)\n",
    "            \n",
    "        feed[decoder_inputs[0].name] = np.array([de_word2idx['<go>']] * len(en_sentences_encoded), dtype = np.int32)\n",
    "        \n",
    "        # translate\n",
    "        output_sequences = sess.run(outputs_proj, feed_dict = feed)\n",
    "        \n",
    "        # decode seq.\n",
    "        for i in range(len(en_sentences_encoded)):\n",
    "            print '{}.\\n--------------------------------'.format(i+1)\n",
    "            ouput_seq = [output_sequences[j][i] for j in range(output_seq_len)]\n",
    "            #decode output sequence\n",
    "            words = decode_output(ouput_seq)\n",
    "        \n",
    "            print en_sentences[i]\n",
    "            for i in range(len(words)):\n",
    "                if words[i] not in ['<eos>', '<pad>', '<go>']:\n",
    "                    print words[i],\n",
    "            \n",
    "            print '\\n--------------------------------'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FRENCH TO ENGLISH TRANSLATOR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X, Y, en_word2idx, en_idx2word, en_vocab, de_word2idx, de_idx2word, de_vocab = data_utils.read_dataset('data_fr_en.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence in English - encoded: [3, 1492, 941, 28]\n",
      "Sentence in German - encoded: [7, 357, 18, 201]\n",
      "Decoded:\n",
      "------------------------\n",
      "- D&apos;abord dis- moi \n",
      "\n",
      "- Tell me first\n"
     ]
    }
   ],
   "source": [
    "print 'Sentence in English - encoded:', X[1]\n",
    "print 'Sentence in German - encoded:', Y[1]\n",
    "print 'Decoded:\\n------------------------'\n",
    "\n",
    "for i in range(len(X[1])):\n",
    "    print en_idx2word[X[1][i]],\n",
    "    \n",
    "print '\\n'\n",
    "\n",
    "for i in range(len(Y[1])):\n",
    "    print de_idx2word[Y[1][i]],"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def data_padding(x, y, length = 15):\n",
    "    for i in range(len(x)):\n",
    "        x[i] = x[i] + (length - len(x[i])) * [en_word2idx['<pad>']]\n",
    "        y[i] = [de_word2idx['<go>']] + y[i] + [de_word2idx['<eos>']] + (length-len(y[i])) * [de_word2idx['<pad>']]\n",
    "\n",
    "data_padding(X, Y)\n",
    "\n",
    "# data splitting\n",
    "X_train,  X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.1)\n",
    "\n",
    "del X\n",
    "del Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "input_seq_len = 15\n",
    "output_seq_len = 17\n",
    "en_vocab_size = len(en_vocab) + 2 # + <pad>, <ukn>\n",
    "de_vocab_size = len(de_vocab) + 4 # + <pad>, <ukn>, <eos>, <go>\n",
    "\n",
    "# placeholders\n",
    "encoder_inputs = [tf.placeholder(dtype = tf.int32, shape = [None], name = 'encoder{}'.format(i)) for i in range(input_seq_len)]\n",
    "decoder_inputs = [tf.placeholder(dtype = tf.int32, shape = [None], name = 'decoder{}'.format(i)) for i in range(output_seq_len)]\n",
    "\n",
    "targets = [decoder_inputs[i+1] for i in range(output_seq_len-1)]\n",
    "# add one more target\n",
    "targets.append(tf.placeholder(dtype = tf.int32, shape = [None], name = 'last_target'))\n",
    "target_weights = [tf.placeholder(dtype = tf.float32, shape = [None], name = 'target_w{}'.format(i)) for i in range(output_seq_len)]\n",
    "\n",
    "# output projection\n",
    "size = 512\n",
    "w_t = tf.get_variable('proj_w', [de_vocab_size, size], tf.float32)\n",
    "b = tf.get_variable('proj_b', [de_vocab_size], tf.float32)\n",
    "w = tf.transpose(w_t)\n",
    "output_projection = (w, b)\n",
    "\n",
    "outputs, states = tf.contrib.legacy_seq2seq.embedding_attention_seq2seq(\n",
    "                                            encoder_inputs,\n",
    "                                            decoder_inputs,\n",
    "                                            tf.contrib.rnn.BasicLSTMCell(size),\n",
    "                                            num_encoder_symbols = en_vocab_size,\n",
    "                                            num_decoder_symbols = de_vocab_size,\n",
    "                                            embedding_size = 100,\n",
    "                                            feed_previous = False,\n",
    "                                            output_projection = output_projection,\n",
    "                                            dtype = tf.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sampled_loss(labels, logits):\n",
    "    return tf.nn.sampled_softmax_loss(\n",
    "                        weights = w_t,\n",
    "                        biases = b,\n",
    "                        labels = tf.reshape(labels, [-1, 1]),\n",
    "                        inputs = logits,\n",
    "                        num_sampled = 512,\n",
    "                        num_classes = de_vocab_size)\n",
    "\n",
    "# Weighted cross-entropy loss for a sequence of logits\n",
    "loss = tf.contrib.legacy_seq2seq.sequence_loss(outputs, targets, target_weights, softmax_loss_function = sampled_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learning_rate = 5e-3\n",
    "batch_size = 64\n",
    "steps = 1000\n",
    "\n",
    "# ops for projecting outputs\n",
    "outputs_proj = [tf.matmul(outputs[i], output_projection[0]) + output_projection[1] for i in range(output_seq_len)]\n",
    "\n",
    "# training op\n",
    "optimizer = tf.train.RMSPropOptimizer(learning_rate).minimize(loss)\n",
    "\n",
    "# init op\n",
    "init = tf.global_variables_initializer()\n",
    "\n",
    "# forward step\n",
    "def forward_step(sess, feed):\n",
    "    output_sequences = sess.run(outputs_proj, feed_dict = feed)\n",
    "    return output_sequences\n",
    "\n",
    "# training step\n",
    "def backward_step(sess, feed):\n",
    "    sess.run(optimizer, feed_dict = feed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------TRAINING------------------\n",
      "step: 0, loss: 9.28539276123\n",
      "step: 4, loss: 9.27874851227\n",
      "step: 9, loss: 9.21777153015\n",
      "step: 14, loss: 9.17447471619\n",
      "step: 19, loss: 9.08433818817\n",
      "Checkpoint is saved\n",
      "step: 24, loss: 9.17327880859\n",
      "step: 29, loss: 8.97542762756\n",
      "step: 34, loss: 8.93514156342\n",
      "step: 39, loss: 8.60582542419\n",
      "Checkpoint is saved\n",
      "step: 44, loss: 7.64345264435\n",
      "step: 49, loss: 7.46297550201\n",
      "step: 54, loss: 7.42553424835\n",
      "step: 59, loss: 6.85999155045\n",
      "Checkpoint is saved\n",
      "step: 64, loss: 6.22163677216\n",
      "step: 69, loss: 6.94149065018\n",
      "step: 74, loss: 5.9013004303\n",
      "step: 79, loss: 6.42337560654\n",
      "Checkpoint is saved\n",
      "step: 84, loss: 6.68325567245\n",
      "step: 89, loss: 5.53163719177\n",
      "step: 94, loss: 5.89535140991\n",
      "step: 99, loss: 5.56395626068\n",
      "Checkpoint is saved\n",
      "step: 104, loss: 6.669090271\n",
      "step: 109, loss: 5.18537902832\n",
      "step: 114, loss: 5.21751308441\n",
      "step: 119, loss: 6.06572628021\n",
      "Checkpoint is saved\n",
      "step: 124, loss: 5.74199199677\n",
      "step: 129, loss: 4.8985004425\n",
      "step: 134, loss: 5.11765527725\n",
      "step: 139, loss: 4.85362434387\n",
      "Checkpoint is saved\n",
      "step: 144, loss: 5.96686315536\n",
      "step: 149, loss: 4.75355100632\n",
      "step: 154, loss: 4.90320491791\n",
      "step: 159, loss: 4.98559951782\n",
      "Checkpoint is saved\n",
      "step: 164, loss: 4.8721241951\n",
      "step: 169, loss: 4.90601301193\n",
      "step: 174, loss: 4.55758333206\n",
      "step: 179, loss: 4.69745731354\n",
      "Checkpoint is saved\n",
      "step: 184, loss: 4.91282558441\n",
      "step: 189, loss: 4.75536155701\n",
      "step: 194, loss: 4.53930854797\n",
      "step: 199, loss: 4.69556617737\n",
      "Checkpoint is saved\n",
      "step: 204, loss: 4.46311855316\n",
      "step: 209, loss: 4.69154882431\n",
      "step: 214, loss: 4.71703767776\n",
      "step: 219, loss: 4.56424999237\n",
      "Checkpoint is saved\n",
      "step: 224, loss: 4.32151889801\n",
      "step: 229, loss: 4.53644275665\n",
      "step: 234, loss: 4.08683681488\n",
      "step: 239, loss: 4.47070598602\n",
      "Checkpoint is saved\n",
      "step: 244, loss: 4.10653209686\n",
      "step: 249, loss: 4.18097639084\n",
      "step: 254, loss: 4.08025169373\n",
      "step: 259, loss: 4.11255216599\n",
      "Checkpoint is saved\n",
      "step: 264, loss: 4.02483797073\n",
      "step: 269, loss: 3.74560976028\n",
      "step: 274, loss: 3.86433005333\n",
      "step: 279, loss: 3.78443288803\n",
      "Checkpoint is saved\n",
      "step: 284, loss: 4.18633890152\n",
      "step: 289, loss: 4.07211256027\n",
      "step: 294, loss: 4.02365779877\n",
      "step: 299, loss: 3.92193460464\n",
      "Checkpoint is saved\n",
      "step: 304, loss: 3.78566169739\n",
      "step: 309, loss: 3.94309687614\n",
      "step: 314, loss: 3.57710909843\n",
      "step: 319, loss: 3.49600505829\n",
      "Checkpoint is saved\n",
      "step: 324, loss: 3.5001206398\n",
      "step: 329, loss: 3.73442959785\n",
      "step: 334, loss: 3.39053010941\n",
      "step: 339, loss: 3.49205160141\n",
      "Checkpoint is saved\n",
      "step: 344, loss: 3.40912842751\n",
      "step: 349, loss: 3.3374774456\n",
      "step: 354, loss: 3.10162401199\n",
      "step: 359, loss: 3.53169608116\n",
      "Checkpoint is saved\n",
      "step: 364, loss: 3.1286714077\n",
      "step: 369, loss: 3.17869520187\n",
      "step: 374, loss: 3.28132152557\n",
      "step: 379, loss: 3.47110843658\n",
      "Checkpoint is saved\n",
      "step: 384, loss: 2.97348976135\n",
      "step: 389, loss: 2.99404525757\n",
      "step: 394, loss: 3.00749135017\n",
      "step: 399, loss: 3.21795463562\n",
      "Checkpoint is saved\n",
      "step: 404, loss: 3.17987966537\n",
      "step: 409, loss: 3.05080652237\n",
      "step: 414, loss: 3.13049340248\n",
      "step: 419, loss: 3.16450929642\n",
      "Checkpoint is saved\n",
      "step: 424, loss: 3.11301994324\n",
      "step: 429, loss: 3.06595182419\n",
      "step: 434, loss: 3.11315250397\n",
      "step: 439, loss: 2.79540348053\n",
      "Checkpoint is saved\n",
      "step: 444, loss: 2.99316263199\n",
      "step: 449, loss: 2.91126585007\n",
      "step: 454, loss: 3.02163267136\n",
      "step: 459, loss: 3.02583026886\n",
      "Checkpoint is saved\n",
      "step: 464, loss: 3.06387996674\n",
      "step: 469, loss: 2.77448177338\n",
      "step: 474, loss: 2.93969774246\n",
      "step: 479, loss: 2.61235260963\n",
      "Checkpoint is saved\n",
      "step: 484, loss: 2.91573905945\n",
      "step: 489, loss: 2.93179750443\n",
      "step: 494, loss: 3.05287146568\n",
      "step: 499, loss: 2.86933422089\n",
      "Checkpoint is saved\n",
      "step: 504, loss: 2.73008728027\n",
      "step: 509, loss: 2.69594287872\n",
      "step: 514, loss: 2.61157798767\n",
      "step: 519, loss: 2.83751654625\n",
      "Checkpoint is saved\n",
      "step: 524, loss: 2.38444375992\n",
      "step: 529, loss: 2.57056212425\n",
      "step: 534, loss: 2.77804327011\n",
      "step: 539, loss: 2.52706885338\n",
      "Checkpoint is saved\n",
      "step: 544, loss: 2.60166049004\n",
      "step: 549, loss: 2.47282481194\n",
      "step: 554, loss: 2.35067510605\n",
      "step: 559, loss: 2.78017663956\n",
      "Checkpoint is saved\n",
      "step: 564, loss: 2.66314983368\n",
      "step: 569, loss: 2.53241872787\n",
      "step: 574, loss: 2.27059030533\n",
      "step: 579, loss: 2.39284229279\n",
      "Checkpoint is saved\n",
      "step: 584, loss: 2.59610390663\n",
      "step: 589, loss: 2.74109792709\n",
      "step: 594, loss: 2.2012386322\n",
      "step: 599, loss: 2.60526442528\n",
      "Checkpoint is saved\n",
      "step: 604, loss: 2.69269633293\n",
      "step: 609, loss: 2.30236911774\n",
      "step: 614, loss: 2.49125862122\n",
      "step: 619, loss: 2.25540208817\n",
      "Checkpoint is saved\n",
      "step: 624, loss: 2.21159434319\n",
      "step: 629, loss: 2.43232440948\n",
      "step: 634, loss: 2.24175238609\n",
      "step: 639, loss: 2.42402982712\n",
      "Checkpoint is saved\n",
      "step: 644, loss: 2.43022251129\n",
      "step: 649, loss: 2.7410697937\n",
      "step: 654, loss: 2.31277370453\n",
      "step: 659, loss: 2.70981431007\n",
      "Checkpoint is saved\n",
      "step: 664, loss: 2.33857488632\n",
      "step: 669, loss: 2.22019314766\n",
      "step: 674, loss: 2.33188533783\n",
      "step: 679, loss: 2.4267539978\n",
      "Checkpoint is saved\n",
      "step: 684, loss: 2.31027078629\n",
      "step: 689, loss: 2.31504440308\n",
      "step: 694, loss: 2.36981678009\n",
      "step: 699, loss: 2.41134119034\n",
      "Checkpoint is saved\n",
      "step: 704, loss: 2.56794166565\n",
      "step: 709, loss: 2.07330822945\n",
      "step: 714, loss: 2.4018239975\n",
      "step: 719, loss: 2.01198792458\n",
      "Checkpoint is saved\n",
      "step: 724, loss: 2.38971138\n",
      "step: 729, loss: 2.19255805016\n",
      "step: 734, loss: 2.36334133148\n",
      "step: 739, loss: 1.92631745338\n",
      "Checkpoint is saved\n",
      "step: 744, loss: 1.93238246441\n",
      "step: 749, loss: 2.33020734787\n",
      "step: 754, loss: 2.32619810104\n",
      "step: 759, loss: 1.92145323753\n",
      "Checkpoint is saved\n",
      "step: 764, loss: 2.05035543442\n",
      "step: 769, loss: 2.42395544052\n",
      "step: 774, loss: 1.76639914513\n",
      "step: 779, loss: 2.2078127861\n",
      "Checkpoint is saved\n",
      "step: 784, loss: 1.87164831161\n",
      "step: 789, loss: 1.97230243683\n",
      "step: 794, loss: 2.16967964172\n",
      "step: 799, loss: 1.89871299267\n",
      "Checkpoint is saved\n",
      "step: 804, loss: 2.15229415894\n",
      "step: 809, loss: 1.81991398335\n",
      "step: 814, loss: 1.9209485054\n",
      "step: 819, loss: 2.09761977196\n",
      "Checkpoint is saved\n",
      "step: 824, loss: 2.15506863594\n",
      "step: 829, loss: 2.02311182022\n",
      "step: 834, loss: 1.94417488575\n",
      "step: 839, loss: 1.89302527905\n",
      "Checkpoint is saved\n",
      "step: 844, loss: 1.97490477562\n",
      "step: 849, loss: 1.67890191078\n",
      "step: 854, loss: 1.98289752007\n",
      "step: 859, loss: 1.97535276413\n",
      "Checkpoint is saved\n",
      "step: 864, loss: 1.83273386955\n",
      "step: 869, loss: 2.17665696144\n",
      "step: 874, loss: 1.88289237022\n",
      "step: 879, loss: 1.89305746555\n",
      "Checkpoint is saved\n",
      "step: 884, loss: 1.822671175\n",
      "step: 889, loss: 1.8908097744\n",
      "step: 894, loss: 1.79941487312\n",
      "step: 899, loss: 1.75234985352\n",
      "Checkpoint is saved\n",
      "step: 904, loss: 1.97990202904\n",
      "step: 909, loss: 1.91769170761\n",
      "step: 914, loss: 2.0175318718\n",
      "step: 919, loss: 2.00002670288\n",
      "Checkpoint is saved\n",
      "step: 924, loss: 1.96078872681\n",
      "step: 929, loss: 2.08117103577\n",
      "step: 934, loss: 2.11363697052\n",
      "step: 939, loss: 2.12364339828\n",
      "Checkpoint is saved\n",
      "step: 944, loss: 1.64213693142\n",
      "step: 949, loss: 1.85605311394\n",
      "step: 954, loss: 1.72118997574\n",
      "step: 959, loss: 1.79674077034\n",
      "Checkpoint is saved\n",
      "step: 964, loss: 1.55656778812\n",
      "step: 969, loss: 1.80174946785\n",
      "step: 974, loss: 1.75626337528\n",
      "step: 979, loss: 1.60011386871\n",
      "Checkpoint is saved\n",
      "step: 984, loss: 1.80630505085\n",
      "step: 989, loss: 1.63899350166\n",
      "step: 994, loss: 1.90044999123\n",
      "step: 999, loss: 1.61265099049\n",
      "Checkpoint is saved\n",
      "Training time for 1000 steps: 3744.46671796s\n"
     ]
    }
   ],
   "source": [
    "losses = []\n",
    "\n",
    "# save a checkpoint so we can restore the model later \n",
    "saver = tf.train.Saver()\n",
    "\n",
    "print '------------------TRAINING------------------'\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    \n",
    "    t = time.time()\n",
    "    for step in range(steps):\n",
    "        feed = feed_dict(X_train, Y_train)\n",
    "            \n",
    "        backward_step(sess, feed)\n",
    "        \n",
    "        if step % 5 == 4 or step == 0:\n",
    "            loss_value = sess.run(loss, feed_dict = feed)\n",
    "            print 'step: {}, loss: {}'.format(step, loss_value)\n",
    "            losses.append(loss_value)\n",
    "        \n",
    "        if step % 20 == 19:\n",
    "            saver.save(sess, 'checkpoints/', global_step=step)\n",
    "            print 'Checkpoint is saved'\n",
    "            \n",
    "    print 'Training time for {} steps: {}s'.format(steps, time.time() - t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbIAAAEiCAYAAACcFVdfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd81PX9B/DX93v3vX25yyaDMBPCSNjIkCkOUFtw1Alo\nq7UiCor1h6O22jqqFax1tdJa66pFbK0VZTjYYclURhghJCFkXpLL7bvv748LR0ISCCS5yzd5Pf/R\n3H3ve+/7PsK98vl8P0Ow2WwyiIiIFEqMdAFEREStwSAjIiJFY5AREZGiMciIiEjRGGRERKRoDDIi\nIlK0sAXZvHnzkJ6ejrFjx4Yee/LJJzFq1ChceumlmDVrFqqrq8NVDhERdRJhC7LbbrsNy5cvb/DY\nlClTkJOTgw0bNqBPnz5YsmRJuMohIqJOImxBNmbMGFit1gaPTZo0CaIYLGHEiBEoLCwMVzlERNRJ\ndJh7ZO+99x4uv/zySJdBREQK0yGC7A9/+AMkScKNN94Y6VKIiEhh1JEu4P3338fq1avx3//+N9Kl\nEBGRAoU1yGS54frEa9aswZ/+9CesWLECWq02nKUQEVEnEbauxbvuugtXXnklDh8+jEGDBuG9997D\nI488ArvdjhkzZmDChAlYuHBhuMqhFsrNzY10CV0Sr3tk8LorU9haZEuXLm302O233x6utyciok6q\nQwz2ICIiulgMMiIiUjQGGRERKRqDjIiIFI1BRkREisYgIyIiRWOQERGRojHIiIhI0RhkRESkaAwy\nIiJSNAYZEREpGoOMiIgUjUFGRESKxiAjIiJFY5AREZGiMciIiEjRGGRERKRoDDIiIlI0BhkRESka\ng4yIiBSNQUZERIrGICMiIkVjkBERkaIxyIiISNEYZEREpGgMMiIiUjQGGRERKRqDjIiIFI1BRkRE\nisYgIyIiRWOQERGRooUtyObNm4f09HSMHTs29JjNZsPMmTMxYsQIXHfddaiqqgpXOURE1EmELchu\nu+02LF++vMFjS5YswaRJk7B9+3ZMmDABS5YsCVc5RETUSYQtyMaMGQOr1drgsRUrVuCWW24BANxy\nyy34/PPPw1UOERF1EhG9R1ZaWoqEhAQAQGJiIkpLSyNZDhERKVCHGuwhCEKkSyAiIoVRR/LNExIS\nUFJSgoSEBJw6dQrx8fHnfU1ubm4YKqP6eM0jg9c9Mnjdwys9Pb3V5whrkMmy3ODnadOm4YMPPsCC\nBQvw4YcfYvr06ec9R1t8aGq53NxcXvMI4HWPDF53ZQpb1+Jdd92FK6+8EocPH8agQYPw3nvv4cEH\nH8Q333yDESNGYO3atViwYEG4yiEiok4ibC2ypUuXNvn4p59+Gq4SiIioE+pQgz2IiIguFIOMiIgU\njUFGRESKxiAjIiJFY5AREZGiMciIiEjRGGRERKRoDDIiIlI0BhkRESma4oLsQKU30iUQEVEHorgg\ne3RrFV7fV4MabyDSpRARUQeguCB7e1IMarwybltTjje+t8PtD66o7/AFsL3UE+HqiIgo3BQXZFat\niP8bGoW3Jsag2OHHoi02lDj9eHizDYtybDjp8Ee6RCIiCiPFBdlpiQYVnhwRhWSDCreuKcegaAk3\n9jHgo8OOSJdGRERhFNEdoltLJQhYONiMaWl6DIxWo9ItY8435ZiVYUCsThXp8oiIKAwU2yI7TRQE\nDIqRIAgCYnQipqbqsHR/LRw+DgYhIuoKFB9kZ5uVbkStT8bNq8vxZb4z0uUQEVE763RBFqMT8fRI\nC34zwoJ/HeH9MiKizq7TBdlpg+MklLkCKHFyFCMRUWfWaYNMJQgYmaDB1hLOLSMi6sw6bZABwCUJ\nGmw5xSAjIurMOnWQjUzQ4rsyD7wBOdKlEBFRO+nUQRatFdHdpMK+Ci40TETUWXXqIAOAYXEa7Cln\nkBERdVadPsgS9SLKXBy5SETUWXX6IIvRqVDu4iofRESdVacPslidiHI3g4yIqLPq/EGmFdkiIyLq\nxDp9kMXoRNjcAQRkDsEnIuqMOn2QSaIAgySgysMgIyLqjDp9kAGnuxc5cpGIqDPqEEG2ePFijB49\nGmPHjsXdd98Nj6dtl5XigA8ios4r4kGWn5+Pd955B+vWrcOmTZvg8/mwfPnyNn2PWK0KFRzwQUTU\nKakjXUBUVBQ0Gg0cDgdEUYTD4UBSUlKbvkeMjiMXiYg6q4i3yKxWK+bNm4dBgwahf//+sFgsmDRp\nUpu+R6xORBmDjIioUxJsNltEh/Pl5eXhpptuwpdffomoqCjMnj0bM2bMwI033tjk8bm5uRf8Htuq\nJWyvlnBvKneMJiLqSNLT01t9joh3Le7cuROXXHIJoqOjAQDXXnsttm7d2myQXcyHdpZ7sPEHO9LT\nU1pVa1eUm5vbJr9odGF43SOD112ZIt612LdvX2zfvh0ulwuyLGPt2rXIyMho0/fgqEUios4r4i2y\nrKws3HzzzZg0aRJEUUR2djbuuOOONn2P06MWZVmGIAhtem4iIoqsiAcZADzwwAN44IEH2u38OrUA\nSRRg98owaxhkRESdScS7FsMlht2LRESdUpcJMq6CT0TUOXWZIEs0qHC02hfpMoiIqI11mSCb2VOP\nj4444PZzFXwios6kywRZZrSE/lYJ/znmjHQpRETUhrpMkAHATzON+OfhWhQ7uKULEVFn0aWCrFeU\nGjf0MeDnayvw+FYbuxmJiDqBLhVkAHBbuhH/ujwOxY4ADtm8kS6HiIhaqcsFGRCcIJ1hUSOvhl2M\nRERK1yWDDAB6mtXIs3M4PhGR0nXhIFMhj/PKiIgUrwsHGbsWiYg6gy4bZAl6EU6/jBoPl60iIlKy\nLhtkgiCgh0mFvBp2LxIRKVmXDTKA3YtERJ0Bg4wtMiIiReviQabCsXpBllvlxVPbqyJYERERXagu\nHmRqHK/XtXi4yofcKrbQiIiUpEsHWYJehMMno8YbHLl40uFHJXeRJiJSlBYH2bp165CXlwcAKC4u\nxi9+8QvMnTsXp06daq/a2p0gCEg1qVBoD7bKimr9qPXJXEyYiEhBWhxkDz/8MFQqFQDgiSeegM/n\ngyiKmD9/frsVFw6pRhVO1AaDrNgRbI2xVUZEpBzqlh548uRJdO/eHT6fD1999RX27t0LjUaDzMzM\n9qyv3aWaVCioW3PxpMOPaK2ISncA3Qyq0DH/PFyLdIuE4fGaRq/3BWSsKXDhqjR92GomIqIzWtwi\nM5vNKCkpwcaNG5GZmQmTyQQA8PmUPTiiu1GFglo/3H4Z1d4AMizqBi0yb0DGB4cdONjMli+nnH78\nca89XOUSEdFZWtwi+/nPf44pU6bA4/HgueeeAwDk5OQgPT293YoLh1STGh8fdaLY4Uc3vQqxOhEV\n9YJsa4kH1R4ZtmaWsrK5ZTj9MnwBGWpRCFfZRERUp8VBtmDBAlxzzTVQqVTo1asXACA5ORmvvPJK\nuxUXDql1LbIihx9JRhWitQ2DbNUJFwZGq1HlbnoAyOnWm90rw6plkBERhdsFDb/v0aMHTp48iU8+\n+QQAkJSUhJ49e7ZHXWETpREhicD3FV4kGVShe2QAUOMNYHupBz/uaWi+ReY5HWQcIEJEFAktDrLv\nv/8ew4cPx/z583H//fcDADZu3Ih58+a1W3HhkmpUYVupp1GQrStyY3i8Bt1NqmaDLNQi83HIPhFR\nJLQ4yBYuXIjHHnsM27Ztg1od7JEcN24ccnJy2q24cEk1qXHI5kOSQURMvSD7vtKLEfEaWDUibM0M\nybfV61okIqLwa3GQ7d+/HzfddBOA4ERiADAajXA6ne1TWRh1N6ogA0g2qhBT7x7ZkWof+kSpYdEK\nqGLXIhFRh9TiIEtLS8OuXbsaPLZjxw707t27zYsKt1RTcM5Yt7quxQpXAH5ZxvEaH3pFqaBXCfDL\ngKuJ7sNKdwCJepEtMiKiCGnxqMXHH38cP/nJT3DnnXfC6/Vi8eLF+Nvf/qb4UYsA0N2ohlkSYJZE\nyHJwiapj1T7EaEUY1MGst2pE2DwBdFOrGrzW5g4g1ahii4yIKEJa3CK76qqr8PHHH6O8vBzjxo3D\niRMn8N5772HKlCmtLqKqqgpz5szBqFGjMHr0aGzfvr3V57wQvaNUeHG0FUCw2zRaK2J7qRd9oqTQ\nMVat2OSAj0qPjO4mNVtkREQR0uIWGQAMHjwYL730UpsXsWjRIlx++eV455134PP54HA42vw9zkUQ\nBGRGnwmtaK2IHaUeDIg+c3msGhFVZw34CMgyqj0BpNTNRSMiovBrcYvs1VdfxZ49ewAA27Ztw6BB\ng5CdnY2tW7e2qoDq6mps3rwZt99+OwBArVYjKiqqVedsrRitiN3lHvSJOhNkFo3QqEVW45VhVAuw\nasVGXYt5NT58X9H0slZERNR2Whxkb7zxBnr06AEAeOqppzB37lw8/PDDePTRR1tVwPHjxxEbG4u5\nc+diwoQJmD9/fsRHQkZrRXgCQO96QWbVirCdtbqHzR2AVSvCJAmNuha/KnTh7YNcg5GIqL0JNput\nRTd3unfvjhMnTqCmpgZZWVk4cuQIVCoV0tLSkJ+ff9EF7Nq1C1OnTsXq1asxdOhQLFq0CFFRUXjs\nsceaPD43N/ei36ulPinR4usKLV7pV43Tyyd+XqaFKwBcn+AOHXewVoVPS3W4LsGFj0t0WNSzNvTc\n34r02FYtYUlGNXTN/LkQkIGNNgnjo9lyI6KuqS3W623xPbKUlBRs2bIFBw4cwNixY6FSqVBdXR3a\no+xiJScnIyUlBUOHDgUA/PjHP8bLL7/c7PHhWKS4j8qB4wEX+mWcea++Gid+qPQiPT0KDl8ABrWI\nwiIXkrxu9O+dAG95FdLTk0PHu0sroRZ9qIpKQ1aStsn3KXX68d7BctwxMh6i0DHXaczNzVX8wtBK\nxOseGbzuytTiIHv66acxZ84cSJKEd999FwCwcuVKDB8+vFUFJCQkICUlBYcPH0bfvn2xdu3aiO9x\nNihGgvaslewtdat7yLKMO7+pwFMjLah0BxDdTNdiqTOAy1K02FLixqXNBJnNE0BABmq9Msyajhlk\nREQdXYuD7IorrsCBAwcaPDZjxgzMmDGj1UX8/ve/x9133w2v14uePXvitddea/U5WyPTKiHTKjV4\nzKoJru6Rb/fjlDOAzcXu0OMmdcPBHrIso9Tlx7U9o/CrrVWQZTm0Gkp9p5e3qvIEYNZc0PrNRERU\np8VBduDAAcTExCAhIQF2ux2vvPIKRFHEAw88AEmSzn+Cc8jKysI333zTqnO0t+A8MhnflXmQYlRh\na4kHGVYJvcwqaFXB+11uvwytSkCNV4ZKENDPooYkBpe66mtpfI0q6waPVHlkpIb7AxERdRItbgb8\n7Gc/Q1VVFQDgV7/6FTZt2oTt27djwYIF7VZcR3J64eDvSr24Ld2AfLsfeTU+WLUiBEGAWRJCrbJS\nZwAJ+uDjw+M12FXe9GCO0+s3NreOIxERnV+LW2QnTpxAeno6ZFnGZ599hi1btkCn02Hw4MHtWV+H\nYZIEuP3BFtn8bBM2xknYWOzBTzONdc8H11uM1QElLj/i9cFBMClGFUocTU+WrnQzyIiIWqvFLTKt\nVouamhrs2LEDqampiI2NhVarhdvtPv+LOwFBEBClERGnExGnU2FUQnAAh7Xu3lb9AR+nW2QAEK9X\nocTV/Mr5RnXzK+sTEdH5tbhFdsMNN+BHP/oR7HY77r77bgDA7t27kZaW1m7FdTRWjYDsWA0A4JKE\n4H+t2tNBdmbAR4nTj3hdsEWWoBNR4my6RWZzB9DDrEKVh+s0EhFdrBYH2XPPPYevv/4aarUaEyZM\nAACIoohnn3223YrraBL0KoyqC7BuBhV+O9KCKCk4GrFBi8wVwJDY4OCOeL0Kpc5mdpf2BNDLrGaL\njIioFS5o0eApU6bgxIkT2Lp1K5KSkkKTmLuKp0daINXrjB1fb36YqcFgDz/idToAQJwuuOO0LyBD\nfdbcNJs7gB5Jauyp8LR/8UREnVSL75EVFxdj+vTpGD58OGbNmoVhw4Zh+vTpOHnyZHvW16FoVEKT\n88GAuq5F35l7ZPF198jUogCL5syu0/XZ3DJ6smuRiKhVWhxkDz30EAYNGoRjx47h4MGDyMvLQ1ZW\nFh588MH2rE8xzJKAGo8cmgx9OsgAIEEvNupedPlk+GQZ3Qwqdi0SEbVCi7sWc3JycPDgwdDkZ6PR\niKeffhr9+/dvt+KUxCSJKHJ4Q5OhT+8sDdSNXHT6MRBnJkXbPMHlrSwakUFGRNQKLW6RWa3WRktU\n5ebmwmKxtHlRSnR6sEf9ofenxetFlJzVIrO5A7BqRJg1wdf5ZXYvEhFdjBa3yObPn48ZM2Zg1qxZ\noS1d3n//fTz++OPtWZ9imCQBpxx+rDzhDE2GPi1Bp0KJq+EQfJsnuJeZShBgUge7Ja1aLhxMRHSh\nWtwimzNnDv72t7+hvLwcX375JcrLy7F06VIUFha2Z32KkahXocjhR7VXxj39TQ2ea+oeWaU7gOi6\nydTsXiQiungXNPx+4sSJmDhxYuhnt9uNmTNnslUGoIdZjU+vim/yueBcsmCL7N/HHJicrAvtLg0w\nyIiIWuOCgqwpMu/tnFdC3T2yArsPf9xrh8cP2DwyrHV7kFm0AofgExFdpFZvgtXcvCo6I0YbbHEt\nO+pEVoyE1QWu87bI+AcCEVHLnLdFtnbt2maf83qb3p6EGlKLAqxaEV+ecOKdybGYt6ESeyo8mJwS\nXBmkfpC5/TKe31mNWp+MZ0dZGq0GQkREDZ03yO6///5zPp+ayi0hWyJBL2JQjIRuBhUuS9HhoyOO\n0Mr5Fo2ASreMGk8Aj26pCk2mfu17O34xwIStJR5kxUihFhwA+GUZ/8tz4pqeeqjYKiaiLuy8QbZn\nz55w1NHpTU3VITtGqvt/bTDI6nUt5tV48er3dqSZVXh4sBkOn4y56ytx3coy6FQCZvbS4/YMY+h8\nm4o9WLLXDkEQ8KOe+oh8JiKijqDVgz2oZa7rZQj9f98oNe7pb0Sc7kyQbS/1QCUAb0+OgSgIMEkC\n3hgfDZdfxqEqHz467GgQZB8fdeC2dAPePmDHxGQtLJpW3+4kIlIkfvtFgCAIuCXdGLr/ZdGIKHMF\nMD/L3GBpK6MkIlanwuBYCQdtPjjrFiU+ZPOiqNaPO/sZMTlFh6X77RH5HEREHQGDrAPoFaXCg9lm\njO2mbfJ5g1pEhlWNPeXB7V6WH3ViZi891KKAOzONWFvkDm3e6Q3IqK43ArLWG8BXhS78aW8NCuy+\n9v8wRERhxiDrAAxqET8+z32uEfEabCv14EClF1tL3LimR/B4syTi8lQdPs1zAgCW7KnBr7ZVhV73\nm+3V+Py4E3avjBd313BYPxF1OgwyhRgRr8GWEg+e21mNeVlmRNW7Jzazlx7/O+7Ed6UebC3x4ITd\nj+M1PuTbfThc5cXzl1jxyFAznD4ZqwpcEfwURERtj4M9FCLDqobNHcDQOA2mJDfsgkw1qTEoRsJj\nW214eHAU8mp8+O9xJwQA09L00KiC9+IeGmzGo1uqMK6bFqa6ra4DsgyRw/eJSMHYIlMIlSDg8WFR\nWDjY3ORqKrf2NWJ8khaXpWhxTQ89Vhe4sOqEC9fW67LMtErIjpHwVV2r7HCVF3d+U8HuRiJSNAaZ\ngoxObH6Y/cAYCY8Ps0AQBHQzqDAgWsKAGAlJhoZbylzdQ4fP84NBtuyoE8ftfhTXrcy/v9KL5Ucd\n7fshiIjaGIOsk3oo24yHss2NHh8er0GVJ4AtJW5sLHYjO0bCgcrgUmPfFLoYZESkOAyyTipBr0LC\nWRt8AsEuyqu66/DbHdWYkqzDqAQN9tuCQbar3ItiRwAnHf5GryMi6qgYZF3QtDQ9XD4Z1/fWI9Mq\n4UClD3ZvACfsfozrpsXOMk+kSyQiajEGWRfUzaDCJ1fGoYdZjX5WNQ5V+bCn3IvMaDUuSdTgu1IG\nGREpB4Osizo9D82sERGvE/FpnhODYzUYGifhuzLvRY9kDMgyAhwFSURh1GGCLBAIYMKECbj55psj\nXUqX08+qxpYSDwbHSkg2qCCJwI4yL+7fUIl/FusuKNSWH3ViwUYbPH6GGRGFR4cJsjfeeAOZmZmR\nLqNL6h8tQRKBAdESBEHAsDgNFuXYMDxeg1ynGn89UIudZR785Qc7imrPPRBkR6kHJS4/Xt7L5bCI\nKDw6RJAVFhZi9erVmDVrVqRL6ZKGxWlwWYoO2roVQG7ua8BLY624o58RC7rXIueUB69/b0eNN4AH\nNlbiaHXTiw8HZBn7Kr1YPCYaB20+vLCrJrSYMRFRe+kQQfbYY4/h6aefbnLFCmp/vaLUWDQ0KvRz\nD7Mag2M1AACzWsbSSTF4a2IMFg6Owr0DTFi4qRI13kCj8+TV+GHViEg2qrBkrBVWrYiffVuBlSec\nYfssRNT1RHytxZUrVyIhIQHZ2dlYv379ebujcnNzw1QZnVb/mqcB6KEx4JNd+Rhr9TY47ttKDXqo\nVaHjp0hA/+4iXtrjR1VpMYaaz7TkbF4BVoldj+fC3/XI4HUPr/T09FafI+JBtmXLFnzxxRdYtWoV\nXC4X7HY77rnnHvz5z39u8vi2+NDUcrm5uY2u+TV6F9YUuDAn3drg8Y92VOHSXhqkp51Z3zEdQGKq\nF/+XY0N2LysyoyVsK3Fj0ZYqfHpVXGjxYmqoqetO7Y/XXZki/i3y5JNPYt++fdi9ezf++te/Yvz4\n8c2GGHUMY7tpsLfCixpPAAV2H/78gx0un4y9FV5kxUiNju9nlTA/y4xndlajxhvAK3vtiJIE7Cn3\nNnF2IqILE/EWGSmPQS1ieLwGqwpc+Oy4E1qVgC2n3PD4ZaQYGy+LBQCTU3TYWOzGPWsr0dOsQv9o\nHXaWeZrdFZuIqKUi3iKr79JLL8U///nPSJdBLTA5WYvX9tkxIFrCm+OjMSlFhympunMO2JmfZUai\nQcR9g0wYFqfBd2VNt8g4qZqILgRbZHRRxiRqcVNfA+7oZ4QgCJidYTzva8waEUvGRgMAEvUyih1+\n2NwBWLUN/5760z47orVii85JRNShWmSkHDq1gHsGmEJzzy6UWhSQFSNhV3nDdR1rvQF8ke/CxmJ3\nW5RJRF0Ag4wiZmicBjvP6l5cU+jGsDgJBfZga42I6HwYZBQxIxM0WFfkCq0UIssyPstzYmYvPQbH\nStjRwu1kuBQWUdfGIKOI6R2lxrxBZjy0qRKfHHXgg8MOOHwBDI/XYESCBttLgkF2stYPb6DpsNpd\n7sGDm2zhLJuIOhgGGUXUZak6/HakBfttXhTV+vFgthmiIGBkvAbbSj3YU+7Bnd+W4/9ybLA3sSzW\nrjIv9lV44W5mtf1X9tbgWDNrQxJR58BRixRxWbEaZNWt7XhaqlEFtQg8trUKT420YHOxBw9sqMTv\nRlmRXG+u2g+VXgQAHLB5Q+tD1n/uk2NOJBlU6BXFX3WizootMuqQBEHAzX0MWDQkCpckaDE/y4Sr\ne+gxd30F1p8MjmiUZRn7K72YkqzF3iZWCXnnYC0yrWocrmKLjKgz45+p1GHN6GUI/b8gCLi+twGZ\nVgmLttiQYYmBJyBDrxYwMVmH/x1vuML+/kovjlb78OTwKLy81x7u0okojNgiI0UZGCNhaqoOK/Kd\n+KHSiwHREgbFSNhX4YW/bvRirTeAxXtqMDvDiAyrhBN2H3esJurEGGSkONf20OPzfBf2lAeDLFor\nIkYrIq/aD7dfxuNbq9DfKuGaHsHNQlOMKuTVNO5ezKvxYVGODasLXM0OFiGijo9BRorTO0qNRL2I\nVQUuDIgOrrafFSvhX0cdmLu+EtFaEfOzTaF1H/ta1Mht4j7Zh4cdMEoCvsh3Yv7GytAQf29AxqZi\nN579rhrLjjgYckQdHIOMFOmaHsE9z9Itwdu8oxI0OFzlwx39jHhyeBRU9RYvTrdIOHzWEPwylx8b\ni92Yn2XGS2OsiNWJeGu/HSVOP36xrhIf5DrQz6rGrnIPbv+qHP877oQvIGNjsRsv76nhJGyiDoSD\nPUiRpqToEKURoalb63FSsg6TknVNHts3So11Jxuu3fjvY05MrTsHADwyJAp3fVuB1SdcuLmvET/p\now8NMPm+wou/HrDjtX12JBtVqHAH8KOeevQ+x5B+vyxje4kHawpd+FmmCd0MTW9vU9+BSi/UItDX\n0nhPNyJqHoOMFEmrEjCuhXuZ9bWocaTKh8JaHw7ZfDhh9+Oz4068fml06BiLRsQzoyyo9sgYkdBw\nPtrAGAmLx0ajqNaPbgYRf9pnx+ZTbvSOUsPlk1Hs9KOnWQ1ZlvHqPjvWFLrg9MnoaVZDrxawusCF\nWedZyf+kw49HttiQoFPhLxOjz3ksETXErkXq9KI0IuJ0IuZvtOGrQhdcfhmLhkQh1dTw77gMq9Qo\nxOpLNqogCgLGJGqw+VRw+az3D9firm8rsOqECx8edmBPhRdLJ8bgv1fF4y8TY3BnP2OD1mCp09/o\nvC6fjF9vq8JtfY0QBTRqPRLRubFFRl3C25NjoBYvbsuZsw2O1eBYdTWKav34NM+Jp0da8Me9NQjI\nwOvjoxGvP9ONmBUroczpR1GtH8UOPx7ebMMnV8bBqhWRV+PDb7ZXoajWjykpOvykjx49o1R4fZ8d\nj6W2SalEXQKDjLqEtgoxINitOTROwm+2V2FUvAZju2kxIFqC2y83CDEAUAkCLk3S4psiF74udMOq\nFbGt1IPLU3X4Mt+FEfEa/HyCKXSvb1S8Bu9qRGy0SejXzPtXewKhe3tExK5FoosyJlGLQ1U+3Joe\nvPdl1YpIbGZAx8QkHf5xqBZGtYA7+hmx5ZQbsixjQ7Ebl6fqQiEGBFcweTDLjE9KdShpohtyV5kH\nt39djsBZoyZlWW7RSEo/R1tSJ8QgI7oIE5K1WJhtPufIxdOGxElI1Ktw3yATRidqsLXUg7ya4OTt\nDEvj1/exqDE1xoPf76xuFE7/PuZEtUdGYW3DkHv7YC3+sr/2nHWUufy4fmVZaP+38yms9bXLNANv\nQEaBnetfUtthkBFdBLMk4tqe+hYdqxYFvDM5Bv2sEhL0KsRqRSw9YMe4btrQpO2zXRXrhl8GZn9d\ngXcP1cJJnaJvAAAZCElEQVTtl1Hq9OO7Mg9GJWiwv/JMENi9AfznmBMr8p3nXIprX4UXalHAE1tt\nqPace/ftArsPP/22ArvOWozZ45dbHITN+arAhV9tqwr9vL3Eg60lHOBCF49BRhQG9QNrdKIWG4s9\nuPQc0wdUArBkrBWLhkbhoM2LhZtt+OdhBy5L0WFYnISDtjMB87/jLoxK0KCXWY2cU80Hwg8VXszs\npcelSVr8dkd1o+7J0wKyjD/sroFeJTQILYcvgEe32LBgY2Wzr22JTaeCLdLauv3l/nvciU/znOd5\nFVHzGGREYTY6UQOTJGBw3LknPguCgIExEp4eacGAaDWWH3Pixz31yLRK2F8XZN6AjOVHHfhJHwOu\n6q7DFydckGUZq064Gt1j+6HShwHREn7e3wSnX8a/jjiafN/Pj7vgDsiYnWEMBZkvIOPhzTYkGlQw\nSgLyahrfv2sJt1/GjlIP0kwq7LcFuy73lnuwu9zL+3d00RhkRGE2OFaDdybHQGrhSEpREDB3oBnv\nTolBryg1MqxqHK32wReQ8VWBC91NKmRYJUxM1mFvhRdPbqvGmz/Y8evtVQ3Wjzxc7UU/qzrYvTgs\nCv887MABW8OuQ7dfxt8P1uKhbDP6WtQ4WrfY8pFqH2q9Mn452IwhsRrsLfdc1GffVe5B7yg1Ridq\n8UOlFwW1fmhUAmK1Yqv2jfvHwVpsOUdrlDo3BhlRBMTqzr9k1dm6103gNqhFdDOokFvlwz8OOTC7\nbtUQvVrA9O46mDUCPrgsFtEaEW/9ENyL7XCVDylGNQzq4D/5bgYV7h1owmv7Gu7VtrrAhb4WNdIt\nEnqZ1cir9iMgy9hb4UV2rARBEJAdK2F3RTAA7d5Ak6Mrm7Op2IOxiRoMjFbjhwov9pR7kRUjYWic\nBjvLGm+O2hIHbV68fbAWG4svLlxJ+RhkRAqUaZXw2j47uhlEDIk7sxrJ3EFmPDIkCjq1gEVDo7D2\npBsbTrrr9m5rOEJySooO+XYfTtaNgPTLMv552IFb+gY3NDVrRBglAaccAewt9yIrJvg+2bES9pR7\nIcsyXtlrx+92VDeqz+VrPD3A7g1gU7E7OO8uRsIPlXVBFqvBkDgJu8qaDqJzjZwMyDJe3luDK7rr\nkFt9cUFIyscgI1KgTKsa+yq9uKNf82s4RmlEPDncgpd2V2NtkTu05c1pkihgcrIOawpdAIANJ90w\nawQMjj1zXO+oYPfi3govsuoeT66bL7e1xIMtJW4U1Pob7Pfml2X89NsKrCkInrfU6cePV5bhJ6vL\n0dOsRppJhTidCjq1gHUn3ciOkYLdlRVe+AINQ2t7qQczVpZhX0XjkCp2+LFkTw1EAPMGmXCs2he6\nz7a6wAWH78zIzMLa1o20LLD7sGBjZavOQe2HQUakQKMStPhJbz2yY5tfGxIILnh8a7oReyq8jYIM\nAC5P1WF1gQsVrgBe3WfHz/qZGoyw7G1WY2OxG6IAdNMHvy5Ody8+t7Ma1/cyYHqaDv87fmbU4eZi\nD5x+Ge8cqoVflvHuIQeuTtNjxfR4vDjGGjr/wGgJkgj0MKuCE8r1Ig7Vu0+2o9SD3+6ownW9DPjN\n9qrQOpWV7gBe3FWNn6+tgF4t4OmRFpglETFaFQrsfrh8Ml7YVY0dpcHwO17jw6yvKnCibu7a9xXe\nJge6uHwyqnwCnL7GLcCtpZ4mg5Y6BgYZkQIlG1WYO8jcomNv6K3Hb0dakGZqfF9uQLQavoCMBzdV\n4qo0XaNFk3tHqfBVoQtZMVKDgBscI8EnA9f11uPqND1W1dtl+5NjDtw30ASLJOKDXAe+LXLh5rru\nyvoGxkjIipUg1p13SooOS/fb4QvIyK3y4ukdVXhqhAVz+hkxs5ce8zZU4t71FZjzTTmMagHvT43F\n3IHm0P3G9LoNVHeVe+ANALvrBqTsKvNAUgU3Ug3IMpbsqcEnxxoH2VM7qvDkETOu/aIUByobtgB3\nlXnhl4FTF3A/kMKHay0SdXKCIGB8UtNz1gRBwNU99NhV1nQ3Za8oNdx+ICumYWvu8u46pFslmCQR\nJgnoZ5Hw94O1mJCkRV6NHxOTtbBoBPwypwpzMgywNLE25LU99JiacmYPuVvSDdhd7sXiPTXYXurB\ng9nm0P2/W/sakB0TDL0EvdhoTUvgzE7gbr+Mcd002FM3mXtXuRd3ZZrw7qFaJBlUkFSA3Smj0h1A\ntDZYl1+Wsafci9/1qcFGXyJ2lnmQWdeCDcgydpUHpwwU1vqRYuTXZkcT8RZZYWEhrr32WowePRpj\nx47Fm2++GemSiLqUW/sa8MJoS4NdtU9LM6khCgjdHzvNoBYbdFXOzzKhqNaP+zZU4toeOkiigBHx\nGtzd34gb+zRujQHBxZet2jNfQSpBwK+GR2FXmRc/7qlvsFGqIAjIitVgYIzUZIgBp1tkXmwtcWN2\nhhEn7H7YvQHsKvfi0iQtpqXp8dcDtZg7wIT+Vgn767W6jlb7EKMTYVbLyIqVsLfePbm8Gj/Mkogh\nsRoU1A2MOVDpxVd19xYjqTUT0zuTiP9poVar8cwzzyA7Oxt2ux2TJk3ClClTkJGREenSiLqE5pbJ\nAoJh88woC/qcZ03JVJMaT4204JTDHwonQRBwW/q5NxQ9W5RGxLuXxTQZqueTblFjb4UXFo2IDIsa\nmdFqfJHvglYFJBlUuKWvAd0MIrJiNegf7cEPlV6MrVtdZW+5F9l1rc5BMRIW765BQJYhCgJ2lnkw\nNE5CqkmFQnswyL4qdOG7Mi8uq9eilGUZT++oxk19Dci0SgjIMlbku3B1mq7RNQ7IMr7IdyFGJ2JM\nYss2iG3K69/bkWRQ4freTf+x0FVEvEWWmJiI7OxsAIDJZEJGRgZOnjwZ4aqI6LQxidrQfazzSTSo\noFW1bsuciwkxIDg3zySJGJ2oCQ5IiZHw4WEHhtQNiLFqRczsFfzC7x8dHP5/2p66eXIAEKcLrl5y\noi60dpZ5MCRWgxSjKtQi22/z4Ui1D2WuM/fMdpV78U2RG5/XDXzZXe7FH3bX4MhZa1OedPhx3/pK\n/C/fied3Vl/0iEpZlrH+pBurCtqnZXi4yos3v7fjZ99W4NovSvHoFlu7vE9biHiQ1Xf8+HHs3bsX\nw4cPj3QpRKRAw+IkTKi7Hzg4VoMKdwBDYhuP1uxvlXDQ5kOgbvubYIvszECXrJjgdABvIHjvbEic\nhNS6IPMFZByu8mFkvAbbSs7MfXvvUC3mZBiw7qQbvoCMlSdcMEsCtpY0nB/36r4aDI/X4LVLozEr\nw4jffVd9UaMh8+1++OXgNIRTjrYdhJJzyo2HN9ugFoGHB5vx+vho7Cn3dtiuzA4TZHa7HXPmzMHz\nzz8Pk8kU6XKISIF+NdyCUQnBIBsQLUEjAoPjGk9RsGpFWDQi8u1+FDn8EASgm+HM12FWjIS95V68\n9YMd2bES4nQqJBlUKHH6kVvlQzeDiMkp2lBI7a9bbuv2DCNSjSpsKHZj/Uk37h1owpZ6QbavwovD\nVT7MyjBCFARc30sPiyTi2Z3VoVGfp9WfCC7LMo5V+1Di9IdCb3tpcCeEcd20WF/c8uW5fAE59F4e\nv4yntldh7voKLN5dgx2lHhTYfXh+ZzV+O9KCu/qb0D9aQneTGiZJQFFtxxy1KdhstohHrM/nw003\n3YSpU6fi3nvvPeexubm5YaqKiJTO5hVglZr+inurUI8eOj9kAMddKvw85cxcuCK3iGfzTDCKMn7V\nyw6TOniORYfNGG72otYv4MfxLjx1zITn+tRgSb4RYyxeTI7x4KsKDT4v06Kn3o97UhxYmBuFF/tW\nQycCLx43YpzVg3HWM92angDw95N6lHlE3J3iQLxGxnfVarxfrMeve9sRpZax167GXwoN0Agy4jQB\nPNKjFq8VGDDG4oVWkPFluRaP9Dz3fnSnfVqqxfpKDW7t5sSW6mDIXxbjRp5ThY1VGpR4RNyQ4MJl\nMQ1bkq+dMGCUxYuRUW27gkp6enqrzxHxwR4AcN9996Ffv37nDTGgbT40tVxubi6veQTwure/qToX\n/vyDHSZJwL0DTUhP1Iauex9ZRlJpBR7MNmNwbLfQa3qWVuI7uxqzM4wY1VOPxFPlWHwyBv3jJdw1\nxAxREBDrCuBfq8pwfaYVg5JTkF1hgy2qOyo9MrxqB2aNSGp0H/CFDBkfHXHg+dzg3LqDNh9So0Sc\nNKRgeA89/rOrGj/tr8aNvfV4eLMNW2ULjrgc+N2lSdCpBPxtZRkKjd3h9ssY200LsyTCV9ctOjTu\nzBxAb0DGxqPluCfLiPdzJSQbVHhmlCW0S/lcWUa+3Y80k6rRAJVhgVrU+GWkp7dNj9nucg9itW3T\nKRjxIMvJycGyZcswYMAAjB8/HoIg4Mknn8TUqVMjXRoRdWJXdNfhiu66Jp8TBQFvT4pp9GWealJj\nR5kX/evWrZyYrMNxuw8LB5tDA2JidCKeG23BsLouzVEJGrx9sBY1Xhkvj7U2OZhFEATc3NeIK7vr\nsSLfifsHmfF9pRerT7gwLU2HTac8uDXdAEEQ8PCQKPzs2wqkmVSh+Xk39Nbj8+MuqETg7wdrce8A\nEz484sDxGj+u7aHHLwYYIQjBJcF6mVWYnqbHFak6CELDwTWCIKCHuelYSLeq8XHdiihbStzwBxAa\n9XmhZFnGy3tqkGmV8H9Doy7qHPVFPMhGjx6NioqKSJdBRNRAU9MSUo0qaFVAr7ov+9kZhiaPuyTh\nzBf8mEQNPjriwO8vsSLVdO6v3GitGJqyYJYEvLQ7ODncqhFCE7GTDCosHGyGt949tTszz7SS1ha5\n8OYPdlzX24DLU3X45WYbXtwdwD0DTPjPMSdu6B3c2Vzdwm2ETsuom3AekGW8us+OSncAf5kQgySD\niLUn3RgWp0GURkRAlvFBrgNjErXoYznzeQ9XeXHA5sM1PfQ4Uu1DpTuAjRdwb+9cIh5kRERK0dOs\nwsBoKRQC55qDd1qqSY1ll8e26Nj6jJKI7BgJf9prx+SUhi2f+vPXzjYxWYeJ9SaTLx5rxZ9/sOP2\nr8qhVQkYd5GtqFidCpIo4LM8J3QqAbMzjHj2u2qkGIPLmM3uZ8TsDCN2l3vxnzwnPjnmxNA4CY8P\ni4IoCHhrfy2+q5uTt6rAhWt66LGjmR0PLlSHGbVIRNTRjYjX4PlLrBf8ugsNsdPGJ2lRUOu/6PAB\nAJMkYuHgKPxlYgx+N8pywS2x+tKtavxlfy1u6K3HDb31MEgC/LKMxWOtWJHvrJsE7sRNfQz4cGos\nih1+rMh34Wi1D7lVPtza14A3vrfjqwI3Lk/VYWJS84F8IdgiIyJqIUEQoLnwPVEv2rhuWuwo86Cf\npfVf1UmG4BSC1siwqHHI5sOUFB1EQcDvL7GEQtosiVh30o1NxR7cN9AMrUrAgmwzHtlsw4AYCdf3\n1uOG3gbM/rocsToRPcxqaFoRqvUxyIiIOiirNrinXEdxZXcdBsZIkJroWr0mTYcXd9VgRLwmtExZ\nukXC5BQdVuS7sGhIFLSq4Iavp+fCJRnb5q8CBhkREbVIilHd7Or/l6Xq8PoPdkxPa9hdeFd/IyYn\na2GuG2E5tIkJ6q3FICMiolYzSSL+Pjk2tAHraQZ1cKHm9sQgIyKiNtHae3AXi6MWiYhI0RhkRESk\naAwyIiJSNAYZEREpGoOMiIgUjUFGRESKxiAjIiJFY5AREZGiMciIiEjRGGRERKRoDDIiIlI0BhkR\nESkag4yIiBSNQUZERIrGICMiIkVjkBERkaIxyIiISNEYZEREpGgMMiIiUjQGGRERKRqDjIiIFI1B\nRkREisYgIyIiRWOQERGRonWIIFuzZg1GjhyJ4cOH4+WXX450OUREpCARD7JAIIBf/vKXWL58OXJy\ncvDxxx/j0KFDkS6LiIgUIuJBtmPHDvTp0wdpaWmQJAnXX389VqxYEemyiIhIISIeZEVFRUhJSQn9\nnJycjKKioghWREREShLxIKOOLT09PdIldEm87pHB665MEQ+y5ORkFBQUhH4uKipCcnJyBCsiIiIl\niXiQDRs2DEePHkV+fj48Hg+WL1+OadOmRbosIiJSCHWkC1CpVHjxxRdx3XXXIRAIYNasWejXr1+k\nyyIiIoUQbDabHOkiiIiILlbEuxbPh5OlwycrKwvjxo3D+PHjMWXKFACAzWbDzJkzMWLECFx33XWo\nqqqKcJXKN2/ePKSnp2Ps2LGhx851nRcvXoxhw4Zh1KhR+PrrryNRcqfQ1HV//vnnMWDAAEyYMAET\nJkzAmjVrQs/xureNwsJCXHvttRg9ejTGjh2LN998E0Db/s536CDjZOnwEkURn3/+OdavXx/65Vmy\nZAkmTZqE7du3Y8KECViyZEmEq1S+2267DcuXL2/wWHPX+cCBA/j3v/+NrVu3YtmyZVi4cCFkmZ0o\nF6Op6w4Ac+fOxbp167Bu3TpMnToVAHDw4EFe9zaiVqvxzDPPICcnB6tWrcLSpUtx6NChNv2d79BB\nxsnS4SXLMgKBQIPHVqxYgVtuuQUAcMstt+Dzzz+PRGmdypgxY2C1Whs81tx1/uKLL3D99ddDrVaj\nR48e6NOnD3bs2BH2mjuDpq47gCa/JFesWMHr3kYSExORnZ0NADCZTMjIyEBRUVGb/s536CDjZOnw\nEgQBM2bMwOTJk/GPf/wDAFBSUoKEhAQAwV/I0tLSSJbYaZWWljZ5nc/+N5CUlMR/A23srbfewqWX\nXor7778/1L3F694+jh8/jr1792LEiBHNfrdczLXv0EFG4bVy5UqsW7cOy5Ytw1tvvYVNmzZBEIQG\nx5z9M7UPXufwuOuuu7B7925s2LABiYmJeOKJJyJdUqdlt9sxZ84cPP/88zCZTG363dKhg4yTpcOr\nW7duAIC4uDhcffXV2LFjBxISElBSUgIAOHXqFOLj4yNZYqfV3HVOTk5GYWFh6Dj+G2hbcXFxoS/Q\n2bNnh7qweN3bls/nw5w5c3DTTTfh6quvBtC2v/MdOsg4WTp8HA4H7HY7AKC2thbffPMNBg4ciGnT\npuGDDz4AAHz44YeYPn16JMvsNM6+L9PcdZ42bRqWL18Oj8eDvLw8HD16FMOHDw97vZ3F2df91KlT\nof//7LPPMGDAAAC87m3tvvvuQ79+/XDvvfeGHmvL3/mIT4g+F06WDp+SkhLcfvvtEAQBfr8fN954\nI6ZMmYKhQ4fijjvuwHvvvYfu3bvj7bffjnSpinfXXXdhw4YNqKiowKBBg7Bo0SI8+OCDmDNnTqPr\nnJmZiZkzZ+KSSy6BJEl46aWX2O14kZq67uvXr8fevXshiiLS0tJCU3x43dtOTk4Oli1bhgEDBmD8\n+PEQBAFPPvkkFixY0OR3y8Vce06IJiIiRevQXYtERETnwyAjIiJFY5AREZGiMciIiEjRGGRERKRo\nDDIiIlI0BhkRESkag4yonWzevBlXXnkl0tLS0Lt3b0ybNg27du3CBx98wBVqiNpQh17Zg0ipampq\ncPPNN+Pll1/GjBkz4PF4sHnzZmg0mkiXRtTpsEVG1A6OHDkCQRAwc+ZMCIIArVaLSZMmQa1W46GH\nHsK2bduQmpqKnj17AgA8Hg+eeOIJDBo0CP369cPChQvhdrsBABs2bMDAgQOxePFi9OnTB4MHD8ay\nZctC77Vq1SqMHj0a3bt3x8CBA/Hqq69G4iMTRQyDjKgd9OnTByqVCvfeey/WrFkDm80GAMjIyMDi\nxYsxcuRIFBQUIC8vDwDw61//GkePHsXGjRvx3XffoaioCC+88ELofKdOnUJlZSUOHDiA119/HQsW\nLMCRI0cAAA888AD++Mc/4sSJE9i0aRMmTJgQ9s9LFEkMMqJ2YDab8eWXX0IURSxYsAB9+/bFrbfe\n2uzGpP/4xz/w7LPPwmKxwGg04sEHH8THH38cel4QBDz++OOQJAnjxo3DFVdcgX//+98AAEmScODA\nAdTU1MBisYR24yXqKhhkRO0kPT0dr732Gvbt24ecnBycPHkSjz76aKPjysrK4HA4MGnSJPTs2RM9\ne/bEDTfcgMrKytAxVqsVOp0u9HP37t1RXFwMAHj33XexcuVKZGVl4ZprrsG2bdva/8MRdSAMMqIw\nON0i279/f6MtKWJjY2EwGJCTk4O8vDzk5eUhPz8f+fn5oWNsNhucTmfo54KCgtBGqEOGDMEHH3yA\nI0eOYPr06bjzzjvD86GIOggGGVE7yM3NxauvvoqioiIAweBZvnw5Ro0ahYSEBBQVFcHr9QIIdhvO\nnj0bjz76KMrKygAEd8X9+uuvQ+eTZRnPPfccvF4vNm3ahFWrVmHmzJnwer1YtmwZqquroVKpYDKZ\nIIr8Z01dC4ffE7UDk8mEHTt24PXXX0d1dTUsFguuuuoqPPXUU9BqtcjMzERGRgZUKhUOHz6M3/zm\nN3jhhRcwdepUVFRUIDk5GT/96U8xZcoUAEC3bt1gtVqRmZkJg8GAJUuWoE+fPvB6vfjoo4/wyCOP\nwO/3Iz09HUuXLo3wpycKL26sSdTBbdiwAb/4xS+wb9++SJdC1CGxD4KIiBSNQUZERIrGrkUiIlI0\ntsiIiEjRGGRERKRoDDIiIlI0BhkRESkag4yIiBSNQUZERIr2/8WmrXApUFNpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x148d232d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with plt.style.context('fivethirtyeight'):\n",
    "    plt.plot(losses, linewidth = 1)\n",
    "    plt.xlabel('Steps')\n",
    "    plt.ylabel('Losses')\n",
    "    plt.ylim((0, 12))\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing Our Trained NMT (French => English)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.\n",
      "--------------------------------\n",
      "Quel est ton nom\n",
      "What is your name \n",
      "--------------------------------\n",
      "2.\n",
      "--------------------------------\n",
      "Mon nom est\n",
      "My name is \n",
      "--------------------------------\n",
      "3.\n",
      "--------------------------------\n",
      "Qu'est-ce que tu fais\n",
      "- You are wrong \n",
      "--------------------------------\n",
      "4.\n",
      "--------------------------------\n",
      "je lis un livre\n",
      "I wanna wanna like a book \n",
      "--------------------------------\n",
      "5.\n",
      "--------------------------------\n",
      "Comment allez-vous\n",
      "How <ukn> \n",
      "--------------------------------\n",
      "6.\n",
      "--------------------------------\n",
      "je vais bien\n",
      "I&apos m going \n",
      "--------------------------------\n",
      "7.\n",
      "--------------------------------\n",
      "Parlez vous anglais\n",
      "- You thought you \n",
      "--------------------------------\n",
      "8.\n",
      "--------------------------------\n",
      "Quelle heure est-il\n",
      "What <ukn> <ukn> \n",
      "--------------------------------\n",
      "9.\n",
      "--------------------------------\n",
      "salut\n",
      "<ukn> \n",
      "--------------------------------\n",
      "10.\n",
      "--------------------------------\n",
      "Au Revoir\n",
      "A <ukn> \n",
      "--------------------------------\n",
      "11.\n",
      "--------------------------------\n",
      "Oui\n",
      "Yeah \n",
      "--------------------------------\n",
      "12.\n",
      "--------------------------------\n",
      "Non\n",
      "No \n",
      "--------------------------------\n"
     ]
    }
   ],
   "source": [
    "with tf.Graph().as_default():\n",
    "    \n",
    "    # placeholders\n",
    "    encoder_inputs = [tf.placeholder(dtype = tf.int32, shape = [None], name = 'encoder{}'.format(i)) for i in range(input_seq_len)]\n",
    "    decoder_inputs = [tf.placeholder(dtype = tf.int32, shape = [None], name = 'decoder{}'.format(i)) for i in range(output_seq_len)]\n",
    "\n",
    "    # output projection\n",
    "    size = 512\n",
    "    w_t = tf.get_variable('proj_w', [de_vocab_size, size], tf.float32)\n",
    "    b = tf.get_variable('proj_b', [de_vocab_size], tf.float32)\n",
    "    w = tf.transpose(w_t)\n",
    "    output_projection = (w, b)\n",
    "    \n",
    "    # change the model so that output at time t can be fed as input at time t+1\n",
    "    outputs, states = tf.contrib.legacy_seq2seq.embedding_attention_seq2seq(\n",
    "                                                encoder_inputs,\n",
    "                                                decoder_inputs,\n",
    "                                                tf.contrib.rnn.BasicLSTMCell(size),\n",
    "                                                num_encoder_symbols = en_vocab_size,\n",
    "                                                num_decoder_symbols = de_vocab_size,\n",
    "                                                embedding_size = 100,\n",
    "                                                feed_previous = True, # <-----this is changed----->\n",
    "                                                output_projection = output_projection,\n",
    "                                                dtype = tf.float32)\n",
    "    \n",
    "    # ops for projecting outputs\n",
    "    outputs_proj = [tf.matmul(outputs[i], output_projection[0]) + output_projection[1] for i in range(output_seq_len)]\n",
    "\n",
    "    # let's translate these sentences     \n",
    "    fr_sentences = [\"Quel est ton nom\", \"Mon nom est\", \"Qu'est-ce que tu fais\", \"je lis un livre\",\"Comment allez-vous\", \"je vais bien\", \"Parlez vous anglais\", \"Quelle heure est-il\", \"salut\", \"Au Revoir\", \"Oui\", \"Non\"]\n",
    "    fr_sentences_encoded = [[en_word2idx.get(word, 0) for word in fr_sentence.split()] for fr_sentence in fr_sentences]\n",
    "    \n",
    "    # padding to fit encoder input\n",
    "    for i in range(len(fr_sentences_encoded)):\n",
    "        fr_sentences_encoded[i] += (15 - len(fr_sentences_encoded[i])) * [en_word2idx['<pad>']]\n",
    "    \n",
    "    # restore all variables - use the last checkpoint saved\n",
    "    saver = tf.train.Saver()\n",
    "    path = tf.train.latest_checkpoint('checkpoints')\n",
    "    \n",
    "    with tf.Session() as sess:\n",
    "        # restore\n",
    "        saver.restore(sess, path)\n",
    "        \n",
    "        # feed data into placeholders\n",
    "        feed = {}\n",
    "        for i in range(input_seq_len):\n",
    "            feed[encoder_inputs[i].name] = np.array([fr_sentences_encoded[j][i] for j in range(len(fr_sentences_encoded))], dtype = np.int32)\n",
    "            \n",
    "        feed[decoder_inputs[0].name] = np.array([de_word2idx['<go>']] * len(fr_sentences_encoded), dtype = np.int32)\n",
    "        \n",
    "        # translate\n",
    "        output_sequences = sess.run(outputs_proj, feed_dict = feed)\n",
    "        \n",
    "        # decode seq.\n",
    "        for i in range(len(fr_sentences_encoded)):\n",
    "            print '{}.\\n--------------------------------'.format(i+1)\n",
    "            ouput_seq = [output_sequences[j][i] for j in range(output_seq_len)]\n",
    "            #decode output sequence\n",
    "            words = decode_output(ouput_seq)\n",
    "        \n",
    "            print fr_sentences[i]\n",
    "            for i in range(len(words)):\n",
    "                if words[i] not in ['<eos>', '<pad>', '<go>']:\n",
    "                    print words[i],\n",
    "            \n",
    "            print '\\n--------------------------------'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be deduced, the results are pretty decent. Now we have Neural Machine Translator (NMT) that can convert English to German and French to English. \n",
    "\n",
    "Training reverse models is just task of interchanging inputs and outputs!\n",
    "\n",
    "Perhaps I must deploy these trained models in an Android app, and I believe I shall be ready for an Euro Trip!!! "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}