Elmo_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Elmo - A short Tutorial\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sklearn.manifold import TSNE\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "from tensorflow.contrib.tensorboard.plugins import projector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre Trained Elmo Model:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the Elmo Model\n",
    "\n",
    "\n",
    "The model trained on One Billion World Language Model Benchmark (http://www.statmt.org/lm-benchmark/) as been exposed on Tensorflow Hub. This can be loaded as :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "elmo = hub.Module(\"https://tfhub.dev/google/elmo/2\", trainable=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We set the trainable parameter to True when creating the module so that the 4 scalar weights (as described in the paper) and all LSTM cell variables can be trained. In this setting, the module still keeps all other parameters fixed.\n",
    "This will help to get the embedding of a word the model has not seen, given the context."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Structure\n",
    "The elmo model consists of two files:\n",
    "\n",
    "**options.json** : These are the parameters/options using which the language model was trained on  <br>\n",
    "\n",
    "**weights.hdf5** : The weights file for the best model\n",
    "\n",
    "\n",
    "The input to the pre trained model (elmo) above can be fed in two different ways:\n",
    "\n",
    " ### 1. **Tokens**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens_input = [[\"Argentina\", \"played\", \"football\", \"very\", \"well\",\"\",\"\",\"\",\"\"],\n",
    "                [\"Brazil\",\"is\",\"a\",\"strong\",\"team\",\"\",\"\",\"\",\"\"],\n",
    "                [\"Artists\",\"all\",\"over\",\"the\",\"world\",\"are\",\"attending\",\"the\",\"play\"],\n",
    "                [\"Child\",\"is\",\"playing\",\"the\",\"guitar\",\"\",\"\",\"\",\"\"],\n",
    "                [\"There\",\"was\",\"absolute\",\"silence\",\"during\",\"the\",\"play\",\"\",\"\"]]\n",
    "\n",
    "tokens_length = [5,5,9,5,7]\n",
    "embeddings = elmo(inputs={\"tokens\": tokens_input,\"sequence_len\": tokens_length},\n",
    "                  signature=\"tokens\",\n",
    "                  as_dict=True)[\"elmo\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. **Default**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = elmo([\"Argentina played football very well\",\"Brazil is a strong team\", \n",
    "                   \"Artists all over the world are attending the play\", \n",
    "                   \"Child is playing the guitar\", \n",
    "                   \"There was absolute silence during the play\"],\n",
    "             signature=\"default\",\n",
    "             as_dict=True)[\"elmo\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Inputs**\n",
    "\n",
    "The module defines two signatures: tokens and default.\n",
    "\n",
    "With the tokens signature, the module takes tokenized sentences as input. The input tensor is a string tensor with shape [batch_size, max_length] and an int32 tensor with shape [batch_size] corresponding to the sentence length. The length input is necessary to exclude padding in the case of sentences with varying length.\n",
    "\n",
    "In case of above Example:\n",
    "\n",
    "    Each element contains one layer of ELMo representations with shape\n",
    "    (5, 9, 1024).\n",
    "     5    - the batch size\n",
    "     9    - the sequence length of the batch\n",
    "     1024 - the dimension of each ELMo vector\n",
    "\n",
    "With the default signature, the module takes untokenized sentences as input. The input tensor is a string tensor with shape [batch_size]. The module tokenizes each string by splitting on spaces.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "**Outputs**\n",
    "\n",
    "The output (_embeddings_) is a dictionary with following keys:\n",
    "\n",
    "  - word_emb: the character-based word representations with shape [batch_size, max_length, 512].\n",
    "  - lstm_outputs1: the first LSTM hidden state with shape [batch_size, max_length, 1024].\n",
    "  - lstm_outputs2: the second LSTM hidden state with shape [batch_size, max_length, 1024].\n",
    "  - elmo: the weighted sum of the 3 layers, where the weights are trainable. This tensor has shape [batch_size, max_length, 1024]\n",
    "  - default: a fixed mean-pooling of all contextualized word representations with shape [batch_size, 1024].\n",
    "  \n",
    "  The \"elmo\" value is selected.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "init = tf.initialize_all_variables()\n",
    "sess = tf.Session()\n",
    "sess.run(init)\n",
    "\n",
    "print(\"Argentina\")\n",
    "print(sess.run(embeddings[0][0]))\n",
    "\n",
    "print(\"played\")\n",
    "print(sess.run(embeddings[0][1]))\n",
    "\n",
    "print(\"play - Theatre\")\n",
    "print(sess.run(embeddings[4][6]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting the Embeddings:\n",
    "\n",
    "Firstly, initialise the session:\n",
    "\n",
    "        init = tf.initialize_all_variables()\n",
    "        sess = tf.Session()\n",
    "        sess.run(init)\n",
    "\n",
    "The embeddings for any token can be obtained using _embeddings_ and passing the position of the token.\n",
    "\n",
    "For Example:\n",
    "\n",
    "**Argentina** ([0][0])\n",
    "\n",
    "    print(sess.run(embeddings[0][0]))\n",
    "\n",
    "**played** ([0][1])\n",
    "\n",
    "    print(sess.run(embeddings[0][1]))\n",
    "    \n",
    "**play**  ([4][6])\n",
    "\n",
    "    print(sess.run(embeddings[4][6]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the Embeddings \n",
    "\n",
    "#### 1. **t-SNE**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tsne_plot():\n",
    "    \"Creates and TSNE model and plots it\"\n",
    "    labels = []\n",
    "    tokens = []\n",
    "    \n",
    "    import tensorflow as tf\n",
    "    import sys\n",
    "    np.set_printoptions(threshold=sys.maxsize)\n",
    "\n",
    "    tokens_input = [[\"Argentina\", \"played\", \"football\", \"very\", \"well\",\"\",\"\",\"\",\"\"],\n",
    "                [\"Brazil\",\"is\",\"a\",\"strong\",\"team\",\"\",\"\",\"\",\"\"],\n",
    "                [\"Artists\",\"all\",\"over\",\"the\",\"world\",\"are\",\"attending\",\"the\",\"play\"],\n",
    "                [\"Child\",\"is\",\"playing\",\"the\",\"guitar\",\"\",\"\",\"\",\"\"],\n",
    "                [\"There\",\"was\",\"absolute\",\"silence\",\"during\",\"the\",\"play\",\"\",\"\"]]\n",
    "\n",
    "    tokens_length = [5,5,9,5,7]\n",
    "    embeddings = elmo(inputs={\"tokens\": tokens_input,\"sequence_len\": tokens_length},\n",
    "                      signature=\"tokens\",\n",
    "                      as_dict=True)[\"elmo\"]\n",
    "\n",
    "\n",
    "\n",
    "    init = tf.initialize_all_variables()\n",
    "    sess = tf.Session()\n",
    "    sess.run(init)\n",
    "    sent = [\"Argentina\", \"played\", \"football\", \"very\", \"well\"]\n",
    "    sent1 = [\"Brazil\",\"is\",\"a\",\"strong\",\"team\"]\n",
    "    sent2 = [\"Artists\",\"all\",\"over\",\"the\",\"world\",\"are\",\"attending\",\"the\",\"play\"]\n",
    "    sent3 = [\"Child\",\"is\",\"playing\",\"the\",\"guitar\"]\n",
    "    sent4 = [\"There\",\"was\",\"absolute\",\"silence\",\"during\",\"the\",\"play\"]\n",
    "    \n",
    "\n",
    "    for i in range(len(sent)):\n",
    "        tokens.append(sess.run(embeddings[0][i]))\n",
    "        labels.append(sent[i])  \n",
    "    for i in range(len(sent1)):\n",
    "        tokens.append(sess.run(embeddings[1][i]))\n",
    "        labels.append(sent1[i])\n",
    "    for i in range(len(sent2)):\n",
    "        tokens.append(sess.run(embeddings[2][i]))\n",
    "        labels.append(sent2[i])\n",
    "    for i in range(len(sent3)):\n",
    "        tokens.append(sess.run(embeddings[3][i]))\n",
    "        labels.append(sent3[i])  \n",
    "    for i in range(len(sent4)):\n",
    "        tokens.append(sess.run(embeddings[4][i]))\n",
    "        labels.append(sent4[i])\n",
    "    \n",
    "    tsne_model = TSNE(perplexity=6, n_components=2, init='random', n_iter=500)\n",
    "    new_values = tsne_model.fit_transform(tokens)\n",
    "\n",
    "    x = []\n",
    "    y = []\n",
    "    for value in new_values:\n",
    "        x.append(value[0])\n",
    "        y.append(value[1])\n",
    "        \n",
    "    plt.figure(figsize=(18, 12)) \n",
    "    for i in range(len(x)):\n",
    "        plt.scatter(x[i],y[i])\n",
    "        plt.annotate(labels[i],\n",
    "                     xy=(x[i], y[i]),\n",
    "                     xytext=(5, 2),\n",
    "                     textcoords='offset points',\n",
    "                     ha='right',\n",
    "                     va='bottom')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABCAAAAKvCAYAAAC72r5SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3XtYVnW+///XEhFBFNRy56kB3Xm68eZGwCAFUbdBeU4tJ8+WznZqW803f+avsdxNTu2Rdg3Tr7yYScudYaaZ2eR4xjQxu1EkVIzRUCsyD18QFAxw/f7gsEPxQLJYIM/Hdc3FvT7rs9b9/njVXPbiczBM0xQAAAAAAICVmthdAAAAAAAAuPURQAAAAAAAAMsRQAAAAAAAAMsRQAAAAAAAAMsRQAAAAAAAAMsRQAAAAAAAAMsRQAAAAAAAAMsRQAAAAAAAAMsRQAAAAAAAAMs1tbuAG3HbbbeZAQEBdpcBAAAAAAAuk5qaeto0zduv169BBBABAQFyu912lwEAAAAAAC5jGMaxG+nHEgwAAAAAAGA5AggAAAAAAGA5AggAAAAAAGA5AggAAAAAAGA5AggAAAAAAGA5AggAAAAAAGA5AggAAAAAAGA5AggAAAAAAGA5AggAAAAAAGA5AggAAAAAAGA5AggAAAAAAGA5AggAAAAAAGA5AggAAAAAAGA5AggAAAAAAGA5AggAAAAAAGA5AggAAAAAAGA5AggAAAAAAGA5AggAAAAAAGA5AgigjqxZs0aGYSgzM7Pa+7m5uXrjjTcqr7///nuNHTv2qu+7vD8AAAAA1GcEEEAdSUpKUv/+/bVixYor7pWWll4RKHTo0EGrVq266vsIIAAAAAA0JAQQQB0oKCjQ559/rrfeeqsygEhOTtbAgQP18MMPq3fv3nrmmWd05MgRuVwuzZkzR9nZ2QoKCpIkHThwQH379pXL5ZLT6VRWVtYV/XNychQdHS2Xy6WgoCDt2LHDziEDAAAAQBVN7S4AaAw++ugjxcXFqVu3bmrTpo327t0rSdqzZ48yMjIUGBio7OxsZWRkKC0tTZKUnZ1d+fzixYv1xBNPaMKECfrpp59UWlqql19+uUr/V155RbGxsXr22WdVWlqqCxcu1Pk4AQAAAOBqmAEB1IGkpCSNHz9ekjR+/HglJSVJkvr27avAwMDrPh8ZGak//vGP+q//+i8dO3ZM3t7eV/QJDw/X0qVLtWDBAn311Vdq2bJl7Q4CAAAAAG4CAQRgsTNnzmjr1q169NFHFRAQoEWLFun999+XaZpq0aLFDb3j4Ycf1scffyxvb2/FxsZq69atV/SJjo7WZ599po4dO2rSpElatmxZbQ8FAGpdQECATp8+LUny9fW1uRoAAGAlAgjAYqtWrdLkyZN17NgxZWdn68SJEwoMDNTOnTur9GvZsqXy8/OrfcfRo0fVpUsXzZ49WyNGjFB6evoV/Y8dO6Z27dppxowZeuSRRyqXeQAAAABAfUAAAVgsKSlJo0ePrtI2ZswYvffee1Xa2rZtq379+ikoKEhz5sypcu/9999XUFCQXC6XMjMzNXny5Cv6Jycny+VyKSQkRKtXr9YTTzxh+dgAoCZGjRql0NBQORwOJSYm2l0OAACoY4ZpmnbXcF1hYWGm2+22uwwAAHATzp49qzZt2qiwsFDh4eHavn27QkND5Xa7ddttt8nX11cFBQV2lwkAAGrIMIxU0zTDrtePUzCAW8BH+77Tog2H9X1uoTr4e2tObHeNCulod1kAUEVCQoLWrFkjSTpx4oSysrJsrggAANQlAgiggfto33ea9+FXKiwulSR9l1uoeR9+JUmEEADqjeTkZG3evFkpKSny8fFRTEyMioqK7C4LAADUIfaAABq4RRsOV4YPFQqLS7Vow2GbKgKAK+Xl5al169by8fFRZmamdu/ebXdJAACgjhFAAA3c97mFNWoHADvExcWppKRETqdT8+fPV0REhN0lAQCAOsYSDKCB6+Dvre+qCRs6+HvbUA0AVM/Ly0vr16+/oj07O7vyMxtQAgBwa2MGBNDAzYntLm9Pjypt3p4emhPb3aaKAOAGpK+UXg2SFviX/UxfaXdFAADAYsyAABq4io0mOQUDQIORvlJaN1sqLp+9lXei7FqSnA/aVxcAALCUYZqm3TVcV1hYmOl2u+0uAwAA1IZXg8pCh8v5dZaeyqj7egAAwE0xDCPVNM2w6/VjCQYAAKhbed/WrB0AANwSCCAAAEDd8utUs3YAAHBLIIAAAAB1a/BzkudlJ/V4epe1AwCAWxYBBAAAqFvOB6XhCWV7Psgo+zk8gQ0oAQC4xXEKBgAAqHvOBwkcAABoZJgBAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALEcAAQAAAAAALFdrAYRhGB6GYewzDOOT8utAwzC+MAwjyzCM9w3DaFbe7lV+/c/y+wG1VQNuHWfOnJHL5ZLL5dIdd9yhjh07yuVyyd/fX7169bK7PAAAAABADdXmDIgnJB362fV/SXrVNM27JP1fSY+Utz8i6f+apvmvkl4t7wdU0bZtW6WlpSktLU3//u//rqeeeqryukmTX/6PbUlJSS1WCQAAAAC4UbUSQBiG0UnSUEl/K782JA2StKq8yzuSRpV/Hll+rfL7g8v7AzektLRUM2bMkMPh0L333qvCwkJJ0pEjRxQXF6fQ0FBFRUUpMzNTkjR16lT97ne/08CBAzV37lydP39e06dPV3h4uEJCQrR27Vo7hwMAAAAAjUJtzYB4TdL/I+lS+XVbSbmmaVb8uvlbSR3LP3eUdEKSyu/nlfcHbkhWVpYee+wxHThwQP7+/lq9erUkaebMmfrLX/6i1NRUxcfH67e//W3lM19//bU2b96sV155RQsXLtSgQYP05Zdfatu2bZozZ47Onz9v13AAAAAAoFFoerMvMAxjmKQfTdNMNQwjpqK5mq7mDdz7+XtnSpopSXfeeefNlolbSGBgoFwulyQpNDRU2dnZKigo0K5duzRu3LjKfhcvXqz8PG7cOHl4eEiSNm7cqI8//ljx8fGSpKKiIh0/flw9e/asw1EAAAAAQONy0wGEpH6SRhiGcb+k5pJaqWxGhL9hGE3LZzl0kvR9ef9vJXWW9K1hGE0l+Uk6e/lLTdNMlJQoSWFhYVcEFGi8vLy8Kj97eHiosLBQly5dkr+/v9LS0qp9pkWLFpWfTdPU6tWr1b17d8trBQAAAACUueklGKZpzjNNs5NpmgGSxkvaaprmBEnbJI0t7zZFUsVC+4/Lr1V+f6tpmgQMuCmtWrVSYGCgPvjgA0llIcP+/fur7RsbG6u//OUvqvjHbt++fXVWJwAAAAA0VrV5Csbl5kr6nWEY/1TZHg9vlbe/JaltefvvJD1jYQ1oRJYvX6633npLwcHBcjgcV91ccv78+SouLpbT6VRQUJDmz59fx5UCAAAAQONjNITJB2FhYabb7ba7DAAAAAAAcBnDMFJN0wy7Xr/a2AMCaDC+/uIHpaw9ooKzF+XbxkuRI7uq29132F0WAAAAANzyCCDQaHz9xQ/atjxTJT+VnRZbcPaiti3PlCRCCAAAAACwmJV7QAD1SsraI5XhQ4WSny4pZe0RmyoCAAAAgMaDAAKNRsHZizVqBwAAAADUHgIINBq+bbxq1A4AAAAAqD0EEGg0Ikd2VdNmVf+Rb9qsiSJHdrWpIgAAAABoPNiEEo1GxUaTnIIBAAAAAHWPAAKNSre77yBwAAAAAAAbsAQDAAAAAABYjgACAAAAAABYjgACAAAAAABYjgACAAAAAABYjgACAAAAAABYjgACAAAAAABYjgACAAAAAABYjgACAADgFvboo4/q4MGDkqSAgACdPn3a5ooAAI1VU7sLAAAAgHX+9re/2V0CAACSmAEBAABwyzh//ryGDh2q4OBgBQUF6f3331dMTIzcbvcVfd9991317dtXLpdLv/nNb1RaWipJ8vX11bPPPqvg4GBFRETo5MmTkqSTJ09q9OjRCg4OVnBwsHbt2nXN9wAAcDkCCAAAgFvEP/7xD3Xo0EH79+9XRkaG4uLiqu136NAhvf/++/r888+VlpYmDw8PLV++XFJZiBEREaH9+/crOjpaf/3rXyVJs2fP1oABA7R//37t3btXDofjmu8BAOByLMEAAAC4RfTu3VtPP/205s6dq2HDhikqKqraflu2bFFqaqrCw8MlSYWFhWrXrp0kqVmzZho2bJgkKTQ0VJs2bZIkbd26VcuWLZMkeXh4yM/PT//zP/9z1fcAAHA5AggAAIBbRLdu3ZSamqpPP/1U8+bN07333lttP9M0NWXKFL300ktX3PP09JRhGJLKgoaSkpKrft+13gMAwOVYggEAqFWjRo1SaGioHA6HEhMT7S4HaFS+//57+fj4aOLEiXr66ae1d+/eavsNHjxYq1at0o8//ihJOnv2rI4dO3bNdw8ePFhvvvmmJKm0tFTnzp37Re8BADReBBAAgFq1ZMkSpaamyu12KyEhQWfOnLG7JKDR+Oqrryo3hFy4cKF+//vfV9uvV69eevHFF3XvvffK6XRqyJAhysnJuea7//znP2vbtm3q3bu3QkNDdeDAgV/0HgBA42WYpml3DdcVFhZmVrd7MwCg/lmwYIHWrFkjScrOztaGDRsUERFhc1UAAACwimEYqaZphl2vH3tAAABqTXJysjZv3qyUlBT5+PgoJiZGRUVFdpcFwCLp6enasmWL8vLy5Ofnp8GDB8vpdNpdFgCgniKAAADUmry8PLVu3Vo+Pj7KzMzU7t277S4JgEXS09O1bt06FRcXSyr793/dunWSRAgBAKgWe0AAAGpNXFycSkpK5HQ6NX/+fJZeALewLVu2VIYPFYqLi7VlyxabKgIA1HfMgAAA1BovLy+tX7/e7jIA1IG8vLwatQMAQAABALhprAMHGh8/P79qwwY/Pz8bqgEANAQswQAA3JSKdeAV/yFSsQ48PT3d5soAWGnw4MHy9PSs0ubp6anBgwfbVBEAoL4jgAAA3BTWgQONk9Pp1PDhwytnPPj5+Wn48OHMfgIAXBVLMAAAN4V14EDj5XQ6CRwAADeMGRAAgJtytfXerAMHAADAzxFAAABuCuvAAQAAcCNYggEAuCkV0685BQMAAADXQgABALhprAMHAADA9bAEAwAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4AAgAAAAAAWI4Aohq5ubl644037C4DAAAAAIBbBgFENQggAAAAAACoXQQQ1XjmmWd05MgRuVwuzZkzR4sWLVJ4eLicTqeef/75yn6jRo1SaGioHA6HEhMTK9t9fX01d+5chYaG6t/+7d+0Z88excTEqEuXLvr444/tGBIAwEJ//OMfKz/XdoidnZ2toKAgSZLb7dbs2bNr7d0AAAB1iQCiGi+//LK6du2qtLQ0DRkyRFlZWdqzZ4/S0tKUmpqqzz77TJK0ZMkSpaamyu12KyEhQWfOnJEknT9/XjExMUpNTVXLli31+9//Xps2bdKaNWv03HPP2Tk0AIAFrAwgfi4sLEwJCQmWvBsAAMBqTe0uoL7buHGjNm7cqJCQEElSQUGBsrKyFB0drYSEBK1Zs0aSdOLECWVlZalt27Zq1qyZ4uLiJEm9e/eWl5eXPD091bt3b2VnZ9s1FABALRg1apROnDihoqIiPfHEEzp69KgKCwvlcrnkcDhUWlpaOYtuyJAhWrRokRYtWqSVK1fq4sWLGj16tP7zP/9T2dnZuu+++9S/f3/t2rVLHTt21Nq1a+Xt7a3U1FRNnz5dPj4+6t+/f+V3JycnKz4+Xp988okWLFig48eP6+jRozp+/LiefPLJytkRf/jDH7R8+XJ17txZt912m0JDQ/X000/b9UcGAAAgiQDiukzT1Lx58/Sb3/ymSntycrI2b96slJQU+fj4KCYmRkVFRZIkT09PGYYhSWrSpIm8vLwqP5eUlNTtAAAAtWrJkiVq06aNCgsLFR4eru3bt+v1119XWlqapLIlExkZGZXXGzdurJxJZ5qmRowYoc8++0x33nmnsrKylJSUpL/+9a968MEHtXr1ak2cOFHTpk3TX/7yFw0YMEBz5sy5ai2ZmZnatm2b8vPz1b17d82aNUv79+/X6tWrtW/fPpWUlKhPnz4KDQ2tkz8bAACAa2EJRjVatmyp/Px8SVJsbKyWLFmigoICSdJ3332nH3/8UXl5eWrdurV8fHyUmZmp3bt321kyAKCOJCQkKDg4WBEREZWz367l5zPp+vTpo8zMzMpnAgMD5XK5JEmhoaHKzs5WXl6ecnNzNWDAAEnSpEmTrvruoUOHysvLS7fddpvatWunkydPaufOnRo5cqS8vb3VsmVLDR8+vJZGDgAAcHOYAVGNtm3bql+/fgoKCtJ9992nhx9+WJGRkZLKNph89913FRcXp8WLF8vpdKp79+6KiIiwuWoAgNWuNfvtaq42ky47O7tyhpwkeXh4qLCwUKZpVs6iu57Lny8pKZFpmjUYEQAAQN0hgLiK9957r8r1E088cUWf9evXV/tsxWwJSVqwYMFV7wEAGparzX7z9PRUcXGxPD09q8yik8pm0s2fP18TJkyQr6+vvvvuO3l6el71O/z9/eXn56edO3eqf//+Wr58eY1q7N+/v37zm99o3rx5Kikp0d///nfNmDHjlw0YAACgFhFAWGz1D2f10tEcfXexWB29PDWvS3uNuaON3WUBAH6Bq81+mzlzppxOp/r06aPly5dXmUW3aNEiHTp06IqZdB4eHlf9nqVLl1ZuQhkbG1ujGsPDwzVixAgFBwfrV7/6lcLCwuTn5/fLBw0AAFBLjIYwVTMsLMx0u912l1Fjq384q6cPn1Dhpf/9M/ZuYii+e2dCCACAZQoKCuTr66sLFy4oOjpaiYmJ6tOnj91lAQCAW5RhGKmmaYZdrx+bUFropaM5VcIHSSq8ZOqlozk2VQQAuJWd3/ejcl7eo4l9R8vRoZtcjmCNGTOG8AEAANQLLMGw0HcXi2vUDgDAL3V+34/K/TBLZvElvT7iOUmS4dlE/nF32VwZAABAGWZAWKijV/WbjF2tHQCAX+rchmyZxZeqtJnFl3RuQ7Y9BQEAAFyGAMJC87q0l3eTqkepeTcxNK9Le5sqAgDcqkpzL9aoHQAAoK4RQFhozB1tFN+9szp5ecqQ1MnLkw0oAQCW8PD3qlE7AABAXWMPCIuNuaMNgQMAwHKtYgMq94CoYHg2UavYAPuKAgAA+BkCCAAAbgEtQtpJKtsLojT3ojz8vdQqNqCyHQAAwG4EEAAA3CJahLQjcAAAAPUWe0AAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADL3XQAYRhGZ8MwthmGccgwjAOGYTxR3t7GMIxNhmFklf9sXd5uGIaRYBjGPw3DSDcMo8/N1gAAAAAAAOq32pgBUSLp/5im2VNShKTHDMPoJekZSVtM07xL0pbya0m6T9Jd5f+bKenNWqgBwC3gT3/6kxISEiRJTz31lAYNGiRJ2rJliyZOnKhZs2YpLCxMDodDzz//fOVzzzzzjHr16iWn06mnn37altoBAAAAXFvTm32BaZo5knLKP+cbhnFIUkdJIyXFlHd7R1KypLnl7ctM0zQl7TYMw98wjPbl7wHQiEVHR+uVV17R7Nmz5Xa7dfHiRRUXF2vnzp2KiorSuHHj1KZNG5WWlmrw4MFKT09Xp06dtGbNGmVmZsowDOXm5to9DAAAAADVqNU9IAzDCJAUIukLSf9SESqU/2xX3q2jpBM/e+zb8jYAjVxoaKhSU1OVn58vLy8vRUZGyu12a8eOHYqKitLKlSvVp08fhYSE6MCBAzp48KBatWql5s2b69FHH9WHH34oHx8fu4cBAAAAoBq1FkAYhuErabWkJ03TPHetrtW0mdW8b6ZhGG7DMNynTp2qrTIB1GOenp4KCAjQ0qVLdc899ygqKkrbtm3TkSNH5O3trfj4eG3ZskXp6ekaOnSoioqK1LRpU+3Zs0djxozRRx99pLi4OLuHAQAAAKAatRJAGIbhqbLwYblpmh+WN580DKN9+f32kn4sb/9WUuefPd5J0veXv9M0zUTTNMNM0wy7/fbba6NMAA1AdHS04uPjFR0draioKC1evFgul0vnzp1TixYt5Ofnp5MnT2r9+vWSpIKCAuXl5en+++/Xa6+9prS0NJtHAAAAAKA6N70HhGEYhqS3JB0yTfO/f3brY0lTJL1c/nPtz9ofNwxjhaS7JeWx/wOAClFRUVq4cKEiIyPVokULNW/eXFFRUQoODlZISIgcDoe6dOmifv36SZLy8/M1cuRIFRUVyTRNvfrqqzaPAAAAAEB1bjqAkNRP0iRJXxmGUfGrx/9XZcHDSsMwHpF0XNK48nufSrpf0j8lXZA0rRZqAHCLGDx4sIqLiyuvv/7668rPb7/99hX9D+3YpkfDHco/c1ot296mvl3urIsyAQAAANRQbZyCsVPV7+sgSYOr6W9KeuxmvxcADu3Ypo2Jr6vkp4uSpPzTp7Qx8XVJUs+ogXaWBgAAAOAytXoKBgDUpR0rllWGDxVKfrqoHSuW2VQRAAAAgKshgADQYOWfOV2jdgAAAAD2IYAA0GC1bHtbjdoBAAAA2IcAAkCDFTV+spo286rS1rSZl6LGT7apIgAAAABXUxunYACALSo2mtyxYlnlKRhR4yezASUAAABQDxFAAGjQekYNJHAAAAAAGgCWYAAAAKDO5Obm6o033pAkJScna9iwYTZXBACoKwQQAAAAqDM/DyAAAI0LAQQAAADqzDPPPKMjR47I5XJpzpw5Kigo0NixY9WjRw9NmDBBpmlKklJTUzVgwACFhoYqNjZWOTk5NlcOALhZBBAAAACoMy+//LK6du2qtLQ0LVq0SPv27dNrr72mgwcP6ujRo/r8889VXFys//iP/9CqVauUmpqq6dOn69lnn7W7dADATWITSgAAANimb9++6tSpkyTJ5XIpOztb/v7+ysjI0JAhQyRJpaWlat++vZ1lAgBqAQEEAAAAbOPl5VX52cPDQyUlJTJNUw6HQykpKTZWBgCobSzBAAAAQJ1p2bKl8vPzr9mne/fuOnXqVGUAUVxcrAMHDtRFeQAACzEDAgAAAHWmbdu26tevn4KCguTt7a1/+Zd/uaJPs2bNtGrVKs2ePVt5eXkqKSnRk08+KYfDYUPFAIDaYlTsNFyfhYWFmW632+4yAAAAAADAZQzDSDVNM+x6/ViCAQAAgHolb906ZQ0arEM9eylr0GDlrVtnd0kAgFrAEgwAAADUG3nr1iln/nMyi4okSSXff6+c+c9JkvyGD7ezNADATWIGBAAAAOqNH199rTJ8qGAWFenHV1+zqSIAQG0hgAAAAEC9UZKTU6N2AEDDQQABAACAeqNp+/Y1agcANBwEEADQAN1zzz12lwAAlmj31JMymjev0mY0b652Tz1pU0UAgNrCJpQA0ADt2rXL7hIAwBIVG03++OprKsnJUdP27dXuqSfZgBIAbgEEEADQAPn6+qqgoEA5OTl66KGHdO7cOZWUlOjNN99UVFSU3eUBwE3xGz6cwAEAbkEEEADQgL333nuKjY3Vs88+q9LSUl24cMHukgAAAIBqEUAAQAMWHh6u6dOnq7i4WKNGjZLL5bK7JAAAAKBabEIJAA1YdHS0PvvsM3Xs2FGTJk3SsmXL7C4JAAAAqBYBBAA0YMeOHVO7du00Y8YMPfLII9q7d6/dJQEAAADVYgkGADRgycnJWrRokTw9PeXr68sMCAAAANRbzIBArfLw8JDL5VJwcLD69OlTa0cFPvroozp48KAkKSAgQKdPn66V9wINVUFBgSRpypQpysjI0L59+7Rjxw4FBgbaXBlwc2JiYuR2u69of/vtt/X444/bUBEAAKgtzIBArfL29lZaWpokacOGDZo3b562b99epU9paak8PDxq9N6//e1vtVYjcCs4v+9HnduQrdLci/KzXZXjAAAgAElEQVTw91Kr2AC1CGlnd1nATSktLbW7BAAAYCFmQMAy586dU+vWrSWVTRMfOHCgHn74YfXu3VuSNGrUKIWGhsrhcCgxMVGS9PHHH8vlcsnlcql79+6Vv8292m/EgMbo/L4flfthlkpzL0qSSnMvKvfDLJ3f96PNlaEx+9Of/qSEhARJ0lNPPaVBgwZJkrZs2aKJEycqKSlJvXv3VlBQkObOnVv5nK+vr5577jndfffdSklJqfLOpUuXqlu3bhowYIA+//zzuhsMAACwBDMgUKsKCwvlcrlUVFSknJwcbd26tfLenj17lJGRURkqLFmyRG3atFFhYaHCw8M1ZswYjRgxQiNGjJAkPfjggxowYIAt4wDqs3MbsmUWX6rSZhZf0rkN2cyCgG2io6P1yiuvaPbs2XK73bp48aKKi4u1c+dO3XXXXZo7d65SU1PVunVr3Xvvvfroo480atQonT9/XkFBQXrhhReqvC8nJ0fPP/+8UlNT5efnp4EDByokJMSm0QEAgNrADAjUqoolGJmZmfrHP/6hyZMnyzRNSVLfvn2rrE9PSEhQcHCwIiIidOLECWVlZVXe+9Of/iRvb2899thjdT4GoL6rmPlwo+1AXQgNDVVqaqry8/Pl5eWlyMhIud1u7dixQ/7+/oqJidHtt9+upk2basKECfrss88kle0dNGbMmCve98UXX1Q+06xZMz300EN1PSQAAFDLmAEBy0RGRur06dM6deqUJKlFixaV95KTk7V582alpKTIx8dHMTExKioqklQ2XfeDDz6o/MspgKo8/L2qDRs8/L1sqAYo4+npqYCAAC1dulT33HOPnE6ntm3bpiNHjujOO+9Uampqtc81b978qvsCGYZhZckAAKCOMQMClsnMzFRpaanatm17xb28vDy1bt1aPj4+yszM1O7duyVJx44d029/+1utXLlS3t7edV0y0CC0ig2Q4Vn1/74NzyZqFRtgT0FAuejoaMXHxys6OlpRUVFavHixXC6XIiIitH37dp0+fVqlpaVKSkq67hK7u+++W8nJyTpz5oyKi4v1wQcf1NEoAACAVZgBgVpVsQeEJJmmqXfeeafa32zFxcVp8eLFcjqd6t69uyIiIiSVHbN25swZjR49WpLUoUMHffrpp3U3AKABqNjngVMwUN9ERUVp4cKFioyMVIsWLdS8eXNFRUWpffv2eumllzRw4ECZpqn7779fI0eOvOa72rdvrwULFigyMlLt27dXnz59OCUDAIAGzqhYn1+fhYWFmZyAAABA4/H1Fz8oZe0RFZy9KN82Xooc2VXd7r7D7rIAAEA1DMNINU0z7Hr9mAGBeu3Qjm3asWKZ8s+cVsu2tylq/GT1jBpod1kAAAt9/cUP2rY8UyU/lZ32UnD2orYtz5QkQggAABow9oBAvXVoxzZtTHxd+adPSaap/NOntDHxdR3asc3u0gAAFkpZe6QyfKhQ8tMlpaw9YlNFAACgNhBAoN7asWKZSn6qutN/yU8XtWPFMpsqAgDUhYKz1R8pe7V2ALe+mJgY/dIl2ffff79yc3NruSIAvwRLMFBv5Z85XaN2AMCtwbeNV7Vhg28bjpoFUHNsaA7UH8yAQL3Vsu1tNWoHANwaIkd2VdNmVf+K0rRZE0WO7GpTRQDqSnZ2tnr06KEpU6bI6XRq7NixunDhQpU+s2bNUlhYmBwOh55//nlJ0pYtWypPUZOkTZs26YEHHpAkBQQE6PTp08rOzlbPnj01Y8YMORwO3XvvvSosLJQkffnll3I6nYqMjNScOXMUFBRURyMGGhcCCNRbUeMnq2mzqr/tatrMS1HjJ9tUEQCgLnS7+w4NnNCjcsaDbxsvDZzQgw0ogUbi8OHDmjlzptLT09WqVSu98cYbVe4vXLhQbrdb6enp2r59u9LT0zVo0CAdOnRIp06dkiQtXbpU06ZNu+LdWVlZeuyxx3TgwAH5+/tr9erVkqRp06Zp8eLFSklJqfYIeQC1gyUYqLcqTrvgFAwAaHy63X0HgQPQSHXu3Fn9+vWTJE2cOFEJCQlV7q9cuVKJiYkqKSlRTk6ODh48KKfTqUmTJundd9/VtGnTlJKSomXLrtw3LDAwUC6XS5IUGhqq7Oxs5ebmKj8/X/fcc48k6eGHH9Ynn3xi8SiBxokAAvVaz6iBBA4AAACNiGEYV73+5ptvFB8fry+//FKtW7fW1KlTVVRUJKlsFsPw4cPVvHlzjRs3Tk2bXvmfOl5e/zu71sPDQ4WFhTJN06KRALgcSzAAAAAA1BvHjx9XSkqKJCkpKUn9+/evvHfu3Dm1aNFCfn5+OnnypNavX195r0OHDurQoYNefPFFTZ069Ya/r3Xr1mrZsqV2794tSVqxYkXtDATAFQggAAAAANQbPXv21DvvvCOn06mzZ89q1qxZlfeCg4MVEhIih8Oh6dOnVy7VqDBhwgR17txZvXr1qtF3vvXWW5o5c6YiIyNlmqb8/PxqZSwAqjIawpSjsLAw85ee+wsAAACgYcjOztawYcOUkZHxi55//PHHFRISokceeaRGzxUUFMjX11eS9PLLLysnJ0d//vOff1ENQGNkGEaqaZph1+vHHhAAAAAAGrzQ0FC1aNFCr7zySs0eTF+pv8f/H7208XuVGJ761V299PbqDdYUCTRyzIAAAAAA0Dilr5TWzZaKC/+3zdNbGp4gOR+0ry6ggbnRGRDsAQEAuCWUlpbaXQIAoKHZ8kLV8EEqu97ygj31ALc4AggAQIMwatQohYaGyuFwKDExUZLk6+ur5557TnfffbdSUlKUmpqqAQMGKDQ0VLGxscrJybG5agBAvZb3bc3aAdwU9oAAADQIS5YsUZs2bVRYWKjw8HCNGTNG58+fV1BQkF544QUVFxdrwIABWrt2rW6//Xa9//77evbZZ7VkyRK7SwcA1Fd+naS8E9W3A6h1BBAAgAYhISFBa9askSSdOHFCWVlZ8vDw0JgxYyRJhw8fVkZGhoYMGSKpbElG+/btbasXANAADH6u+j0gBj9nX03ALYwAAgBQ7yUnJ2vz5s1KSUmRj4+PYmJiVFRUpObNm8vDw0OSZJqmHA6HUlJSbK4WANBgVGw0ueWFsmUXfp3Kwgc2oAQsQQABAKj38vLy1Lp1a/n4+CgzM1O7d+++ok/37t116tQppaSkKDIyUsXFxfr666/lcDhsqBgA0GA4HyRwAOoIm1ACAOq9uLg4lZSUyOl0av78+YqIiLiiT7NmzbRq1SrNnTtXwcHBcrlc2rVrlw3VAgAAoDrMgAAA1HteXl5av379Fe0FBQWSpEM7tmnHimXKP3NaE3t3VdT4yeoZNbCuywQAAMA1MAMCANCgHdqxTRsTX1f+6VOSaSr/9CltTHxdh3Zss7s0AABuSExMjNxut2XvT05O1rBhwyx7P3CjCCAAAA3ajhXLVPLTxSptJT9d1I4Vy2yqCAAAANUhgAAANGj5Z07XqB0AALtkZ2erR48emjJlipxOp8aOHasLFy5U6TNr1iyFhYXJ4XDo+eeflyRt2bJFo0ePruyzadMmPfDAA5KkjRs3KjIyUn369NG4ceMqlyf+4x//UI8ePdS/f399+OGHdTRC4NoIIAAADVrLtrfVqB0AADsdPnxYM2fOVHp6ulq1aqU33nijyv2FCxfK7XYrPT1d27dvV3p6ugYNGqRDhw7p1KlTkqSlS5dq2rRpOn36tF588UVt3rxZe/fuVVhYmP77v/9bRUVFmjFjhtatW6cdO3bohx9+sGOowBUIIAAADVrU+Mlq2syrSlvTZl6KGj/ZpooAALi6zp07q1+/fpKkiRMnaufOnVXur1y5Un369FFISIgOHDiggwcPyjAMTZo0Se+++65yc3OVkpKi++67T7t379bBgwfVr18/uVwuvfPOOzp27JgyMzMVGBiou+66S4ZhaOLEiXYMFbgCp2AAABq0itMuKk7BaNn2Nk7BAADUW4ZhXPX6m2++UXx8vL788ku1bt1aU6dOVVFRkSRp2rRpGj58uJo3b65x48apadOmMk1TQ4YMUVJSUpV3pqWlXfE9QH1AAAEAaPB6Rg0kcAAANAjHjx9XSkqKIiMjlZSUpP79+2vdunWSpHPnzqlFixby8/PTyZMntX79esXExEiSOnTooA4dOujFF1/Upk2bJEkRERF67LHH9M9//lP/+q//qgsXLujbb79Vjx499M033+jIkSPq2rXrFQEFYBeWYAAAAABAHenZs6feeecdOZ1OnT17VrNmzaq8FxwcrJCQEDkcDk2fPr1yqUaFCRMmqHPnzurVq5ck6fbbb9fbb7+tX//613I6nYqIiFBmZqaaN2+uxMREDR06VP3799evfvWrOh0jcDWGaZp213BdYWFhppXn4gIAAACA1bKzszVs2DBlZGT8oucff/xxhYSE6JFHHrl2x/SV0pYXpLxvJb9O0uDnJOeDv+g7gRthGEaqaZph1+vHDIh6aM2aNTIMQ5mZmZZ/V3Z2tt57773Ka7fbrdmzZ1v+vQAAAABuXGhoqNLT06+/oWT6SmndbCnvhCSz7Oe62WXtgM2YAVEPPfjgg8rJydHgwYO1YMGCKvdKS0vl4eFRa9+VnJys+Ph4ffLJJ7X2TgAAAAA2eTWoPHy4jF9n6alfNvMCuB5mQDRQBQUF+vzzz/XWW29pxYoVkspCgoEDB+rhhx9W7969JUl/+MMf1KNHDw0ZMkS//vWvFR8fL0k6cuSI4uLiFBoaqqioqMpZFFOnTtXs2bN1zz33qEuXLlq1apUk6ZlnntGOHTvkcrn06quvKjk5WcOGDZMkLViwQNOnT1dMTIy6dOmihISEyjpHjRql0NBQORwOJSYm1tmfDwAAAIBryPu2Zu1AHeIUjHrmo48+UlxcnLp166Y2bdpo7969kqQ9e/YoIyNDgYGBcrvdWr16tfbt26eSkhL16dNHoaGhkqSZM2dq8eLFuuuuu/TFF1/ot7/9rbZu3SpJysnJ0c6dO5WZmakRI0Zo7Nixevnll6vMgEhOTq5ST2ZmprZt26b8/Hx1795ds2bNkqenp5YsWaI2bdqosLBQ4eHhGjNmjNq2bVt3f1AAAAAAruTX6SozIDrVfS3AZQgg6pmkpCQ9+eSTkqTx48crKSlJQ4cOVd++fRUYGChJ2rlzp0aOHClvb29J0vDhwyWVzZ7YtWuXxo0bV/m+ixcvVn4eNWqUmjRpol69eunkyZM3VM/QoUPl5eUlLy8vtWvXTidPnlSnTp2UkJCgNWvWSJJOnDihrKwsAggAAADAboOfK9vzobjwf9s8vcvaAZsRQNQjZ86c0datW5WRkSHDMFRaWirDMHT//ferRYsWlf2utm/HpUuX5O/vr7S0tGrve3l5Xfcd13rGw8NDJSUlSk5O1ubNm5WSkiIfHx/FxMSoqKjoht4HAAAAwEIVp11wCgbqIfaAqEdWrVqlyZMn69ixY8rOztaJEycUGBionTt3VunXv39/rVu3TkVFRSooKNDf//53SVKrVq0UGBioDz74QFJZyLB///5rfmfLli2Vn59fozrz8vLUunVr+fj4KDMzU7t3767R8wAAAAAs5HywbMPJBbllPwkfUE8QQNQjSUlJGj16dJW2MWPGVDkmU5LCw8M1YsQIBQcH64EHHlBYWJj8/PwkScuXL9dbb72l4OBgORwOrV279prf6XQ61bRpUwUHB+vVV1+9oTrj4uJUUlIip9Op+fPnKyIiogajBAAAAAA0RhzD2UAVFBTI19dXFy5cUHR0tBITE9WnTx+7ywJQB+655x7t2rXL7jIAAAAASTd+DCd7QDRQM2fO1MGDB1VUVKQpU6bUafjw0b7vtGjDYX2fW6gO/t6aE9tdo0I61tn3A40d4QMAAAAaIgKIBuryZRl15aN932neh1+psLhUkvRdbqHmffiVJBFCAHXE19dXBQUFysnJ0UMPPaRz586ppKREb775pqKiouwuDwAAAKgWe0CgRhZtOFwZPlQoLC7Vog2HbaoIaLzee+89xcbGKi0tTfv375fL5bK7JAAAAOCqmAGBGvk+t7BG7QCsEx4erunTp6u4uFijRo0igAAAAEC9xgwI1EgHf+8atQOwTnR0tD777DN17NhRkyZN0rJly+wuCQAAALgqAgjUyJzY7vL29KjS5u3poTmx3W2qCGi8jh07pnbt2mnGjBl65JFHtHfvXrtLAgAAAK6KJRiokYqNJjkFA7BfcnKyFi1aJE9PT/n6+jIDAgAAAPWaYZqm3TVcV1hYmOl2u+0uAwAAAAAAXMYwjFTTNMOu148ZEADQwHy07ztmIQEAAKDBIYAAgAbko33fad6HX1Ueh/tdbqHmffiVJBFCAAAAoF5jE0oAaEAWbThcGT5UKCwu1aINh22qCAAAALgxBBAA0IB8n1tYo3YAAACgviCAAIAGpIO/d43aAQAAgPqCAAIAGpA5sd3l7elRpc3b00NzYrvbVBEAAABwY9iEEgAakIqNJjkFAwAAAA0NAQQANDCjQjoSOAAAAKDBYQkGAAAAAACwHAEEAAAAAACwHAEEAAAAAACwHAEEAAAAAACwHAEEAAAAAACwHAEEAAAAAACwHAEEAAAAAACwHAEEAAAAAACwHAEEAAAAAACwHAEEAAAAAACwHAEEAAAAAACwHAEEAAAAAACwHAEEAAAAAACwHAEEAAAAAACwHAEEAAAAAACwHAEEAAAAAACwHAEEAAAAAACwHAEEAAAAAACwHAEEAACAxWJiYuR2u+0uAwAAWxFAAAAAAAAAyxFAAAAA1JLs7Gz16NFDU6ZMkdPp1NixY3XhwoUqfWbNmqWwsDA5HA49//zzkqQtW7Zo9OjRlX02bdqkBx54oE5rBwDAarYFEIZhxBmGcdgwjH8ahvGMXXUAAADUpsOHD2vmzJlKT09Xq1at9MYbb1S5v3DhQrndbqWnp2v79u1KT0/XoEGDdOjQIZ06dUqStHTpUk2bNs2O8gEAsIwtAYRhGB6S/j9J90nqJenXhmH0sqMWAACA2tS5c2f169dPkjRx4kTt3Lmzyv2VK1eqT58+CgkJ0YEDB3Tw4EEZhqFJkybp3XffVW5urlJSUnTffffZUT4AAJZpatP39pX0T9M0j0qSYRgrJI2UdNCmegAAAGqFYRhXvf7mm28UHx+vL7/8Uq1bt9bUqVNVVFQkSZo2bZqGDx+u5s2ba9y4cWra1K6/pgEAYA27lmB0lHTiZ9fflrcBAAA0aMePH1dKSookKSkpSf3796+8d+7cObVo0UJ+fn46efKk1q9fX3mvQ4cO6tChg1588UVNnTq1rssGAMBydgUQRjVtZpUOhjHTMAy3YRjuivWQAAAA9V3Pnj31zjvvyOl06uzZs5o1a1blveDgYIWEhMjhcGj69OmVSzUqTJgwQZ07d1avXqxMBQDceuya2/etpM4/u+4k6fufdzBNM1FSoiSFhYVVCScAAADqqyZNmmjx4sVV2pKTkys/v/3221d9dufOnZoxY4ZFlQEAYC+7ZkB8KekuwzACDcNoJmm8pI9tqgUAAMA2OT+s1eefR6lbNy/t3Pk/GvxvfnaXBACAJWyZAWGaZolhGI9L2iDJQ9IS0zQP2FELAABAbQkICFBGRsYN98/5Ya0yM5/VpUuFenNxJ0nSN98skJeXp9rfMdKqMgEAsIVt2yubpvmppE/t+n4AAAC7HT0Sr0uXCqu0XbpUqKNH4gkgAAC3HLuWYAAAADR6RRdzatQOAEBDRgABAABgk+Ze7WvUDgBAQ0YAAQAAYJMuXZ9WkybeVdqaNPFWl65P21QRAADWsW0PCAAAgMauYp+Ho0fiVXQxR8292qtL16fZ/wEAcEsigAAAALBR+ztGEjgAABoFlmAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAAAADLEUAAAAAA+P/bu/fwqKp7/+PvRcAEAwKKSgAVoQoCxhAioggoVNEqKhYUixW12mqPiPbgUevTGn/W1lOxIu1Rq62XeqnaWKuIraiFCoqXBCgKigjNqUo4oggFFEzI+v2RSQoYbpLN5PJ+PU8eZ9ZeM/PdzHbP5JO11pakxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASEqr5cuXM3r0aLp160bPnj35xje+wd13382pp55aa/+LLrqIhQsXAtClSxc+/vjjL/UpLCxk4sSJidYtSZIkaec0T3cBkpquGCMjRoxg7NixPProowDMmzePKVOmbPUxv/nNb3ZXeZIkSZLqkCMgJKXN9OnTadGiBZdccklNW15eHgMHDmTt2rWMHDmSHj16MGbMGGKMABx33HEUFxd/6bluuukmunfvzte//nUWLVq02/ZBkiRJ0o5xBISktHnrrbfo27dvrdvmzp3LggUL6NixIwMGDODll1/m2GOPrbVvSUkJjz76KHPnzqWiooL8/PytPq8kSZKk9HAEhKR6qV+/fnTu3JlmzZqRl5dHaWnpVvvOnDmTESNGsOeee7LXXntx2mmn7b5CJUmSJO0QAwhJadOrVy9KSkpq3ZaZmVlzOyMjg4qKim0+VwihTmuTpIZsa9PVmprJkydz2GGHMWbMmJ163IwZM3jllVdq7p9//vkUFRXt8ONLS0vp3bt3zXNtbWFlSWpqDCAkpc2QIUPYsGED99xzT03bG2+8wd/+9redep5Bgwbx5JNP8vnnn7NmzZptLmIpSWo67rjjDp599lkefvjhnXrclgGEJKluGEBISpsQAk8++STPP/883bp1o1evXhQWFtKxY8edep78/HzOPvts8vLy+OY3v8nAgQMTqliS6pfS0lJ69OjB2LFjyc3NZeTIkXz22Web9bn00kspKCigV69eXH/99QC8+OKLjBgxoqbP888/z5lnnrlba0/aJZdcwtKlSznttNO49dZbOeOMM8jNzaV///7Mnz8fgJUrV36pvbS0lLvuuovbbruNvLw8Zs6cCcALL7zAwIEDOfTQQ3nmmWeAqn//gQMHkp+fT35+vqGFJG1HqF5Zvj4rKCiIDiOUJEnaXGlpKQcffDCzZs1iwIABXHjhhfTs2ZNnnnmGiRMnUlBQwMqVK9l7773ZuHEjQ4cOZfLkyRx++OEcdthhzJw5k3333ZdvfetbnHPOOQwfPjzdu1SnunTpQnFxMTfccAPt27fn+uuv569//Ss/+MEPmDdvHuPGjau1vbCwkFatWjFhwgSgagrG8uXLefbZZ1myZAnHH3887733HpWVlTRr1oysrCwWL17MOeecQ3FxMaWlpZx66qm89dZbzJgxg4kTJ9aEFpLUGIUQSmKMBdvr5wgISQ3eE8tXUvDKAnKmz6PglQU8sXxlukuSpN3mgAMOYMCAAQCce+65zJo1a7Ptjz/+OPn5+fTp04cFCxawcOFCQgh8+9vf5qGHHmLVqlXMnj2bk08+OR3l7xazZs3i29/+NlA1/e+TTz5h9erVW22vzVlnnUWzZs045JBD6Nq1K++88w7l5eVcfPHFHH744YwaNYqFCxfutn2SpIbIy3BKatCeWL6SCYve5/PKqtFcH2woZ8Ki9wH4Zoe901maJO0WWy7Cu+n9f/zjH0ycOJE33niDdu3acf7557N+/XoALrjgAoYPH05WVhajRo2iefPG+7WwthG/IYStttemtn/n2267jf3335+///3vVFZWkpWVVTcFS1Ij5QgISQ3az5aW1YQP1T6vjPxsaVmaKpKk3euf//wns2fPBuD3v/89xx57bM22f/3rX2RnZ9OmTRv+7//+jz//+c812zp27EjHjh35yU9+wvnnn7+7y96tBg0aVLMQ5YwZM2jfvj177bXXVttbt27NmjVrNnuOP/zhD1RWVrJkyRKWLl1K9+7dWb16NTk5OTRr1owHH3yQjRs37vZ9k6SGxABCUoP24YbynWqXpMbmsMMO44EHHiA3N5eVK1dy6aWX1mw74ogj6NOnD7169eLCCy+smapRbcyYMRxwwAH07Nlzd5e9WxUWFlJcXExubi7XXHMNDzzwwDbbhw8fzpNPPrnZIpTdu3dn8ODBnHzyydx1111kZWXx/e9/nwceeID+/fvz7rvvkp2dnbZ9lKSGwEUoJTVoBa8s4INawobOmS0oPqZXGiqSpN1n08UOv4rLLruMPn368J3vfKeOK2vapi6dyu1zbmf5uuV0yO7A+PzxnNL1lHSXJUmJcRFKSU3CtV1zaNls83m5LZsFru2ak6aKJKl++9PcDxlw81/J7PA1Hpz6Env1Pj7dJTUqU5dOpfCVQsrWlRGJlK0ro/CVQqYunZru0iQp7RwBIanBe2L5Sn62tIwPN5TTKbMF13bNcQFKSarFn+Z+yLV/fJPPy/+9VkHLFhn87MzDOaNPpzRW1nicWHQiZeu+vA5RTnYO00ZOS0NFkpS8HR0B0XiXO5bUZHyzw94GDpK0A255btFm4QPA5+UbueW5RQYQdWT5uuU71S5JTYlTMCRJkpqIZas+36l27bwO2R12ql2SmhIDCEmSpCaiY9uWO9WunTc+fzxZGVmbtWVlZDE+f3yaKpKk+sMAQpIkqYm4alh3WrbI2KytZYsMrhrWPU0VNT6ndD2FwmMKycnOIRDIyc6h8JhCr4IhSbgGhCRJUpNRvc7DLc8tYtmqz+nYtiVXDevu+g917JSupxg4SFItDCAkSZKakDP6dDJwkCSlhVMwJEmqJ1atWsUdd9wBwIwZMzj11FPTXJEkSVLdMYCQJKme2DSAkCRJamwMICRJqieuueYalixZQl5eHldddRVr165l5MiR9OjRgzFjxhBjBKCkpITBgwfTt29fhg0bRllZWZorlyRJ2j4DCEmS6ombb76Zbt26MV18OScAACAASURBVG/ePG655Rbmzp3LpEmTWLhwIUuXLuXll1+mvLyccePGUVRURElJCRdeeCHXXXddukuXJEnaLhehlCSpnurXrx+dO3cGIC8vj9LSUtq2bctbb73FCSecAMDGjRvJyclJZ5mSJEk7xABCkqR6KjMzs+Z2RkYGFRUVxBjp1asXs2fPTmNlkiRJO88pGJIk1ROtW7dmzZo12+zTvXt3VqxYURNAlJeXs2DBgt1RniRJ0i5xBIQkSfXEPvvsw4ABA+jduzctW7Zk//33/1KfPfbYg6KiIi6//HJWr15NRUUFV1xxBb169UpDxZIkSTsuVK+oXZ8VFBTE4uLidJchSZIkSZK2EEIoiTEWbK+fIyAkSWpA5s+fz4svvsjq1atp06YNQ4cOJTc3N91lSZIkbZcBhCRJDcT8+fOZMmUK5eXlAKxevZopU6YAGEJIkqR6z0UoJUlqIF588cWa8KFaeXk5L774YpoqkiRJ2nEGEJIkNRCrV6/eqXZJkqT6xABCkqQGok2bNjvVLkmSVJ8YQEiS1EAMHTqUFi1abNbWokULhg4dmqaKJEmSdpyLUEqS1EBULzTpVTAkSVJDZAAhSVIDkpuba+AgSZIaJKdgSJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJIkSZKkxBlASJK0hVWrVnHHHXcAMGPGDE499dQ0VyRJktTwGUBIkrSFTQMISZIk1Q0DCEmStnDNNdewZMkS8vLyuOqqq1i7di0jR46kR48ejBkzhhgjACUlJQwePJi+ffsybNgwysrK0ly5JElK0qRJk/jss8/SXUaDZQAhSdIWbr75Zrp168a8efO45ZZbmDt3LpMmTWLhwoUsXbqUl19+mfLycsaNG0dRURElJSVceOGFXHfddekuXZIkJWhbAcTGjRt3czUNT/N0FyBJUn3Xr18/OnfuDEBeXh6lpaW0bduWt956ixNOOAGo+tKRk5OTzjIlSVIdWrduHWeddRYffPABGzduZNSoUSxbtozjjz+e9u3bM336dFq1asUPfvADnnvuOW699VY2bNjAhAkTqKio4Mgjj+TOO+8kMzOTLl26MHbsWKZMmUJ5eTl/+MMf6NGjBytWrOBb3/oWn3zyCUceeSR/+ctfKCkpoX379une/UQ4AkKSpO3IzMysuZ2RkUFFRQUxRnr16sW8efOYN28eb775JtOmTUtjlZIkqS795S9/oWPHjvz973/nrbfe4oorrqBjx45Mnz6d6dOnA1UhRe/evXnttdcoKCjg/PPP57HHHuPNN9+koqKCO++8s+b52rdvz5w5c7j00kuZOHEiADfccANDhgxhzpw5jBgxgn/+859p2dfdxQBCkqQttG7dmjVr1myzT/fu3VmxYgWzZ88GoLy8nAULFuyO8iQprYqLi7n88suBqisFvfLKK2muSErG4YcfzgsvvMDVV1/NzJkzadOmzZf6ZGRk8M1vfhOARYsWcfDBB3PooYcCMHbsWF566aWavmeeeSYAffv2pbS0FIBZs2YxevRoAE466STatWuX5C6lnVMwJEnawj777MOAAQPo3bs3LVu2ZP/99/9Snz322IOioiIuv/xyVq9eTUVFBVdccQW9evVKQ8WStPsUFBRQUFAAVAUQrVq14phjjtnhx1dUVNC8ub+GqP479NBDKSkp4dlnn+Xaa6/lxBNP/FKfrKwsMjIyAGoWqd6a6hGV1aMpd+QxjY3/50uSVItHHnmk1vZf/epXNbfz8vI2+8uGJDVUN954Iw8//DAHHHAA7du3p2/fvjzzzDNMnDiRgoICPv74YwoKCigtLWXGjBlMnDiRX/3qV9x1111kZGTw0EMP8ctf/pJVq1bxk5/8hC+++IJ99tmHhx9+mP3335/CwkKWLVtGaWkp7du33+o5VqpPli1bxt577825555Lq1atuP/++2tGSda2RkOPHj0oLS3lvffe42tf+xoPPvgggwcP3uZrHHvssTz++ONcffXVTJs2jU8//TSp3akXDCAkSfoKnli+kp8tLePDDeV0ymzBtV1z+GaHvdNdliTttOLiYp544gnmzp1LRUUF+fn59O3bd7uP69KlC5dccgmtWrViwoQJAHz66ae8+uqrhBD4zW9+w89//nNuvfVWoOrSxbNmzaJly5aJ7o9UV958802uuuoqmjVrRosWLbjzzjuZPXs2J598Mjk5OTXrQFTLysrivvvuY9SoUTWLUF5yySXbfI3rr7+ec845h8cee4zBgweTk5ND69atk9yttDKAkCRpJz2xfCUTFr3P55VVwyY/2FDOhEXvAxhCSGpwZs2axemnn14TDAwfPvwrP9cHH3zA2WefTVlZGV988QUHH3xwzbbTTjvN8EENyrBhwxg2bNhmbQUFBYwbN67m/tq1azfbPnToUObOnful56pe86H6OWbMmAFAmzZteO6552jevDmzZ89m+vTpmy1+3di4CKUkSTvpZ0vLasKHap9XRn62tCxNFUn113HHHUdxcTFQ9Rfzjz/+OM0VaUtbm4PevHlzKisrAVi/fv0OPde4ceO47LLLePPNN/n1r3+92eOys7N3vVipkSkuuZeePdvSrVsmF154Ij+7+dx0l5QoAwhJknbShxvKd6pdkuqzY489lilTprB+/XrWrl3L1KlTgarAqKSkBICioqJaH7vlVYNWr15Np06dAHjggQcSrlxq2MqWP8UXX/wPd961P3ff05lf/c9+ZGXdT9nyp9JdWmIMICRJ2kmdMlvsVLvUGPz85z9n8uTJAFx55ZUMGTIEgBdffJFzzz2XadOmcfTRR5Ofn8+oUaO+NCxZ9deRRx7JaaedxhFHHMGZZ55JQUEBbdq0YcKECdx5550cc8wxWx25Mnz4cJ588kny8vKYOXMmhYWFjBo1ioEDB9a6SJ+kf1u6ZCKVlZ9v1lZZ+TlLl0xMU0XJCw3hsh8FBQWxeuieJEnptuUaEAAtmwUmdj/ANSDUaL366qvceuut/OEPf2DgwIFs2LCBl19+mZ/+9KdkZWUxdepU/vznP5Odnc1///d/s2HDBn784x9z3HHH1VxJoUuXLhQXF/uLaT20du1aWrVqxWeffcagQYO4++67yc/PT3dZUqP24l+/BtT2+3hg6JD3dnc5uySEUBJjLNhePxehlCRpJ1WHDF4FQ01J3759KSkpYc2aNWRmZpKfn09xcTEzZ87ktNNOY+HChQwYMACAL774gqOPPjrNFWtnfPe732XhwoWsX7+esWPH1kn4sHrKFD66bRIVZWU0z8lhvyuvoM0uLHApNTZZmTms37Cs1vbGygBCkqSv4Jsd9jZwUJPSokULunTpwn333ccxxxxDbm4u06dPZ8mSJRx88MGccMIJ/P73v093mfqKHnnkkTp9vtVTplD2ox8TU4tQVixbRtmPfgxgCCGldO02gXfeuW6zaRjNmrWka7cJaawqWa4BIUmSpB0yaNAgJk6cyKBBgxg4cCB33XUXeXl59O/fn5dffpn33qsaMvzZZ5/x7rvvprlapdNHt02qCR+qxfXr+ei2SWmqSKp/cjqcTo8eN5GV2REIZGV2pEePm8jpcHq6S0uMIyAkSZK0QwYOHMhNN93E0UcfTXZ2NllZWQwcOJB9992X+++/n3POOYcNGzYA8JOf/IRDDz00zRUrXSrKar8s8dbapaYqp8PpjTpw2JKLUEqSJEmqU4uHDKVi2Zfntjfv2JFD/vpiGiqSlCQXoZQkSVJavfvacmY/tYS1KzfQau9Mjj69G4ce1SHdZWk32O/KKzZbAwIgZGWx35VXpLEqSelmACFJkqQ69+5ry5n+8DtUfFEJwNqVG5j+8DsAhhBNQPVCk14FQ9KmDCAkSZJU52Y/taQmfKhW8UUls59aYgDRRLQZPtzAQdJmvAqGJEmS6tzalRt2ql2S1PgZQEiSJKnOtdo7c6faJUmNnwGEJEmS6tzRp3ej+R6bf9Vsvkczjj69W5oqkiSlm2tASJIkqc5Vr/PgVTAkSdUMICRJkpSIQ4/qYOAgSarhFAxJkrRNq1at4o477gBgxowZnHrqqWmuSJIkNUS7FECEEG4JIbwTQpgfQngyhNB2k23XhhDeCyEsCiEM26T9pFTbeyGEa3bl9SVJUvI2DSAkSZK+ql0dAfE80DvGmAu8C1wLEELoCYwGegEnAXeEEDJCCBnA/wAnAz2Bc1J9JUlSPXXNNdewZMkS8vLyuOqqq1i7di0jR46kR48ejBkzhhgjACUlJQwePJi+ffsybNgwysrK0ly5JEmqT3YpgIgxTosxVqTuvgp0Tt0+HXg0xrghxvgP4D2gX+rnvRjj0hjjF8Cjqb6SJKmeuvnmm+nWrRvz5s3jlltuYe7cuUyaNImFCxeydOlSXn75ZcrLyxk3bhxFRUWUlJRw4YUXct1116W7dEmSVI/U5SKUFwKPpW53oiqQqPZBqg3g/S3aj6rtyUII3wW+C3DggQfWYZmSJGlX9OvXj86dq/7mkJeXR2lpKW3btuWtt97ihBNOAGDjxo3k5OSks0xJklTPbDeACCG8ANS2fPF1McanUn2uAyqAh6sfVkv/SO0jLmJtrxtjvBu4G6CgoKDWPpIkaffLzMysuZ2RkUFFRQUxRnr16sXs2bPTWJkkSarPthtAxBi/vq3tIYSxwKnA0Fg9CbRqZMMBm3TrDCxL3d5auyRJqodat27NmjVrttmne/furFixgtmzZ3P00UdTXl7Ou+++S69evXZTlZIkqb7bpSkYIYSTgKuBwTHGzzbZ9DTwSAjhF0BH4BDgdapGRhwSQjgY+JCqhSq/tSs1SJKkZO2zzz4MGDCA3r1707JlS/bff/8v9dljjz0oKiri8ssvZ/Xq1VRUVHDFFVcYQEiSpBrh34MWvsKDQ3gPyAQ+STW9GmO8JLXtOqrWhagArogx/jnV/g1gEpAB3BtjvGl7r1NQUBCLi4u/cp2SJEmSJCkZIYSSGGPBdvvtSgCxuxhASJJUv01dOpXb59zO8nXL6ZDdgfH54zml6ynpLkuSJO0GOxpA1OVVMCRJUhM0delUCl8pZP3G9QCUrSuj8JVCAEMISZJUo7arUkiSJO2w2+fcXhM+VFu/cT23z7k9TRVJkqT6yABCkiTtkuXrlu9UuyRJapoMICRJ0i7pkN1hp9olSVLTZAAhSZJ2yfj88WRlZG3WlpWRxfj88WmqSJIk1UcuQilJknZJ9UKTXgVDkiRtiwGEJEnaZad0PcXAQZIkbZNTMCRJkiRJUuIMICRJkiRJUuIMICRJkiRJUuIMICRJkiRJUuIMICSpCSssLGTixIk73H/ZsmWMHDkywYokSZLUWBlASJJ2SEVFBR07dqSoqCjdpUiSJKkBMoCQpCbmpptuonv37nz9619n0aJFABx33HEUFxcD8PHHH9OlSxcA7r//fkaNGsXw4cM58cQTKS0tpXfv3jXbzjzzTE466SQOOeQQ/uu//qvmNX77299y6KGHctxxx3HxxRdz2WWX7d6dlCRJUr3TPN0FSJJ2n5KSEh599FHmzp1LRUUF+fn59O3bd5uPmT17NvPnz2fvvfemtLR0s23z5s1j7ty5ZGZm0r17d8aNG0dGRgY33ngjc+bMoXXr1gwZMoQjjjgiwb2SJElSQ2AAIUlNyMyZMxkxYgR77rknAKeddtp2H3PCCSew995717pt6NChtGnTBoCePXvyv//7v3z88ccMHjy45jGjRo3i3XffraM9kCRJUkPlFAxJamJCCF9qa968OZWVlQCsX79+s23Z2dlbfa7MzMya2xkZGVRUVBBjrKNKJUmS1JgYQEhSEzJo0CCefPJJPv/8c9asWcOUKVMA6NKlCyUlJQC7vMhkv379+Nvf/sann35KRUUFTzzxxC7XLUmSpIbPKRiS1ITk5+dz9tlnk5eXx0EHHcTAgQMBmDBhAmeddRYPPvggQ4YM2aXX6NSpEz/84Q856qij6NixIz179qyZpiFJkqSmKzSEobIFBQWxenV2SVL9t3btWlq1akVFRQUjRozgwgsvZMSIEekuS5IkSQkIIZTEGAu2188pGJKkOlW2/CkuvrgHX/taJt267cV++0fOOOOMdJclSZKkNHMKhiSpzpQtf4p33rmOiy7O5KKLOwPQrFkpy//vaXI6nJ7m6iRJkpROjoCQJNWZpUsmUln5+WZtlZWfs3TJxDRVJEmSpPrCAEKSVGfWbyjbqXZJkiQ1HQYQkqQ6k5WZs1PtkiRJajoMICRJdaZrtwk0a9Zys7ZmzVrStduENFUkSZKk+sJFKCVJdaZ6ocmlSyayfkMZWZk5dO02wQUoJUmSZAAhSapbOR1ON3CQJEnSlzgFQ5IkSZIkJc4AQpIkSZIkJc4AQpIkSZIkJc4AQpIkSZIkJc4AQpIkSZIkJc4AQpIkSZIkJc4AQpIkSZIkJc4AQpIkSZIkJc4AQpIkSXUmxkhlZWW6y5Ak1UPN012AJEmS6p+rr76agw46iO9///sAFBYW0rp1ayorK3n88cfZsGEDI0aM4IYbbqC0tJSTTz6Z448/ntmzZ3PGGWewatUqbrvtNgDuuece3n77bX7xi1+kc5ckSWnmCAhJkiR9yejRo3nsscdq7j/++OPsu+++LF68mNdff5158+ZRUlLCSy+9BMCiRYs477zzmDt3LhMmTODpp5+mvLwcgPvuu48LLrggLfshSao/HAEhSZKkL+nTpw8fffQRy5YtY8WKFbRr14758+czbdo0+vTpA8DatWtZvHgxBx54IAcddBD9+/cHIDs7myFDhvDMM89w2GGHUV5ezuGHH57O3ZEk1QMGEJIkSarVyJEjKSoqYvny5YwePZrS0lKuvfZavve9723Wr7S0lOzs7M3aLrroIn7605/So0cPRz9IkgADCEmSJG3F6NGjufjii/n444/529/+xptvvsmPfvQjxowZQ6tWrfjwww9p0aJFrY896qijeP/995kzZw7z58/fzZVLkuojAwhJkiTVqlevXqxZs4ZOnTqRk5NDTk4Ob7/9NkcffTQArVq14qGHHiIjI6PWx5911lnMmzePdu3a7c6yJUn1VIgxpruG7SooKIjFxcXpLkOSJEk74dRTT+XKK69k6NCh6S5FkpSgEEJJjLFge/28CoYkSZLqzOopUygZOIgue2Sy8fXXKfjss3SXJEmqJ5yCIUmSpDqxesoUyn70Y/Zcv54/d+0KQNmPfgxAm+HD01maJKkecASEJEmS6sRHt00irl+/WVtcv56PbpuUpookSfWJAYQkSZLqREVZ2U61S5KaFgMISZIk1YnmOTk71S5JaloMICRJklQn9rvyCkJW1mZtISuL/a68Ik0VSZLqExehlCRJUp2oXmjyo9smUVFWRvOcHPa78goXoJQkAQYQkiRJqkNthg83cJAk1copGJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSZIkKXEGEJIkSVIj1KpVq11+jvvvv5/LLrtsm31KS0t55JFHdvm1JDV+BhCSJEmSvjIDCEk7ygBCkiRJauDOOOMM+vbtS69evbj77rtr2v/zP/+T/Px8hg4dyooVKwCYPHkyPXv2JDc3l9GjRwOwcuVKzjjjDHJzc+nfvz/z58//0mucf/75FBUV1dyvHmFxzTXXMHPmTPLy8rjtttvYuHEjV111FUceeSS5ubn8+te/TnLXJTUgBhCSJElSA3fvvfdSUlJCcXExkydP5pNPPmHdunXk5+czZ84cBg8ezA033ADAzTffzNy5c5k/fz533XUXANdffz19+vRh/vz5/PSnP+W8887b4de++eabGThwIPPmzePKK6/kt7/9LW3atOGNN97gjTfe4J577uEf//hHIvstqWExgJAkSZIauMmTJ3PEEUfQv39/3n//fRYvXkyzZs04++yzATj33HOZNWsWALm5uYwZM4aHHnqI5s2bAzBr1iy+/e1vAzBkyBA++eQTVq9e/ZVqmTZtGr/73e/Iy8vjqKOO4pNPPmHx4sV1sJeSGrrm6S5AkiRJ0lc3Y8YMXnjhBWbPns2ee+7Jcccdx/r167/UL4QAwNSpU3nppZd4+umnufHGG1mwYAExxq32r9a8eXMqKysBiDHyxRdf1FpPjJFf/vKXDBs2bFd3TVIj4wgISZIkqQFbvXo17dq1Y8899+Sdd97h1VdfBaCysrJmzYZHHnmEY489lsrKSt5//32OP/54fv7zn7Nq1SrWrl3LoEGDePjhh4GqQKN9+/bstddem71Oly5dKCkpAeCpp56ivLwcgNatW7NmzZqafsOGDePOO++s2f7uu++ybt26ZP8RJDUIjoCQJEmSGrCTTjqJu+66i9zcXLp3707//v0ByM7OZsGCBfTt25c2bdrw2GOPsXHjRs4991xWr15NjJErr7yStm3bUlhYyAUXXEBubi577rknDzzwwJde5+KLL+b000+nX79+DB06lOzsbKBqSkfz5s054ogjOP/88xk/fjylpaXk5+cTY2TfffflT3/60279N5FUP4XahlvVNwUFBbG4uDjdZUiSJEnahnVzP+Jfz5WycdUGMtpmstewLmT32S/dZUlKWAihJMZYsL1+joCQJEmStMvWzf2IVX9cTCyvWidi46oNrPpj1eKThhCSwDUgJEmSJNWBfz1XWhM+VIvllfzrudL0FCSp3qmTACKEMCGEEEMI7VP3QwhhcgjhvRDC/BBC/iZ9x4YQFqd+xtbF60uSJElKr42rNuxUu6SmZ5enYIQQDgBOAP65SfPJwCGpn6OAO4GjQgh7A9cDBUAESkIIT8cYP93VOiRJkiSlT0bbzFrDhoy2mWmoRlJ9VBcjIG4D/ouqQKHa6cDvYpVXgbYhhBxgGPB8jHFlKnR4HjipDmqQJEmSlEZ7DetCaLH5rxehRTP2GtYlPQVJqnd2aQRECOE04MMY499DCJtu6gS8v8n9D1JtW2uXJEmS1IBVLzTpVTAkbc12A4gQwgtAh1o2XQf8EDixtofV0ha30V7b634X+C7AgQceuL0yJUmSJKVZdp/9DBwkbdV2A4gY49draw8hHA4cDFSPfugMzAkh9KNqZMMBm3TvDCxLtR+3RfuMrbzu3cDdAAUFBbWGFJIkSZIkqWH4ymtAxBjfjDHuF2PsEmPsQlW4kB9jXA48DZyXuhpGf2B1jLEMeA44MYTQLoTQjqrRE8/t+m5IkiRJkqT6bJevgrEVzwLfAN4DPgMuAIgxrgwh3Ai8ker3/2KMKxOqQZIkSZIk1RN1FkCkRkFU347Af2yl373AvXX1upIkSZIkqf6ri8twSpIkSZIkbZMBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCRJkiRJSpwBhCTVUzFGKisr012GJEmSVCcMICSpDv3iF7+gd+/e9O7dm0mTJnH11Vdzxx131GwvLCzk1ltvBeCWW27hyCOPJDc3l+uvvx6A0tJSDjvsML7//e+Tn5/P+++/n5b9kCRJkuqaAYQk1ZGSkhLuu+8+XnvtNV599VXuueceRo8ezWOPPVbT5/HHH2fUqFFMmzaNxYsX8/rrrzNv3jxKSkp46aWXAFi0aBHnnXcec+fO5aCDDkrX7kiSJEl1qnm6C5CkxmLWrFmMGDGC7OxsAM4880xmzpzJRx99xLJly1ixYgXt2rXjwAMPZPLkyUybNo0+ffoAsHbtWhYvXsyBBx7IQQcdRP/+/dO5K5IkSVKdM4CQpDoSY6y1feTIkRQVFbF8+XJGjx5d0/faa6/le9/73mZ9S0tLawIMSZIkqTFxCoYk1ZFBgwbxpz/9ic8++4x169bx5JNPMnDgQEaPHs2jjz5KUVERI0eOBGDYsGHce++9rF27FoAPP/yQjz76KJ3lS5IkSYlyBIQk1ZH8/HzOP/98+vXrB8BFF11UM8VizZo1dOrUiZycHABOPPFE3n77bY4++mgAWrVqxUMPPURGAScC2wAACLhJREFURkZ6ipckSZISFrY2ZLg+KSgoiMXFxekuQ5IkSZIkbSGEUBJjLNheP6dgSFI9MHXpVE4sOpHcB3I5sehEpi6dmu6SJEmSpDrlFAxJSrOpS6dS+Eoh6zeuB6BsXRmFrxQCcErXU9JYmSRJklR3HAEhSWl2+5zba8KHaus3ruf2ObenqSJJkiSp7hlASFKaLV+3fKfaJUmSpIbIAEKS0qxDdoedapckSZIaIgMISUqz8fnjycrI2qwtKyOL8fnj01SRJEmSVPdchFKS0qx6ocnb59zO8nXL6ZDdgfH5412AUpIkSY2KAYQk1QOndD3FwEGSJEmNmlMwJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4gwgJEmSJElS4kKMMd01bFcIYQXwv+muoxbtgY/TXYTqHY8LbcljQlvymFBtPC60JY8J1cbjQluqD8fEQTHGfbfXqUEEEPVVCKE4xliQ7jpUv3hcaEseE9qSx4Rq43GhLXlMqDYeF9pSQzomnIIhSZIkSZISZwAhSZIkSZISZwCxa+5OdwGqlzwutCWPCW3JY0K18bjQljwmVBuPC22pwRwTrgEhSZIkSZIS5wgISZIkSZKUOAOInRBCGBdCWBRCWBBC+Pkm7deGEN5LbRu2SftJqbb3QgjXpKdqJS2EMCGEEEMI7VP3Qwhhcup9nx9CyN+k79gQwuLUz9j0Va0khBBuCSG8k3rfnwwhtN1km+cJAb7nTVUI4YAQwvQQwtup7xHjU+17hxCeT30uPB9CaJdq3+pniRqXEEJGCGFuCOGZ1P2DQwivpY6Jx0IIe6TaM1P330tt75LOupWcEELbEEJR6jvF2yGEoz1XKIRwZerz460Qwu9DCFkN8XxhALGDQgjHA6cDuTHGXsDEVHtPYDTQCzgJuCP1QZIB/A9wMtATOCfVV41ICOEA4ATgn5s0nwwckvr5LnBnqu/ewPXAUUA/4PrqDw81Gs8DvWOMucC7wLXgeUL/5nvepFUA/xljPAzoD/xH6r2/BngxxngI8GLqPmzls0SN0njg7U3u/zdwW+qY+BT4Tqr9O8CnMcavAbel+qlxuh34S4yxB3AEVceH54omLITQCbgcKIgx9gYyqPpu2eDOFwYQO+5S4OYY4waAGONHqfbTgUdjjBtijP8A3qPql8t+wHsxxqUxxi+AR1N91bjcBvwXsOliKqcDv4tVXgXahhBygGHA8zHGlTHGT6n6ZfWk3V6xEhNjnBZjrEjdfRXonLrteULVfM+bqBhjWYxxTur2Gqp+oehE1fv/QKrbA8AZqdtb+yxRIxJC6AycAvwmdT8AQ4CiVJctj4nqY6UIGJrqr0YkhLAXMAj4LUCM8YsY4yo8VwiaAy1DCM2BPYEyGuD5wgBixx0KDEwNYflbCOHIVHsn4P1N+n2QattauxqJEMJpwIcxxr9vscljQgAXAn9O3faYUDXfc5EaCtsHeA3YP8ZYBlUhBbBfqpvHStMwiao/ZFSm7u8DrNokzN70fa85JlLbV6f6q3HpCqwA7ktNzflNCCEbzxVNWozxQ6pG4P+TquBhNVBCAzxfNE93AfVJCOEFoEMtm66j6t+qHVXDJo8EHg8hdAVqS5IitYc7XnKkgdnOMfFD4MTaHlZLW9xGuxqQbR0TMcanUn2uo2q49cPVD6ulv+eJpsnzQBMXQmgFPAFcEWP81zb+IOWx0siFEE4FPooxloQQjqturqVr3IFtajyaA/nAuBjjayGE2/n3dIvaeFw0Aalp26cDBwOrgD9QNf1mS/X+fGEAsYkY49e3ti2EcCnwx1h13dLXQwiVQHuqkqYDNunaGViWur21djUQWzsmQgiHU3UC+Hvqy2NnYE4IoR9bPyY+AI7bon1GnRetRG3rPAFVC40CpwJD47+vc+x5QtW2dSyokQshtKAqfHg4xvjHVPP/hRByYoxlqWHT1VM8PVYavwHAaSGEbwBZwF5UjYhoG0Jonvqr5abve/Ux8UFqCHYbYOXuL1sJ+wD4IMb4Wup+EVUBhOeKpu3rwD9ijCsAQgh/BI6hAZ4vnIKx4/5E1RwbQgiHAnsAHwNPA6NTK40eTNUCMK8DbwCHpFYm3YOqRUKeTkvlqnMxxjdjjPvFGLvEGLtQ9T95foxxOVXv83mpVYn7A6tTQ+WeA04MIbRLpZgnptrUSIQQTgKuBk6LMX62ySbPE6rme95Epebe/hZ4O8b4i002PQ1UXxVpLPDUJu21fZaokYgxXhtj7Jz6HjEa+GuMcQwwHRiZ6rblMVF9rIxM9a8Xf9FU3Ul9l3w/hNA91TQUWIjniqbun0D/EMKeqc+T6uOiwZ0vHAGx4+4F7g0hvAV8AYxNvYkLQgiPU3UAVAD/EWPcCBBCuIyqXzAzgHtjjAvSU7p2s2eBb1C10OBnwAUAMcaVIYQbqfoFBOD/xRjrRRKpOvMrIBN4PjUy5tUY4yUxRs8TAqrmYfqeN1kDgG8Db4YQ5qXafgjcTNW0zu9Q9QVzVGpbrZ8lahKuBh4NIfwEmEtqMcLUfx8MIbxH1V8yR6epPiVvHPBwKqheStX//83wXNFkpabjFAFzqPouORe4G5hKAztfhHoShEiSJEmSpEbMKRiSJEmSJClxBhCSJEmSJClxBhCSJEmSJClxBhCSJEmSJClxBhCSJEmSJClxBhCSJEmSJClxBhCSJEmSJClxBhCSJEmSJClx/x+RWdEsI6vJnAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a2347fa20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tsne_plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that \"play\" has been used in different context and Elmo is able to distinguish."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. **Tensorboard**\n",
    "\n",
    "Visualizing using Tensorboard involves the following:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Preparing the Batch** (embedding matrix) <br>\n",
    "    \n",
    "   The batch is an embedding matrix of shape **(_tokens_ , embedding dimension)** where each      entry is the embedding of the token at that position."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_batch():\n",
    "    \"Creates matrix for embeddings\"\n",
    "    \n",
    "    sent =  [\"Argentina\", \"played\", \"football\", \"very\", \"well\"]\n",
    "    sent1 = [\"Brazil\",\"is\",\"a\",\"strong\",\"team\"]\n",
    "    sent2 = [\"Artists\",\"all\",\"over\",\"the\",\"world\",\"are\",\"attending\",\"the\",\"play\"]\n",
    "    sent3 = [\"Child\",\"is\",\"playing\",\"the\",\"guitar\"]\n",
    "    sent4 = [\"There\",\"was\",\"absolute\",\"silence\",\"during\",\"the\",\"play\"]\n",
    "    \n",
    "    counter=0\n",
    "    batch_xs = np.zeros((31,1024)) # of shape(tokens,embedding_dimension)\n",
    "    \n",
    "    for i in range(len(sent)):\n",
    "        batch_xs[counter] = sess.run(embeddings[0][i])\n",
    "        counter = counter+1    \n",
    "    counter = len(sent) \n",
    "    for i in range(len(sent1)):\n",
    "        batch_xs[counter] = sess.run(embeddings[0][i])\n",
    "        counter = counter+1\n",
    "    counter = len(sent)+len(sent1)\n",
    "    for i in range(len(sent2)):\n",
    "        batch_xs[counter] = sess.run(embeddings[0][i])\n",
    "        counter = counter+1\n",
    "    counter = len(sent)+len(sent1)+len(sent2)\n",
    "    for i in range(len(sent3)):\n",
    "        batch_xs[counter] = sess.run(embeddings[0][i])\n",
    "        counter = counter+1\n",
    "    counter = len(sent)  \n",
    "    for i in range(len(sent4)):\n",
    "        batch_xs[counter] = sess.run(embeddings[0][i])\n",
    "        counter = counter+1\n",
    "    return batch_xs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Specifying the log_directory** <br>\n",
    "    The log_directory is the location where you save the metadata.tsv and where all the       checkpoints will get created.\n",
    "    \n",
    "    \n",
    "- **Creating the metadata.tsv** <br>\n",
    "    This is a file which contains the index and token separated by a tab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "token_list = [\"Argentina\", \"played\", \"football\", \"very\", \"well\",\"Brazil\",\"is\",\"a\",\"strong\",\n",
    "              \"team\",\"Artists\",\"all\",\"over\",\"the\",\"world\",\"are\",\"attending\",\"the\",\"play\",\n",
    "              \"Child\",\"is\",\"playing\",\"the\",\"guitar\",\"There\",\"was\",\"absolute\",\"silence\",\n",
    "              \"during\",\"the\",\"play\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('/Users/a554755/Elmo/log_dir/Elmo_metadata.tsv','w') as f:\n",
    "    f.write(\"Index\\tLabel\\n\")\n",
    "    for index,label in enumerate(token_list):\n",
    "        f.write(\"%d\\t%s\\n\" % (index,label))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "LOG_DIR = 'log_dir'\n",
    "NAME_TO_VISUALISE_VARIABLE = \"elmoembedding\"\n",
    "TO_EMBED_COUNT = 25\n",
    "\n",
    "\n",
    "#path_for_mnist_sprites =  os.path.join(LOG_DIR,'mnistdigits.png')\n",
    "path_for_elmo_metadata =  os.path.join(LOG_DIR,'Elmo_metadata.tsv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_var = tf.Variable(batch_x, name=NAME_TO_VISUALISE_VARIABLE)\n",
    "summary_writer = tf.summary.FileWriter(LOG_DIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = projector.ProjectorConfig()\n",
    "embedding = config.embeddings.add()\n",
    "embedding.tensor_name = embedding_var.name\n",
    "\n",
    "# Specify where you find the metadata\n",
    "embedding.metadata_path = 'Elmo_metadata.tsv' #'metadata.tsv'\n",
    "\n",
    "# Say that you want to visualise the embeddings\n",
    "projector.visualize_embeddings(summary_writer, config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'log_dir/model.ckpt-1'"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sess = tf.InteractiveSession()\n",
    "sess.run(tf.global_variables_initializer())\n",
    "saver = tf.train.Saver()\n",
    "saver.save(sess, os.path.join(LOG_DIR, \"model.ckpt\"), 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running tensorboard :\n",
    "\n",
    "    tensorboard --logdir=\"your_log_directory\"\n",
    "    \n",
    "Tensorboard will be running at: http://localhost:6006 \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# Training Elmo Model on new data:\n",
    "\n",
    "\n",
    "\n",
    "To train and evaluate a biLM, you need to provide:\n",
    "\n",
    "    a vocabulary file \n",
    "    a set of training files \n",
    "    a set of heldout files \n",
    "\n",
    "The vocabulary file is a a text file with one token per line. It must also include the special tokens <S>, </S> and <UNK> (case sensitive) in the file.\n",
    "\n",
    "The vocabulary file should be sorted in descending order by token count in your training data. The first three lines should be the special tokens (<S>, </S> and <UNK>), then the most common token in the training data, ending with the least common token.\n",
    "\n",
    "The training data should be randomly split into many training files, each containing one slice of the data. Each file contains pre-tokenized and white space separated text, one sentence per line. Don't include the <S> or </S> tokens in your training data.\n",
    "\n",
    "\n",
    "Once done, git clone https://github.com/allenai/bilm-tf.git \n",
    "and run:\n",
    "\n",
    "    python bin/train_elmo.py --train_prefix= <path to training folder> --vocab_file <path to vocab file> --save_dir <path where models will be checkpointed>\n",
    "    \n",
    "To get the weights file, run:\n",
    "    \n",
    "    python bin/dump_weights.py --save_dir /output_path/to/checkpoint --outfile/output_path/to/weights.hdf5\n",
    "    \n",
    "    \n",
    "In the save dir, one options.json will be dumped and above command will give you a weights file required to create an Elmo model (options file and json file)\n",
    "\n",
    "### To use Elmo programatically :\n",
    "\n",
    "from allennlp.modules.elmo import Elmo, batch_to_ids\n",
    "\n",
    "options_file = \"path to options file\" <br>\n",
    "weight_file = \"path to weights file\"\n",
    "\n",
    "elmo = Elmo(options_file, weight_file, 2, dropout=0)\n",
    "\n",
    "2 is an integer which represents ``num_output_representations``.\n",
    "Typically ``num_output_representations`` is 1 or 2.  For example, in the case of the SRL model in the above paper, ``num_output_representations=1`` where ELMo was included at\n",
    "the input token representation layer.  In the case of the SQuAD model,``num_output_representations=2``as ELMo was also included at the GRU output layer.\n",
    "\n",
    "\n",
    "use batch_to_ids to convert sentences to character ids:\n",
    "\n",
    "    sentences = [['First', 'sentence', '.'], ['Another', '.']]\n",
    "    character_ids = batch_to_ids(sentences)\n",
    "\n",
    "    embeddings = elmo(character_ids)\n",
    "\n",
    "    embeddings[elmo] is length two list of tensors. \n",
    "    Each element contains one layer of ELMo representations with shape\n",
    "    (2, 3, 1024).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Incremental Learning\n",
    "\n",
    "In order to do the incremental learning, one would want to update an existing weights file based on new vocab file. The previously checkpointed file will have a different structure given the different vocab size. \n",
    "\n",
    "See: https://github.com/tensorflow/nmt/issues/134 \n",
    "\n",
    "### The problem:\n",
    "We have trained a model and created the checkpoint and have a saved weights file. We have new data with new vocab file and we create the same model structure. Now when we try to load the weights from the checkpointed file you may encounter an error:\n",
    "\n",
    "**tensor shape mis-match** \n",
    "\n",
    "The most likely reason being the vocab size mis match and hence loading the weights of the embedding layer from checkpointed file(x,y) will not align with shape of the new model structure (embedding layer (m,y)) and hence will be a problem.\n",
    "\n",
    "### Solution\n",
    "Load all the layers weights except for the embedding layer and learn the weights of the embedding during the new training process.\n",
    "\n",
    "train with the same command as above with a new parameter:\n",
    "\n",
    "    python bin/train_elmo_updated.py --train_prefix= <path to training folder> --vocab_file <path to vocab file> --save_dir <path where models will be checkpointed> --restart_ckpt_file <path to checkpointed model>\n",
    "     \n",
    "replace the _training.py_ within _bilm_ with the updated _training_updated.py attached at _home_\n",
    "\n",
    "In the _train_elmo_updated.py_ within _bin_, set these options based on your data:\n",
    "\n",
    "    batch_size = 128  # batch size for each GPU\n",
    "    n_gpus = 3\n",
    "\n",
    "    # number of tokens in training data \n",
    "    n_train_tokens = \n",
    "\n",
    "    options = {\n",
    "     'bidirectional': True,\n",
    "     'dropout': 0.1,\n",
    "     'all_clip_norm_val': 10.0,\n",
    "    \n",
    "     'n_epochs': 10,\n",
    "     'n_train_tokens': n_train_tokens,\n",
    "     'batch_size': batch_size,\n",
    "     'n_tokens_vocab': vocab.size,\n",
    "     'unroll_steps': 20,\n",
    "     'n_negative_samples_batch': 8192,\n",
    "\n",
    "NOTE: Try running the training on a gpu as recommended, else the training will be slow.\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Errors\n",
    "\n",
    "Errors you may run into:\n",
    "\n",
    "  **Cudnn_internal_status_error**<br>\n",
    "  **failed to enqueue convolution/max pooling on stream** <br>\n",
    "     \n",
    "#### Solution\n",
    "     try reducing the no. of paramters -  lesser batch size etc.\n",
    "     \n",
    "Kindly let me know more in the issues section if theres any.\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Elmo Embedding layer in consequent models\n",
    "\n",
    "if you want to use Elmo Embedding layer in consequent model build refer :\n",
    "https://github.com/PrashantRanjan09/WordEmbeddings-Elmo-Fasttext-Word2Vec "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}