analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Indentification of Semantic Duplicates\n",
    "## Dilettante Dive into the Quora Data Set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finding and quantifying semantic patterns is key to successful natural\n",
    "language processing and knowledge organisation. One of the most\n",
    "important challenges in NLP is determining whether two texts have the\n",
    "same intent or not, and machine learning techniques provide us a\n",
    "necessary and diverse toolset to approach this problem.\n",
    "In January 2017, [Quora published its first public data set](https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs) on\n",
    "duplicate and related human-labelled questions compiled from the\n",
    "actual Quora data. How can we use it to predict whether two\n",
    "question-strings are duplicate or not?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Modules and Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O\n",
    "import matplotlib.pyplot as plt # visualisation\n",
    "\n",
    "training_filename = \"data/train.csv\"\n",
    "testing_filename = \"data/test.csv\"\n",
    "training_data = pd.read_csv(training_filename, encoding = 'utf8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get a sneak peek at the training data at hand:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>qid1</th>\n",
       "      <th>qid2</th>\n",
       "      <th>question1</th>\n",
       "      <th>question2</th>\n",
       "      <th>is_duplicate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>What is the step by step guide to invest in sh...</td>\n",
       "      <td>What is the step by step guide to invest in sh...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>What is the story of Kohinoor (Koh-i-Noor) Dia...</td>\n",
       "      <td>What would happen if the Indian government sto...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>How can I increase the speed of my internet co...</td>\n",
       "      <td>How can Internet speed be increased by hacking...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>Why am I mentally very lonely? How can I solve...</td>\n",
       "      <td>Find the remainder when [math]23^{24}[/math] i...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>Which one dissolve in water quikly sugar, salt...</td>\n",
       "      <td>Which fish would survive in salt water?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>Astrology: I am a Capricorn Sun Cap moon and c...</td>\n",
       "      <td>I'm a triple Capricorn (Sun, Moon and ascendan...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>6</td>\n",
       "      <td>13</td>\n",
       "      <td>14</td>\n",
       "      <td>Should I buy tiago?</td>\n",
       "      <td>What keeps childern active and far from phone ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7</td>\n",
       "      <td>15</td>\n",
       "      <td>16</td>\n",
       "      <td>How can I be a good geologist?</td>\n",
       "      <td>What should I do to be a great geologist?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>17</td>\n",
       "      <td>18</td>\n",
       "      <td>When do you use シ instead of し?</td>\n",
       "      <td>When do you use \"&amp;\" instead of \"and\"?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>9</td>\n",
       "      <td>19</td>\n",
       "      <td>20</td>\n",
       "      <td>Motorola (company): Can I hack my Charter Moto...</td>\n",
       "      <td>How do I hack Motorola DCX3400 for free internet?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  qid1  qid2                                          question1  \\\n",
       "0   0     1     2  What is the step by step guide to invest in sh...   \n",
       "1   1     3     4  What is the story of Kohinoor (Koh-i-Noor) Dia...   \n",
       "2   2     5     6  How can I increase the speed of my internet co...   \n",
       "3   3     7     8  Why am I mentally very lonely? How can I solve...   \n",
       "4   4     9    10  Which one dissolve in water quikly sugar, salt...   \n",
       "5   5    11    12  Astrology: I am a Capricorn Sun Cap moon and c...   \n",
       "6   6    13    14                                Should I buy tiago?   \n",
       "7   7    15    16                     How can I be a good geologist?   \n",
       "8   8    17    18                    When do you use シ instead of し?   \n",
       "9   9    19    20  Motorola (company): Can I hack my Charter Moto...   \n",
       "\n",
       "                                           question2  is_duplicate  \n",
       "0  What is the step by step guide to invest in sh...             0  \n",
       "1  What would happen if the Indian government sto...             0  \n",
       "2  How can Internet speed be increased by hacking...             0  \n",
       "3  Find the remainder when [math]23^{24}[/math] i...             0  \n",
       "4            Which fish would survive in salt water?             0  \n",
       "5  I'm a triple Capricorn (Sun, Moon and ascendan...             1  \n",
       "6  What keeps childern active and far from phone ...             0  \n",
       "7          What should I do to be a great geologist?             1  \n",
       "8              When do you use \"&\" instead of \"and\"?             0  \n",
       "9  How do I hack Motorola DCX3400 for free internet?             0  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_data.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are given the following parameters:\n",
    "\n",
    "-   **id** - the id of a training set question pair\n",
    "-   **qid1**, **qid2** - unique ids of each question\n",
    "-   **question1**, **question2** - the full text of each question\n",
    "-   **is\\_duplicate** - the target variable, set to 1 if question1 and\n",
    "    question2 have essentially the same meaning, and 0 otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 404290 question pairs available for training.\n",
      "Among them, 36.92% are duplicates.\n",
      "There are 537933 unique questions considered.\n",
      "The number of questions that appear multiple times is 111780.\n"
     ]
    }
   ],
   "source": [
    "print('There are {} question pairs available for training.'.format(len(training_data)))\n",
    "print('Among them, {}% are duplicates.'.format(round(training_data['is_duplicate'].mean()*100, 2)))\n",
    "# find unique question ids\n",
    "qids = pd.Series(training_data['qid1'].tolist() + training_data['qid2'].tolist())\n",
    "print('There are {} unique questions considered.'.format(len(np.unique(qids))))\n",
    "print('The number of questions that appear multiple times is {}.'.format(np.sum(qids.value_counts() > 1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the distribution of question lengths:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAt0AAAFQCAYAAABj8cJpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X2cXGV5+P/Plc2CG0Q2QGpNeAiiDYooaATUfutjQVEg\nRVFRW7U+1Odaf02FlhZ86Bdsvm3VWvXrI1ot8mgE0QYroJWvQQMhRJQoIg9ZqMZCQGGBZXP9/jhn\nwuxkZnZ2M7Mzs/t5v1557cx97nPONTOH5Zp7r3PfkZlIkiRJ6px53Q5AkiRJmu1MuiVJkqQOM+mW\nJEmSOsykW5IkSeowk25JkiSpw0y6JUmSpA4z6ZbU8yLitxHx2G7H0U0R8daI+GX5XuzV7XiaiYhP\nRsTfdjuOTomI0yPiS92OQ1J/MemWVFdE3BwRL5iB82REPK6mbUJSk5mPzMybJjnOcyJic6fi7KaI\nGAT+CTiqfC/+p06fXSPijIi4NSJGI+JnEfGXEREdju11EfG96rbMfEtmfqAD55rxZLcd11UU3hUR\nP4qIeyNic0ScFxGHtCvOBuddWv73Nb+T55HUGv9DlKQWRMRAZo536fSPBh4BXN+kz3nA7wLHADcA\ny4F/AxYD7+l0gGrqI8CLgTcBVwIDwB+VbRu7GJekGeRIt6Qpi4g3RcSNEXFnRFwUEYurth0VEZsi\n4u6I+HhEfCci3riT59s+Gh4Rx0TEjyPiNxExUo7m7gZ8E1hcll/8NiIWl6O/H46I28t/H46IXauO\n+1cRcUe57Y015zkrIj4REd+IiHuB50bEiyNifUTcExG3RcTpVceqjCq+vtx2V0S8JSKeHhHXRcTW\niPhYk9dYN9aI+D1gU9lta0RcVmff5wNHAS/NzB9l5kOZuRZ4DfDnldKc2r9e1I4cR8SREfH/ylg3\nRMRzqra9LiJuKt/3X0TEqyPiCcAngWeU7/nWqvfug1X7Nrtesnyffla+Z/86ndH58vO+ICK2lPG9\nq+Z1nhsRXyzjvz4illdtf2r5uf6mHIE+JyI+2Oi6KnfbpdHxauJ6PPB24KTMvCwzH8jM+zLzy5l5\nZtlnj/JYWyLilog4NSLmNfiMJoxeR8QVEfGBiLiyjOXSiNi77P7d8ufWMvZnRMTjovhv8u6I+HVE\nnDPV91rS9Jh0S5qSiHgecAbwcuAxwC3AV8ptewPnA6cAe1Eki89scwifBf4sM3cHngRclpn3Ai8C\nbi/LLx6ZmbcDfwMcCRwKPAU4HDi1jPWFFCPALwAeBzy7zrleBfw9sDvwPeBe4E+AYYpRyrdGxIqa\nfY4AHg+8AvhwGcMLgIOBl0dEvfPQKNbM/Gm5L8BwZj6vzr5/CFyVmbdVN2bmVcBm4PkNzrldRCwB\nLgE+COwJ/CVwQUQsKpPPjwIvKt/3ZwLXZuZPgLcA3y/f8+E6x214vVR5CfD08nW/HDh6snhrzjEP\nuBjYACwpX++7I6L6OMeV5x0GLgI+Vu67C/BV4KzydZ9NMQpNk+uq4fHqeD6wOTN/0OQl/AuwB/BY\niuvwT4DXt/jyobhOXw/8DrALxWcH8Aflz+Ey9u8DHwAuBRYC+5TnljQDTLolTdWrgc9l5jWZ+QBF\ngv2MiFhKUdpwfWZemJkPUSRq/93CMa8pR1e3lqOlJzfpOwY8MSIelZl3ZeY1k8T6/sz8VWZuAd4H\n/HG57eXA5zPz+sy8r9xW62uZeWVmbsvM+zPziszcWD6/jiJBq02iP1D2vZQiST+7PP8I8F/AYdOI\ndTJ7A3c02HYHsKiFY7wG+EZmfqN8fd8C1lF8pgDbgCdFxFBm3pGZzUpdqjW7XirOzMytmXkrcDnF\nF4+peDqwKDPfn5kPlvX/nwZeWdXne+VrG6cou3lK2X4kRanlRzNzLDMvBJolyJMdr9ZeNP5siIgB\nii9op2TmbzLzZuAfaf2zh+I6/mlmjgLn0vz9GwP2BxaX1+n3mvSV1EYm3ZKmajHFaCUAmflb4H8o\nRhgXA7dVbUuKkVYAyj/DV/5M/7+qjvnUzByu/APObHL+l1IkgreUfyZ/Rquxlo8XV22rHhmeMEpc\nry0ijoiIy8sygLspRnn3rtnnl1WPR+s8f+Q0Yp3MrylGket5DLClhWPsD5xY8+Xn94HHlCO+r6B4\nvXdExCURcVCLsTW7Xiqqv5jdR+P3qFnsi2ti/2uKWvhG53hEWaKxGBgpr9WKetdCrUbHq/U/NP5s\noLh+dmHHz35J/e4txdLs/fsrIIAflP89/ukUziNpJ5h0S5qq2ymSHADK0oO9gBGKEb19qrZF9fPM\nPLjqz/T/NZ2TZ+YPM/N4ij+lr6YY2QPIOt0nxArsV7ZRGyuwb73T1Tz/d4pSgn0zcw+KeuZ2zQ7S\nLNbJ/CdwRERMeA0RcXh5nEpt773Agqouv1v1+Dbg36q//GTmbpW648xck5l/SJFA3kAxkgz13/eG\nr6vmemmX24Bf1MS+e2YeM+mexXWwpKaOvPp9nOz1TebbwD6Nar4pvjBVRp8r9uPh96fZZzaZHWLP\nzP/OzDdl5mLgz4CPR83sQZI6w6RbUjODEfGIqn/zKRLP10fEoVHclPi/KeqJb6aoCT4kIlaUfd/O\n1JKEpiJilyhu4NsjM8eAe4DKjCK/BPaKiD2qdjkbOLWsS94b+DugclPaueXreEJELCi3TWZ34M7M\nvL9MaF/VjtfVQqxNZeZ/UiR3F0TEwRExEBFHAl8GvpiZlRsxrwVeGRGDZRL4sqrDfAk4NiKOLvd/\nRBTT5e0TEY+OiOPKhPkB4LdMfN/3KWuj62l2vUzHvJprcleKcpB7IuK9ETFUxv+kiHh6C8f7fvla\n3hER8yPieIp6+op611XLMvNnwMeBs8v3c5cy7ldGxMllecq5wN9HxO4RsT/FvQaVz/5a4A8iYr8y\nhlOmcPotFGVB2+e4j4gTI6LyZfMuisS8W7PySHOKSbekZr5BURJR+Xd6Zn4b+FvgAopRwgMpa2cz\n89fAicA/UPxZ/YkUdcEPtDGmPwZujoh7KModXlOe+waKxPWmssRgMcVNgeuA6yimZrumbCMzv0lR\nc345cCNF8sUksb4NeH9E/IYiKT63Sd+pahhri15K8Vr+A7if4vX8B/Dmqj5/S/F53UVRM/7vlQ3l\nTZjHU5RlbKEYPV5J8f+JecD/RzFqfSdFHfvbyl0vo5jK8L8j4te1QTW7XqbpJCZekz8vE9djKWqZ\nf0ExevwZipsTm8rMB4ETgDcAWymup69TXgcNrqupehfFjZb/Wp7j5xQ3a15cbn8nxYj2TRQ37P47\n8Lny/N8CzqG4Lq4uY2tJea/C3wNXlrEfSVH/flVE/JbirzZ/npm/mMZrkjRFMbGMTZLap5xVYjPw\n6sy8vNvxNBPF9Hc/AnYtbwLtaxHxBYq64GPKxFItioirgE9m5ue7HYuk2cORbkltVZYnDJd/9v9r\niprntV0Oq66I+KPyz/0LgQ8BF8+GhLv0RuBbwFO7HUivi4hnR8TvluUlrwWeTPFXAklqG5NuSe32\nDIo/n/+a4k/+K8qpzHrRn1GUUvycoq71rd0Np33K6e8+lMUiOWpuGcUc33dTlNG8LDMbTvMnSdNh\neYkkSZLUYY50S5IkSR1WbyL/vrf33nvn0qVLux2GJEmSZrmrr77615k56cq/szLpXrp0KevWret2\nGJIkSZrlIuKWyXtZXiJJkiR1XMeS7oj4XET8KiJ+VNW2KiJuiIjrIuKrETFcte2UiLgxIjZFxNFV\n7S8s226MiJM7Fa8kSZLUKZ0c6T4LeGFN27eAJ2Xmk4GfUi5nGxFPpFih7OByn4+Xy/gOUKzg9SKK\nle1OKvtKkiRJfaNjSXdmfpdiueDqtkurFp5YC+xTPj4e+EpmPlAuR3sjcHj578bMvKlcUe0rZV9J\nkiSpb3SzpvtPgW+Wj5cAt1Vt21y2NWrfQUS8OSLWRcS6LVu2dCBcSZIkaXq6knRHxN8ADwFfrjTV\n6ZZN2ndszPxUZi7PzOWLFk06a4skSZI0Y2Z8ysCIeC3wEuD5+fBymJuBfau67QPcXj5u1C5JkiT1\nhRkd6Y6IFwLvBY7LzPuqNl0EvDIido2IA4DHAz8Afgg8PiIOiIhdKG62vGgmY5YkSZJ2VsdGuiPi\nbOA5wN4RsRk4jWK2kl2Bb0UEwNrMfEtmXh8R5wI/pig7eXtmjpfHeQewBhgAPpeZ13cq5p2xev0I\nq9Zs4vatoyweHmLl0ctYcVjd8nNJkiTNMfFwhcfssXz58pzJFSlXrx/hlAs3Mjo2vr1taHCAM044\nxMRbkiRpFouIqzNz+WT9XJGyDVat2TQh4QYYHRtn1ZpNXYpIkiRJvcSkuw1u3zo6pXZJkiTNLSbd\nbTC8YHBK7ZIkSZpbTLrboFFZ/Cwsl5ckSdI0mHS3wd2jY1NqlyRJ0txi0t0Gi4eHptQuSZKkucWk\nuw1WHr2MocGBCW1DgwOsPHpZlyKSJElSL5nxZeBno8pc3C6OI0mSpHpMuttkxWFLTLIlSZJUl+Ul\nkiRJUoeZdEuSJEkdZtItSZIkdZhJtyRJktRhJt2SJElSh5l0S5IkSR1m0i1JkiR1mEm3JEmS1GEm\n3ZIkSVKHmXRLkiRJHWbSLUmSJHWYSbckSZLUYSbdkiRJUoeZdEuSJEkdZtItSZIkdZhJtyRJktRh\nJt2SJElSh5l0S5IkSR1m0i1JkiR1mEm3JEmS1GEm3ZIkSVKHmXRLkiRJHWbSLUmSJHWYSbckSZLU\nYR1LuiPicxHxq4j4UVXbnhHxrYj4WflzYdkeEfHRiLgxIq6LiKdW7fPasv/PIuK1nYpXkiRJ6pRO\njnSfBbywpu1k4NuZ+Xjg2+VzgBcBjy//vRn4BBRJOnAacARwOHBaJVGXJEmS+kXHku7M/C5wZ03z\n8cAXysdfAFZUtX8xC2uB4Yh4DHA08K3MvDMz7wK+xY6JvCRJktTTZrqm+9GZeQdA+fN3yvYlwG1V\n/TaXbY3adxARb46IdRGxbsuWLW0PXJIkSZquXrmRMuq0ZZP2HRszP5WZyzNz+aJFi9oanCRJkrQz\nZjrp/mVZNkL581dl+2Zg36p++wC3N2mXJEmS+sZMJ90XAZUZSF4LfK2q/U/KWUyOBO4uy0/WAEdF\nxMLyBsqjyjZJkiSpb8zv1IEj4mzgOcDeEbGZYhaSM4FzI+INwK3AiWX3bwDHADcC9wGvB8jMOyPi\nA8APy37vz8zamzMlSZKknhaZdUuk+9ry5ctz3bp13Q5DkiRJs1xEXJ2Zyyfr1ys3UkqSJEmzlkm3\nJEmS1GEm3ZIkSVKHmXRLkiRJHWbSLUmSJHWYSbckSZLUYSbdkiRJUoeZdEuSJEkdZtItSZIkdZhJ\ntyRJktRhJt2SJElSh5l0S5IkSR1m0i1JkiR1mEm3JEmS1GEm3ZIkSVKHmXRLkiRJHWbSLUmSJHWY\nSbckSZLUYSbdkiRJUoeZdEuSJEkdZtItSZIkdZhJtyRJktRhJt2SJElSh5l0S5IkSR1m0i1JkiR1\nmEm3JEmS1GEm3ZIkSVKHmXRLkiRJHWbSLUmSJHXY/G4HMFusXj/CqjWbuH3rKIuHh1h59DJWHLak\n22FJkiSpB5h0t8Hq9SOccuFGRsfGARjZOsopF24EMPGWJEmS5SXtsGrNpu0Jd8Xo2Dir1mzqUkSS\nJEnqJSbdbXD71tEptUuSJGluMelug8XDQ1NqlyRJ0tzSlaQ7Iv4iIq6PiB9FxNkR8YiIOCAiroqI\nn0XEORGxS9l31/L5jeX2pd2IuZmVRy9jaHBgQtvQ4AArj17WpYgkSZLUS2Y86Y6IJcC7gOWZ+SRg\nAHgl8CHgnzPz8cBdwBvKXd4A3JWZjwP+uezXU1YctoQzTjiEJcNDBLBkeIgzTjjEmyglSZIEdG/2\nkvnAUESMAQuAO4DnAa8qt38BOB34BHB8+RjgfOBjERGZmTMZ8GRWHLbEJFuSJEl1zfhId2aOAP8H\nuJUi2b4buBrYmpkPld02A5UMdglwW7nvQ2X/vWqPGxFvjoh1EbFuy5YtnX0RkiRJ0hR0o7xkIcXo\n9QHAYmA34EV1ulZGsqPJtocbMj+Vmcszc/miRYvaFa4kSZK007pxI+ULgF9k5pbMHAMuBJ4JDEdE\npdxlH+D28vFmYF+AcvsewJ0zG7IkSZI0fd1Ium8FjoyIBRERwPOBHwOXAy8r+7wW+Fr5+KLyOeX2\ny3qtnluSJElqphs13VdR3BB5DbCxjOFTwHuB90TEjRQ1258td/kssFfZ/h7g5JmOWZIkSdoZMRsH\njZcvX57r1q3rdhiSJEma5SLi6sxcPlm/SacMjIhdgZcCS6v7Z+b7dyZASZIkaa5oZZ7ur/HwtH4P\ndDYcSZIkafZpJeneJzNf2PFIJEmSpFmqlRsp/19EHNLxSCRJkqRZquFId0RspFiEZj7w+oi4iaK8\nJIDMzCfPTIiSJElSf2tWXvKSGYtCkiRJmsUaJt2ZeQtARPxbZv5x9baI+Dfgj+vuOEedunojZ191\nG+OZDERw0hH78sEVVuVIkiSptRspD65+EhEDwNM6E05/OnX1Rr609tbtz8cztz838ZYkSVLDGykj\n4pSI+A3w5Ii4p/z3G+BXPLxEu4Czr7ptSu2SJEmaWxom3Zl5RmbuDqzKzEeV/3bPzL0y85QZjLHn\njTdY1bNRuyRJkuaWVspLzouIp9a03Q3ckpkPdSCmvjMQUTfBHojoQjSSJEnqNa3M0/1xYC3wKeDT\n5eOvAD+NiKM6GFvfOOmIfafULkmSpLmllaT7ZuCwzFyemU8DDgV+BLwA+IcOxtY3PrjiEF5z5H7b\nR7YHInjNkft5E6UkSZKA1spLDsrM6ytPMvPHEXFYZt4Ulk9s98EVh5hkS5Ikqa5Wku5NEfEJipIS\ngFdQlJbsCox1LDJJkiRplmilvOR1wI3Au4G/AG4q28aA53YqMEmSJGm2mHSkOzNHgX8s/9X6bdsj\nkiRJkmaZSZPuiHgWcDqwf3X/zHxs58KSJEmSZo9Waro/S1FWcjUw3tlwJEmSpNmnlaT77sz8Zscj\nkSRJkmapVpLuyyNiFXAh8EClMTOv6VhUkiRJ0izSStJ9RPlzeVVbAs9rfziSJEnS7NPK7CVOCyhJ\nkiTthEnn6Y6IR0fEZyPim+XzJ0bEGzofmiRJkjQ7tLI4zlnAGmBx+fynFAvlSJIkSWpBK0n33pl5\nLrANIDMfwqkDJUmSpJa1knTfGxF7Udw8SUQcCdzd0agkSZKkWaSV2UveA1wEHBgRVwKLgJd1NKo+\ntHr9CKvWbOL2raMsHh5i5dHLWHHYkm6HJUmSpB7Qyuwl10TEs4FlQACbgKd2OrB+snr9CCvP38DY\neAIwsnWUledvADDxliRJUkvlJWTmQ5l5fWb+KDPHgPM6HFdfed/F129PuCvGxpP3XXx9lyKSJElS\nL2kp6a4j2hpFn7vrvrEptUuSJGlumW7SnZN3kSRJkgRNaroj4mLqJ9cB7NWxiPrQ8NAgW0d3HNUe\nHhrsQjSSJEnqNc1upPw/09w2qYgYBj4DPIkisf9Tihs0zwGWAjcDL8/MuyIigI8AxwD3Aa/LzGt2\n5vztdvpxB7PyvA2MbXv4O8rgvOD04w7uYlSSJEnqFQ2T7sz8TgfP+xHgPzLzZRGxC7AA+Gvg25l5\nZkScDJwMvBd4EfD48t8RwCfKnz2jMkOJUwZKkiSpnunWdE9bRDwK+APgswCZ+WBmbgWOB75QdvsC\nsKJ8fDzwxSysBYYj4jEzHLYkSZI0bTOedAOPBbYAn4+I9RHxmYjYDXh0Zt4BUP78nbL/EuC2qv03\nl20TRMSbI2JdRKzbsmVLZ19BjdXrRzjlwo2MbB0lKebpPuXCjaxePzKjcUiSJKk3dSPpnk+xuM4n\nMvMw4F6KUpJG6k1PuMMNnpn5qcxcnpnLFy1a1J5IW7RqzSZGx8YntI2OjbNqzaYZjUOSJEm9adIV\nKSPi94CVwP7V/TPzedM852Zgc2ZeVT4/nyLp/mVEPCYz7yjLR35V1X/fqv33AW6f5rk7YmTr6JTa\nJUmSNLdMmnRTrD75SeDTwPgkfSeVmf8dEbdFxLLM3AQ8H/hx+e+1wJnlz6+Vu1wEvCMivkJxA+Xd\nlTKUXjEQwXjuOLviQLiGkCRJklpLuh/KzE+0+bzvBL5czlxyE/B6ilKXcyPiDcCtwIll329QTBd4\nI8WUga9vcyw7rV7C3axdkiRJc0srSffFEfE24KvAA5XGzLxzuifNzGuB5XU2Pb9O3wTePt1zzQRH\nuiVJktRMK0n3a8ufK6vakmIWEuFItyRJkpqbNOnOzANmIpB+5jLwkiRJaqaV2UsGgbdSLGgDcAXw\nfzNzxyxzjmpURWJ1iSRJkqC18pJPAIPAx8vnf1y2vbFTQfWbrffV//7RqF2SJElzSytJ99Mz8ylV\nzy+LiA2dCqgfLR4eqjsn9+LhoS5EI0mSpF7TyoqU4xFxYOVJRDyWNszXPZs896D6K2A2apckSdLc\n0spI90rg8oi4iWJJ9v3pwbmyu+nyG7ZMqV2SJElzSyuzl3w7Ih4PLKNIum/IzAcm2W1OcRl4SZIk\nNdMw6Y6I52XmZRFxQs2mAyOCzLyww7H1DRfHkSRJUjPNRrqfDVwGHFtnWwIm3SUXx5EkSVIzDZPu\nzDytfPj+zPxF9baIcMGcKo50S5IkqZlWZi+5oE7b+e0OpJ850i1JkqRmmtV0HwQcDOxRU9f9KOAR\nnQ6sn7gMvCRJkpppVtO9DHgJMMzEuu7fAG/qZFD9Zmx825TaJUmSNLc0q+n+GvC1iHhGZn5/BmPq\nO/c+WH+toEbtkiRJmltaqen+o4h4VEQMRsS3I+LXEfGajkcmSZIkzRKtJN1HZeY9FKUmm4Hfo1il\nUpIkSVILWkm6K3cDHgOcnZl3djCevtRoYkAnDJQkSRK0sAw8cHFE3ACMAm+LiEXA/Z0Nq780mhjQ\nCQMlSZIELYx0Z+bJwDOA5Zk5BtwHHN/pwCRJkqTZYtKkOyIWAG8HPlE2LQaWdzKofrNwQf35uBu1\nS5IkaW5ppab788CDwDPL55uBD3Ysoj704ic/ZkrtkiRJmltaSboPzMx/AMYAMnMU7xGc4PIbtkyp\nXZIkSXNLKzdSPhgRQ5T3BUbEgcADHY2qz9y+dXRK7d22ev0Iq9Zs4vatoyweHmLl0ctYcdiSbocl\nSZI0a7WSdJ8G/Aewb0R8GXgW8LpOBtVvhhcMctd9Y3Xbe83q9SOccuFGRseK1TJHto5yyoUbAUy8\nJUmSOqSV2Uu+BZxAkWifTTGLyRWdDau/PDBWf7n3Ru3dtGrNpu0Jd8Xo2Dir1mzqUkSSJEmz36Qj\n3RHxB+XD35Q/nxgRZOZ3OxdWf7lvbNuU2rup30phJEmSZoNWykuql3x/BHA4cDXwvI5EpI5aPDzE\nSJ0Ee/HwUBeikSRJmhtaKS85turfHwJPAn7Z+dD6x4LB+m9jo/ZuWnn0MoYGBya0DQ0OsPLoZV2K\nSJIkafabTla4mSLxVmmX+QNTau+mFYct4aVPW8JAFLM+DkTw0qct8SZKSZKkDmqlpvtfKKcLpEjS\nDwU2dDKofnP36I4zlzRr76bV60e44OoRxrP4SMczueDqEZbvv6eJtyRJUoe0UtO9rurxQ8DZmXll\nh+LpS/PnQb17Juf3XnVJ09lLTLolSZI6o5Wk+zzgceXjTZnpwjg1Gk1S0oOTlzh7iSRJUhc0HIuN\niMGI+DBwG/B54AvATRFxcrn9sJ05cUQMRMT6iPh6+fyAiLgqIn4WEedExC5l+67l8xvL7Ut35rxz\nXaNZSpy9RJIkqXOaFUD8I/BIYGlmPi0zDwOeADw2Ij4BXLiT5/5z4CdVzz8E/HNmPh64C3hD2f4G\n4K7MfBzwz2U/TZOzl0iSJM28Zkn3McCbMrOyKA6ZeQ/wVuCVwEnTPWlE7AO8GPhM+Two5v0+v+zy\nBWBF+fj48jnl9ueX/TUNzl4iSZI085ol3dsyM2sbM3Mc2JKZa3fivB8G/gqoVD3vBWzNzIfK55uB\nSha4hKLEhXL73WX/CSLizRGxLiLWbdmyZSdCm91Wrx/hnB/cNmH2knN+cBur1490OTJJkqTZq1nS\n/eOI+JPaxoh4DRPLQqYkIl4C/Cozr65urtM1W9j2cEPmpzJzeWYuX7Ro0XTDm/VOv+h6xrZNfPvG\ntiWnX3R9lyKSJEma/ZrNXvJ24MKI+FOKZd8TeDowBPzRTpzzWcBxEXEMxbLyj6IY+R6OiPnlaPY+\nwO1l/83AvsDmiJgP7AHcuRPnn9O2Npg7vFG7JEmSdl7Dke7MHMnMI4D3AzcDtwLvz8zDM3PatQiZ\neUpm7pOZSylqwy/LzFcDlwMvK7u9Fvha+fii8jnl9svqlb1IkiRJvWrSeboz8zLgshmI5b3AVyLi\ng8B64LNl+2eBf4uIGylGuF85A7HMWgsXDHLXfTuOai9cMNiFaCRJkuaGVhbH6ZjMvAK4onx8E3B4\nnT73AyfOaGCz2GnHHszK8zcwNv7wHwsGB4LTjj24i1FJkiTNbl1NujXzKlMDrlqzidu3jrJ4eIiV\nRy9zykBJkqQOMulWR6xeP2JiL0mSVDLpnmNWrx/hlAs3Mjo2DsDI1lFOuXAjQNuS4pk4hyRJUj9p\nNk+3ZqFVazZtT4YrRsfGWbVmU1+dQ5IkqZ+YdM8xI1tHp9Q+Hbc3OFajdkmSpNnOpHuOmVdvfc8m\n7dOxeHhoSu2SJEmznUn3HLOtwbJCjdqnY+le9ZPrRu2SJEmznUm32m7tTXdNqV2SJGm2M+lW241n\n/WHzRu2SJEmznVMGqu0CqJdet7FsfFLOEy5JknqJSbfartF49nTGuaeTPE91nvBWztGuPpIkaW4y\n6VbPmu4iO83mCa+XKK88fwNj47n9HCvP3zDhHKvXj7DyvA2Mbavqc16dPpMcR5IkzV3WdKtnTXeR\nnanMRf6+i6/fnihXjI0n77v4+u3PT7/o+u0J9/Y+25LTL3q4TyvHqWf1+hGedeZlHHDyJTzrzMtY\nvX6kaf9OHUOSJHWWI91tsHDBIHfdN1a3vdcsGJzHfWPb6rb3mukusjMQUfemzYHYsaq83udW2751\ntH6f6vbT7IKDAAAeS0lEQVRWjlNrOiP5tSUszz1oERdcPTLhGCvP38DpF13P3aNj2/tcfsOWaZW9\nTLdkxlIbSZIm6r1Mqw+dduzBDA5MTOgGB4LTjj24SxE19r9PePIOC+HMi6K9XRp92Zjql5DpLrLT\nL7OnTHUkv5Kkj2wdJSkS7C+vvXWHY4yNJ1tHx7b3+dLaWyfss/L8DRz6vksnHRmvd75TLtw46Uh6\nvf3+4pxrWepIvCRpDnOkuw0qI3j9MLI3E7GeduzBE+qbYXpfQlYevWzCSDDA0OAAK49e1nS/JcND\ndUtJlnRwRczhocG6I+LDQ42/aDQbya83UlwvSZ/O14hKUg5FQvyec6/l3edcu337sw7cky+/6RmT\nfilodA01i7PVunxJkmYbk+42WXHYkr5JIjoda7sS++keZyrJeiulQRFQb5C8ulrl9OMOnnCzJcDg\nvOD04xp/0Vjc4MvB8ILBumUntYlsu9SuRnrlz+/k1Z/+fsMvBbXxVCfSlefNNLqpVZKk2Syyx/7k\n3g7Lly/PdevWdTsMdVGrNcW1s45AMSq/6mVP2d7/1NUb+dLaW3fY9zVH7scHVxwy5XNW96/35WDX\n+fPqjpo3Sv5nWqOa+YULBrl/bFvLXw6WDA/1/F+GJEmaTERcnZnLJ+1n0q25rpVk+dTVGzn7qtsY\nz2QggpOO2HdCwt3Oc//FOddOq2xkJgzOix1mcmnLcWu+6EiS1C9Muk26Z4W5OAvGs868bNISjdlo\n4YJB1v/dUd0OQ5KkKWk16Xb2EvWs6c6e0e9WHr2MocGBCW1DgwNNb8qsJ6g/TWIntHqWZjezNpte\nUZKkfmfSrZ413cVx+km9hW1WHLaEM044hCXDQwRFonrGCYdw+nEH75CMNzI8NMgvznwx22boL1mT\nnWVocIAPv+JQrjz5eTMSjyRJvcbZS9Szprs4Tr+YbHn5RmU0tYvjnPOD2xrOmrJHg6kMZ9LQ4DzO\nOOGQ7a9nOtMrSpLU70y61bMaTak32eI4/aLZ8vKNEu56yfjy/fdsWPc+Q9UlTT34UE6IeTrTK0qS\n1O9MutWzprs4Tr9oZXn5VjQbFd/aA3XS45kceMo3Jsz88orD950wG8wrDt931t8gK0ma20y61bP6\naaXPXtXorwUDEdvrvWei6rsyr/d4Jl9aeyvzgG1V2y64eoTl++8J+HlLkmYnk271tH5a6XOqWlkN\nc2c996BFdRf2qcwzvvTkS9p2rqnYVvN8dGx8wlL00PqS8XNxWklJUv9x9hKpS0479mAGByYWXQ8O\nBKcd277a5stv2DKl9l4zOjbO/3fuhobTRNabVvLd51zL0pMv4dD3XTrrp5eUJPUPk26pS1YctoRV\nL3vKhKkB270q42QzwPTDjCHjmbz7nGvrJtH1ppWs2Do61nA/SZJmmuUlUhd1unxmshlg6s0kMi/g\nUY8Y5O7RMYYXDJLJhMet3ui5ZHiIpXsNceXP72zLa9k6Ojah3GT1+pGWVu7cOjo2YSpGSZK6wWXg\npVmsUn5ROwNM9bzZU62JrnfMem4+88UAnLp6Y9268ulaUs5P/uW1t07rJtAl1n1Lktqo1WXgTbql\nWa4TNxpWjtlopHnJ8NCE1ScPe/+lbV3mPdj5WVeGhwY5/biDd/jyMbJ1lIEIxjO3J/iX37Cl4fvn\njZySNLeZdJt0Sx3Xykh6pd/K8zcwNt5bv28CePWR+7F8/z1bGr2vVknIL7h6ZNLXL0mavUy6Tbql\nGdHqSG/1SPJkI9WDAzFjCXoAezRYmn66BiL4x5e396ZYSVJv6tmkOyL2Bb4I/C7FdL2fysyPRMSe\nwDnAUuBm4OWZeVdEBPAR4BjgPuB1mXlNs3OYdEu9rV4pR3VJx8qjlzUsX6n063WD84JVJz4F2HHB\nn0pbs9dvwi5J/aGXk+7HAI/JzGsiYnfgamAF8Drgzsw8MyJOBhZm5nsj4hjgnRRJ9xHARzLziGbn\nMOmW+l+j0pWXPm3JDiUdvWqXgWBg3rwJsQ4OBOPjucMCQdUqZS8fXHFIx2OUJO2cVpPuGZ8yMDPv\nAO4oH/8mIn4CLAGOB55TdvsCcAXw3rL9i1l8O1gbEcMR8ZjyOJJmqcpIb73SleX779n0Rs6FCwZZ\nsMv8CSPJ8wK2NRhjWLhgkPvHtrU9kX9wPGF84jFbKZtJ4Etrb90+60vtTZ+SpP7T1ZruiFgKfBd4\nEnBrZg5XbbsrMxdGxNeBMzPze2X7t4H3Zua6mmO9GXgzwH777fe0W265ZWZehKSuafVGzopTV2/c\nYarBSn+gaSLfa0zEJak39Gx5yfYTRzwS+A7w95l5YURsbZB0XwKcUZN0/1VmXt3o2JaXSHPHdOYZ\nb9a/1XnIe41JuCR1R08n3RExCHwdWJOZ/1S2bQKek5l3lHXfV2Tmsoj4v+Xjs2v7NTq+SbeknVGb\nmNfO1V3veTsXANpZ1Qm484hLUmf1bE13ORvJZ4GfVBLu0kXAa4Ezy59fq2p/R0R8heJGyrut55bU\nSSsOWzKtxHS6q2S229bRMVaet4F1t9w54abTka2jnHLhRgATb0maYd2YveT3gf8CNsL2G/j/GrgK\nOBfYD7gVODEz7yyT9I8BL6SYMvD1tfXctRzpltQNzeYiHxwISBhrdDdnBzSaXrFyo+ntW0cZXjBI\nZpGoO22hJE1dT5eXdJpJt6Ruq1fWAbS8QFC3OW2hJLXGpNukW1IPW71+hPddfD133dd4JczhnVwp\nc2cXEgrgn19x6ITa8OppGGtHxFvpI0mzjUm3SbekPnDo+y6tm1gvXDDI+r87qmmfZgbnBa84fN+d\nXkiokjQ3mtGlesrFRn0GB4LddpnP3aNjHb2Z05tGu6vZ++9no9nMpNukW1IfaGWu8dXrR1h53oaW\n68HrzV4y3fnHA1g8PNR0/yXDQwAtn6PZXOrTNdU522ezyWbfqZfwtjKVZqNyqcq9Ab+9/6EJ12iz\nL2Rz9bPR7GTSbdItqU+0MgpYnTw3Wl1zcF6w6sSn1E1knnXmZdNKvJcMD3H71tGm9edR/pzK/02W\nDA9x5cnPm3I8jTR6fe0+T69qpVypot6XumZJcb3trd4Y3OwL2Vz5bDT79eyUgZKkiVqZorC2T22S\nNdniOM1KRBqJcr/JRsoXT3GkG+D2Nq/82eh47T5Przl19Ub+/apb634Ja2R0bJxVazZtv1ZWrdm0\nw3VR3afe9rHx1k7Y7P2f7Z+NVMukW5L60FTnEq9OsG7fOsou8+fxwEPbGvavzF5S2a9ZTXel1GAq\nSX0lUW+XRiUw7T5PL5jKqHYj1QnvZF9YdiY5bvaFbDZ+NlIzJt2SNEc0S9SblbhUJ+yTzUxS22dh\ng1rfSqLeLvVG8jtxnm5bvX6EledvaHmkuZHqhHeyLyyT1fQ30uwL2Wz8bKTJWNMtSeqomZq5Yi7M\nkDHd2vxauwwEDzZJ3KdT091slpq58Nlo7vJGSpNuSdIsc8DJl8zIokqvqVkYabLZS0ykNZd5I6Uk\nSbPMdEs9pupLa2/lS2tvJYBnHrgnN//PaNPSI0mTc6RbkqQ+0a6a7nYYGpzHGSc8eULiferqjZx9\n1W2MZzIQwUlH7DthxFyajRzpliRplqkkuH/z1Y3c+2Dr0z8ODc5jdKzxbDXTMTq2jfecc+32uE5d\nvZEvrb11+/bxzO0j5tXq3YArzQWOdEuS1Idqpw4MigWKdttlgPseHCdh+2jz8v33nPI87a2qLHJz\n4CnfYHyKOcXgPNht18G6N19K/cKRbkmSZrGpztUOTLrQ0XRU5vGeasINMLYNto4WXxpGto5yyoUb\nAerOejK8YJDMh/tXc/Rc/cCkW5KkOaA6Sa8ks+1IwCvzeVfmZt8Z1Sth1k5V2GwxoJGto6w8fwPr\nbrmTy2/YMmGu+AprzNVtJt2SJM0xtaPk1Ul4pUwF2D57ybW33V23hnwebJ8+8KQj9t2hfns6KiPn\n9Zafb2ZsPHeoKa9WXWM+PDTI6ccdDDChRKfS7oi5OsGabkmSNKlTV2/k36+6lcp6OPVmL3n1p7/P\nlT+/c6fOU6kR7/Sc5POi+LmtwUnqrbzqIj+qx8VxTLolSZpxjUbNW1G9Ema7Vt9sh6HBAV76tCVc\ncPVI3dH34aFBXvKUx3D5DVu2J+TPPWjRhOcm6LOXSbdJtyRJPaV2tLzZ7CX1lp/vpp2tWa83u8y8\ngF3nz+P+sW0m5n3MpNukW5Kkvtbq7CVQJLADEYw1qhfpQ9aY9wenDJQkSX2t2bSI9eqr4eFpEScb\nmZ4Xjeu562nH7CxTtXV0jJXnbQBo+D5UrwJarTKy7nSKvcORbkmSNKvVLiRUGUFed8udLc24Uqnp\n/vLaWzt6c2cjlZtLa9WuAtqq2oQc8AbRnWB5iUm3JEmaRKOVPevNXjLdJHdnBfCLM1+8Q/t0VgGt\nNTgQkOxQllP5YnLeulsnzEgzOA8e2obJeRWTbpNuSZLUZrVJOkycvWQ6s7ZMptFI99KTL2njWXbU\nagnOa47cb04vOmRNtyRJUps1qzOvqK03r54+cEHV7CWtJOeD82J7CUitTteZt1rzXll0CLz5sxmT\nbkmSpDZqJTGvqDdyXjFZAtuuVUDbqZWbP+cqk25JkqQumUqCXqtS0lFv9pJuGtuWrFqzyaS7hkm3\nJElSBzVaPr7ZsvL15iivXURo9foRLr9hC9uqbvgEemJRodt7ZDXRXuKNlJIkqW80S1Rb2T6VY02l\nX7PEujYJbrSs/NDgAGecUIxeN0ucm+2/6/x5DRcQmkmNbv6cjZy9xKRbkjQLTCWJ7MT+7T7mdEZ9\nq/etl8CeccIhTRPcyvapHGsq/Zr1qSzWU6vRTZBLhocA6u7Tyv69YHBesOrEp8yZ8hKTbpNuSeqq\nXkv2OnHsTsZTOX6rSWQn9m/3Macz6lt9zGedeVndZLQyqjrZ9mqt9m2lX7M+t28dndL0gVH+7Nfs\nbC7OXuKUgeqqdv2PqNP/Q2unfop1Omb762tkrr7unVWbXI1sHeWUCzcC05/RoBPH3JljdzKeilVr\nNu1QYjA6Nt7yTWo7u3+7j9lo33o3AtY7ZqM64Ur7ZNsna6vX3kq/Zn0WDw9NaaR78U6OdC9cMMj9\nY9smvM+NFsCpmBfwqEcMsnV0bPtxF9bUkT/3oEVcct0dO6zq6e/D1pl0q+3a9T+i1etHWHnehu2/\nJEa2jvbsNESTxXrq6o3b/6cyEMFJR+zbdCGBXkv0ZiK56EVz9XW3Q68le504difjqZhKEtmJ/dt9\nzEZ9GpVJ1PZvlMBWEtXJtk/lWFPp16zPyqOXTWl0v5WbIZvtf9qxBwM7LuteaRvZOjph0ZupJM9z\neQGcduib8pKIeCHwEWAA+Exmntmo7+67755Pe9rTZiw2TbT+1q088NCOvyh2nT/AYfsNt3ycdTff\nxUPbtu3QPn/ePJYvXbhTMbZbs1j3euQu/PKe+3fY9uhHPYID9t5th/Zf//YBbtpyL9uq/tucF8Fj\nF+3G3o/ctb2Bt6hdn2m/mauvux3W3vQ/Dbcd+di9euaYO3PsTsZTsbPXYCeu4Z05ZqN9gyDrFFTU\nHnOy349T+f3Zat9W+rUS1213jvLAQ+PsOn+AffccatpeOWZl2/yBeZDw0LZtLe8/F11xxRVdOe+s\nKi+JiAHgX4E/BDYDP4yIizLzx92NTPXU+4XarL2Reklss/Zuahbrr+55oO62X93zQN2k+7Y7Ryf8\n4gbYlsltd4527Zdpuz7TfjNXX3c77Dp/oGFi1kvH3JljdzKein33HKqbzO27544jt53Yv93HbLTv\not13ZctvHpj0mJXfgY0Szcm2T+VYU+nXSlyNYmj0e73Ztqn0Ue/oi6QbOBy4MTNvAoiIrwDHA3WT\n7mXLlnXt246a31ByxRSmD1p68iUNt11x5ounFVunNIu1mXqv44CTL6l7A0006D8T2vWZ9pu5+rrb\noddu4OvEsTsZT+15nL1E6n/9knQvAW6rer4ZOKK6Q0S8GXgzwH777TdzkWkHjerXKjVlrVq4YLDu\nsrgLFwzudIwzqdHNLgMRdXpPrSZxprTrM+03c/V1t0MlWWpnEtWJY+7MsTsZT+15dvZ966WYGu3b\niTilXtIXNd0RcSJwdGa+sXz+x8DhmfnOev2dMrD72jFisXr9CCvP38DY+MPX6OBAsOplvTf352Hv\nv7ThF4QXP/kxfGntrTtse82R+9W9KWWmRs+maq6OQs3V1y1Jas2sqummGNnet+r5PsDtXYpFLWjH\niMVMjSK1w2nHHlz3C8Jpxz58R3irs5f06uueq6NQc/V1S5Laq19GuucDPwWeD4wAPwRelZnX1+vv\nSLe6wRFRSZLmnlk10p2ZD0XEO4A1FFMGfq5Rwi11iyOikiSpkb5IugEy8xvAN7odhyRJkjRV87od\ngCRJkjTbmXRLkiRJHWbSLUmSJHWYSbckSZLUYSbdkiRJUoeZdEuSJEkd1heL40xVRGwBbpmk2x7A\n3R3Yvjfw60nO3Ssme429dI7pHmcq+7XSd2f7eN3M7Dl65bpppd90rhvon2vH62Z6fef6dQP9c+14\n3fSWmbxu9s/MRZP2zsw5+Q/4VCe2A+u6/dra9R700jmme5yp7NdK353t43UzN6+bVvpN57opt/XF\nteN143XT7c+10+fwuumtf7143czl8pKLO7y9H8zEa2jXOaZ7nKns10rfne3jdTOz5+iV66aVfl43\nvXMOr5ve0i/XjtdNb+m562ZWlpd0U0Ssy8zl3Y5D/cXrRtPltaPp8LrRdHjd7Jy5PNLdKZ/qdgDq\nS143mi6vHU2H142mw+tmJzjSLUmSJHWYI92SJElSh5l0S5IkSR1m0i1JkiR1mEm3JEmS1GEm3R0W\nEbtFxBci4tMR8epux6P+EBGPjYjPRsT53Y5F/SMiVpS/a74WEUd1Ox71h4h4QkR8MiLOj4i3djse\n9Zcyz7k6Il7S7Vh6nUn3NETE5yLiVxHxo5r2F0bEpoi4MSJOLptPAM7PzDcBx814sOoZU7luMvOm\nzHxDdyJVL5nidbO6/F3zOuAVXQhXPWKK181PMvMtwMsB52Ce46aY4wC8Fzh3ZqPsTybd03MW8MLq\nhogYAP4VeBHwROCkiHgisA9wW9ltfAZjVO85i9avG6niLKZ+3ZxabtfcdRZTuG4i4jjge8C3ZzZM\n9aCzaPHaiYgXAD8GfjnTQfYjk+5pyMzvAnfWNB8O3FiOUD4IfAU4HthMkXiD7/ecNsXrRgKmdt1E\n4UPANzPzmpmOVb1jqr9vMvOizHwmYBnkHDfFa+e5wJHAq4A3RYR5ThPzux3ALLKEh0e0oUi2jwA+\nCnwsIl4MXNyNwNTT6l43EbEX8PfAYRFxSmae0ZXo1Ksa/b55J/ACYI+IeFxmfrIbwalnNfp98xyK\nUshdgW90IS71vrrXTma+AyAiXgf8OjO3dSG2vmHS3T5Rpy0z817g9TMdjPpGo+vmf4C3zHQw6huN\nrpuPUnzRl+ppdN1cAVwxs6Goz9S9drY/yDxr5kLpX/4ZoH02A/tWPd8HuL1Lsah/eN1oOrxuNB1e\nN5our502MOlunx8Cj4+IAyJiF+CVwEVdjkm9z+tG0+F1o+nwutF0ee20gUn3NETE2cD3gWURsTki\n3pCZDwHvANYAPwHOzczruxmneovXjabD60bT4XWj6fLa6ZzIzMl7SZIkSZo2R7olSZKkDjPpliRJ\nkjrMpFuSJEnqMJNuSZIkqcNMuiVJkqQOM+mWJEmSOsykW9KMiIiMiH+sev6XEXF6m459VkS8rB3H\nmuQ8J0bETyLi8jrbDo6IyyLipxHxs4j424iot3TyrBcRu0bEf0bEtRHxipptERGnlu/RTyPiOxHx\n5Daff2lEvKrq+fKI+Gibjn1zROzdjmM1OP6KiHhi1fMrImJ5p84naeaYdEuaKQ8AJ3QyYZmOiBiY\nQvc3AG/LzOfWHGOIYnW2MzPz94CnAM8E3ta2QCeeLyKil39/HwYMZuahmXlOzba3U7w3Tynfq78H\nLo6I3dp4/qXA9qQ7M9dl5rvaePxOWgE8cdJekvpOL//SljS7PAR8CviL2g21I9UR8dvy53PKkdBz\ny1HRMyPi1RHxg4jYGBEHVh3mBRHxX2W/l5T7D0TEqoj4YURcFxF/VnXcyyPi34GNdeI5qTz+jyLi\nQ2Xb3wG/D3wyIlbV7PIq4MrMvBQgM++jWL3t5HLfR0bE58tjXhcRLy3bXxgR10TEhoj4dtl2ekT8\nZVUsPypHbpeWo+wfB64B9o2IoyLi++UxzouIR5b73BwR7yvbN0bEQZPE0eg4Z0bEj8u+/6fO+7Rn\nRKwut6+NiCdHxO8AXwIOLUe6D6zZ7b3AO8v3iPI9+y7w6urPvnz8sog4q3y8KCIuKD/LH0bEs8r2\nZ5fnuTYi1kfE7sCZwP8q2/6i/Ly/3ijmqvf9c+XI8k0R0XKSHhG7lfv+sIzh+LL9dRFxYUT8RxQj\n+/9Qtc8bymv1ioj4dER8LCKeCRwHrKp5704sr/mfRsT/ajUuSb1lfrcDkDSn/CtwXXXy0YKnAE8A\n7gRuAj6TmYdHxJ8D7wTeXfZbCjwbOBC4PCIeB/wJcHdmPj0idgWujIhLy/6HA0/KzF9UnywiFgMf\nAp4G3AVcGhErMvP9EfE84C8zc11NjAcDV1c3ZObPyyT3UcDflHEcUp5jYUQsAj4N/EFm/iIi9mzh\nvVgGvD4z3xbFXwxOBV6QmfdGxHuB9wDvL/v+OjOfGhFvA/4SeCPwt3XiqHuciPgY8EfAQZmZETFc\nJ573Aeszc0X53nwxMw+NiDeW79NLat7bRwG7ZebPa46zjslHdz8C/HNmfi8i9qNYjvoJ5Wt7e2Ze\nWX5ZuJ/iy87280fEc5rFDBxabjsIeC6wO7ApIj6RmWOTxAXF53tZZv5p+T79ICL+s9x2KMXI/wPl\nMf8FGKf4LJ4K/Aa4DNiQmf8vIi4Cvp6Z55exA8wvr/ljgNOAF7QQk6QeY9ItacZk5j0R8UXgXcBo\ni7v9MDPvAIiInwOVpHkjRYJUcW5mbgN+FhE3USRQRwFPjodH0fcAHg88CPygNuEuPR24IjO3lOf8\nMvAHwOomMQaQDbYlRZL0yu0NmXdFxLHAdysxZOadTY5fcUtmri0fH0mRqF5ZJma7AN+v6nth+fNq\n4ITycb04XtLgOPdQJLCfiYhLgK/Xief3gZeWx7osIvaKiD1aeB21Wql9fwHwxHi4TP5R5aj2lcA/\nlZ/ThZm5OZqX0jeL+ZLMfAB4ICJ+BTwa2NxCbEcBx8XDf6F4BLBf+fjbmXk3QET8GNgf2Bv4TuUz\nj4jzgN9rcvzqz3JpC/FI6kEm3ZJm2ocpyiM+X9X2EGW5WxQZ0y5V2x6oeryt6vk2Jv4Oq016kyKZ\ne2dmrqneUI583tsgvunc/Hg9RWJefY7HAr/NzN+Ur6k2vkaJ+vb3ovSIqsfVMQfwrcw8qUFMlfdp\nnIffp0Zx1D1ORBwOPJ8iUX8H8Lw6+9Zq9OWj8qXr3oh4bGbeVLXpqTz8Zap6/+rXPg94RmbWflk7\ns/xScAywNiImGwVuFnP1tVb9vk0mgJdm5qYJjRFHNDjmVK+xep+lpD5jTbekGVWO7p1LcVNixc0U\n5RwAxwOD0zj0iRExr6yDfSywiaIE4a0RMQgQEb8Xk9+wdxXw7IjYO4qbLE8CvjPJPl8Gfr+S8EVx\nY+VHgUoZzaUUSSvl9oUUo8nPjogDyrZKecnNFEkoEfFU4IAG51wLPKssoyEiFkREs9HSRnHUPU5Z\nqrFHZn6DooTn0DrHq67Ffg5FScs9k8SwCvho+R5RvmcHA+eX238ZEU+I4kbRP2oS+6HlzwMzc2Nm\nfoiiTOUgipKN3RucfzoxT2YN8M7yyxURcdgk/X9A8dkvjIj5lCPvpWaxS+pjJt2SuuEfKf7EXvFp\niiTkB8ARNB6FbmYTRXL8TeAtmXk/8Bngx8A1EfEj4P8yyUhhWcpyCnA5sAG4JjO/Nsk+oxRfFk6N\niE0UpS8/BD5WdvkgsDCKmyI3AM8ty1feDFxYtlVm+bgA2DMirgXeCvy0wTm3AK8Dzo6I6yiS54Oa\nxdkkjnrH2R34etn2HercAAucDiwv+5wJvHaS8wP8C0XSeV1E3ExRU/2H5ecFRT321ynqnO+o2u9d\nlXOVZRpvKdvfXfV6Rik+/+uAh6K4QbU27unEXOu6iNhc/vsn4AMUXxSvK6+zDzTbOTNHgP9N8QXv\nPymu0bvLzV8BVkZxQ2btTaiS+lhkNvxLoCRJHVOOpn+Vom7/r7sdz0yKiEdm5m/Lke6vAp/LzK92\nOy5JnWPSLUnSDItiCsYXUNStXwr8efo/ZGlWM+mWJEmSOsyabkmSJKnDTLolSZKkDjPpliRJkjrM\npFuSJEnqMJNuSZIkqcP+f45UatH6joQjAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f59b193e320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "questions = pd.Series(list(set(training_data['question1'].tolist() + training_data['question2'].tolist())))\n",
    "question_lengths = questions.str.len().value_counts()\n",
    "plt.figure(figsize=(12, 5))\n",
    "plt.scatter(question_lengths, question_lengths.index)\n",
    "plt.xscale('log', nonposy='clip')\n",
    "plt.title('Log-Histogram of Question Length Counts')\n",
    "plt.xlabel('Number of Occurences of Question Length')\n",
    "plt.ylabel('Question Length')\n",
    "plt.hlines(10, 0, 1.9e4)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems that there is a minority of very short questions. We will see\n",
    "whether dropping them from the training data set will help us improve\n",
    "the accuracy of predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Balancing Classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have already noticed that in the training data set the number of\n",
    "duplicates constitute about a third of the entire corpus. This may\n",
    "skew our predictions by increasing the number of false positives.\n",
    "There are 2 approaches I have tried to remediate this problem:\n",
    "\n",
    "-   generate an additional set of duplicate questions by\n",
    "    replacing each noun in a randomly chosen duplicate question with its\n",
    "    synonym, increasing the percentage content of duplicate questions to a\n",
    "    half of the entire data sets\n",
    "-   apply up-down scaling to the training dataframe, decreasing the\n",
    "    percentage content to about a fifth of the entire dataframe, which\n",
    "    [was estimated](https://www.kaggle.com/badat0202/estimate-distribution-of-data-in-lb) to be close to the percentage content of duplicates in\n",
    "    the test data set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our cleaning-up process is threefold:\n",
    "\n",
    "1.  We remove unnecessary punctuation, pad the words, deal with\n",
    "    non-ascii characters and fix simple typos.\n",
    "2.  SpaCy is used to replace quantities and other entities\n",
    "    with something more amenable to later processing by our machine\n",
    "    learning model.\n",
    "3.  We remove questions deemed too short to bear any meaningful content.\n",
    "4.  Finally, we oversample duplicates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with Missing Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since reproducible builds of [NixOS](https://nixos.org/) disallow the vanilla usage of `pip`\n",
    "for Python package management, we [first](https://github.com/NixOS/nixpkgs/pull/25804) [add](https://github.com/NixOS/nixpkgs/pull/25791) tools like `gensim` and\n",
    "`spaCy` to [Nixpkgs](https://nixos.org/nixpkgs/manual/).\n",
    "Now we are ready to proceed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from preprocess import Preprocessor\n",
    "prep = Preprocessor(training_filename)\n",
    "training_data = prep.processed_questions()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We save the data sets after each step so that the corresponding models\n",
    "could be compared later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modelling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building Naive Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To avoid the time-wasting bog of fine-tuning a deliberately chosen\n",
    "model which might not be the best choice to analyse the data, we build\n",
    "and improve upon a benchmark model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Naive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LSTM (long short-term memory) recurrent neural network with a\n",
    "non-trainable word2vec embedding layer, inspired by [@lystdo](https://www.kaggle.com/lystdo/lstm-with-word2vec-embeddings), seems\n",
    "suitable, considering the success of LSTM models in a diverse range of\n",
    "natural language processing such as text compression and handwriting\n",
    "recognition.\n",
    "Note that we are going to train on the data set with oversampled and\n",
    "perturbated positive class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.naive_model import NaiveModel\n",
    "nm = NaiveModel()\n",
    "nm.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ten epochs of the model require substantial computing resources &#x2013;\n",
    "running the snippet above took almost 17 hours on my laptop. The model\n",
    "with minimal fine-tuning gives us a score of 0.31985, while the\n",
    "logloss of the model validated on the training dataframe was 0.2480,\n",
    "which we would now try to improve.\n",
    "\n",
    "Our strategy is to identify relevant features and build a faster\n",
    "model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aleph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will try to use less computationally expensive techniques.  \n",
    "First of all, we dispense with recurrent neural networks in our naive\n",
    "model and build upon [the work of Bradley Pallen](https://github.com/bradleypallen/keras-quora-question-pairs), constructing a\n",
    "Siamese neural network with 300-dimensional Google News Word2Vec\n",
    "embeddings branches passing through TimeDistributed Dense layers and\n",
    "merged to be processed via fully-connected Dense layers activated with\n",
    "ReLU.\n",
    "Then we reweigh the classes to account for the disparity in the class\n",
    "prevalence.\n",
    "\n",
    "-   **Training data set**: oversampled\n",
    "-   **Dense dimensions**: 137\n",
    "-   **Dropout rate for hidden layers**: 0.38"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.aleph_model import Aleph\n",
    "aleph = Aleph()\n",
    "aleph.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the speed the price is logloss: Aleph scores only 0.32497 on the\n",
    "LB.\n",
    "![img](img/accuracy_dynamics.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Aleph\n",
    "-   added kernel constraints to Dense layers to ensure the maximum norm\n",
    "    does not exceed 3\n",
    "-   ordered Dense layers in the form of an inverted pyramid\n",
    "-   **Training data set**: oversampled\n",
    "-   **Dense dimensions**: 126\n",
    "-   **Dropout rate for hidden layers**: 0.37"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.bet_model import Bet\n",
    "bet = Bet()\n",
    "bet.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model seems to undertrain, judging by the fact that the validation\n",
    "accuracy is always bigger than the training accuracy, and gives us a\n",
    "worse score 0.33641.\n",
    "\n",
    "![img](img/2017-05-26-1043-beta_126_0.37-accuracy_dynamics.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.vet_model import Vet\n",
    "vet = Vet()\n",
    "vet.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Bet\n",
    "-   made the choice of the dropout rate for hidden layers more uniform\n",
    "    in the range 0.20 to 0.40\n",
    "-   increased the hidden layer dimensions to the range [280, 300]\n",
    "-   changed ReLU to PReLU\n",
    "-   removed the dropout after the batch normalisation of the merge layer\n",
    "-   removed kernel constraints\n",
    "-   flattened the pyramidal structure of the hidden layers\n",
    "-   **Training data set**: oversampled\n",
    "-   **Dense dimensions**: 282\n",
    "-   **Dropout rate for hidden layers**: 0.27\n",
    "\n",
    "![img](img/2017-05-26-1818-vet_282_0.27-accuracy_dynamics.png)\n",
    "\n",
    "The model evidently overtrains after the third epoch, and scores 0.35179, even\n",
    "worse than all the other models we have looked at."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gimmel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.gimmel_model import Gimmel\n",
    "gimmel = Gimmel()\n",
    "gimmel.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Vet\n",
    "-   added Krzysztof Dziedzic's [magic feature II](https://www.kaggle.com/c/quora-question-pairs/discussion/33287), inspired by @justfor's [implementation](https://www.kaggle.com/justfor/edges/code)\n",
    "-   added @tarobxl's [max kcore feature](https://www.kaggle.com/c/quora-question-pairs/discussion/33371).\n",
    "-   feeding custom features through a separate hidden layer with\n",
    "    sandwiching dropout and batch normalisation\n",
    "-   increased the range of the dropout rate to [0.25, 0.50]\n",
    "-   decreased the range of the hidden dimensions to [190, 275]\n",
    "-   organised the hidden layers into the pyramidal form\n",
    "-   switched to adam with a clipnorm as an optimizer\n",
    "-   changed the training data set\n",
    "-   **Training data set**: spaCy-treated, with shorties trimmed\n",
    "-   **Dense dimensions**: 214\n",
    "-   **Dropout rate for hidden layers**: 0.42\n",
    "\n",
    "![img](img/2017-05-27-1204-gimmel_214_0.42-accuracy_dynamics.png)\n",
    "\n",
    "The magic features play in, giving us the Public LB score of 0.24602."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dalet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.dalet_model import Dalet\n",
    "dalet = Dalet()\n",
    "dalet.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Gimmel\n",
    "-   custom features as a *pure* input layer with Dropout\n",
    "-   added [@abhishek's features](https://github.com/abhishekkrthakur/is_that_a_duplicate_quora_question) [provided](https://www.kaggle.com/c/quora-question-pairs/discussion/31284) by @raddar\n",
    "-   **Training data set**: spaCy-treated, with shorties trimmed\n",
    "-   **Dense dimensions**: 242\n",
    "-   **Dropout rate for hidden layers**: 0.32\n",
    "\n",
    "![img](img/2017-05-27-1945-dalet_242_0.32-accuracy_dynamics.png)\n",
    "\n",
    "The performance is very similar to Gimmel's, but Dalet seems to\n",
    "overfit more, as reflected in its score of 0.24801."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### He"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.he_model import He\n",
    "he = He()\n",
    "he.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Dalet\n",
    "-   used the train data set with words stemmed and stopwords removed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from helpers.stem import Stem\n",
    "s = Stem()\n",
    "s.stem_questions()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   removed @abhishek's features in the attempt to avoid overfitting\n",
    "-   added @jturkewitz's [magic feature](https://www.kaggle.com/jturkewitz/magic-features-0-03-gain)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from helpers.magic import Magic\n",
    "m = Magic()\n",
    "m.spell()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   added a hidden layer for custom features for feature detection\n",
    "-   **Training data set**: spaCy-treated, with shorties trimmed, words\n",
    "    stemmed and stopwords removed\n",
    "-   **Dense dimensions**: 199\n",
    "-   **Dropout rate for hidden layers**: 0.49\n",
    "\n",
    "![img](img/2017-05-28-0605-he_199_0.49-accuracy_dynamics.png) Now, the\n",
    "model converges well, and no significant overlearning seems to\n",
    "occur. However, the score jumps up to 0.26713, and I make a hypothesis\n",
    "that stemming is to blame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vav"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.vav_model import Vav\n",
    "vav = Vav()\n",
    "vav.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on the naive model (switched to LSTM)\n",
    "-   removed stemming, with stopwords still excluded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from helpers.stopword import StopwordTrim\n",
    "s = StopwordTrim()\n",
    "s.remove_stopwords()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   **Training data set**: spaCy-treated, with shorties trimmed and stopwords removed\n",
    "-   **Dense dimensions**: 198\n",
    "-   **Dropout rate for hidden layers**: 0.46\n",
    "\n",
    "![img](img/2017-05-28-2256-vav_198_0.46-accuracy_dynamics.png)\n",
    "The model converges quickly and seems to underlearn, but gives the\n",
    "best score so far &#x2013; 0.24360."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Zayin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.zayin_model import Zayin\n",
    "zayin = Zayin()\n",
    "zayin.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Vav\n",
    "-   included Abhishek's features\n",
    "-   increased the dimensionality of the Dense layer for custom features\n",
    "    to the default for the others\n",
    "-   **Training data set**: spaCy-treated, with shorties trimmed and stopwords removed\n",
    "-   **Dense dimensions**: 215\n",
    "-   **Dropout rate for hidden layers**: 0.40\n",
    "\n",
    "![img](img/2017-05-29-1751-zayin_215_0.40-accuracy_dynamics.png)\n",
    "\n",
    "As we can see on the graph, the model is not as stable as the others\n",
    "in the beginning, but converges after the 7 epoch. The disparity\n",
    "between the training and validation set is not as big now if compared\n",
    "with Vav's, but the public LB score of 0.25464 indicates that the\n",
    "model generalises a bit worse than Vav."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Khet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.khet_model import Khet\n",
    "khet = Khet()\n",
    "khet.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Zayin (switched from LSTM to the Convolution1D layers with\n",
    "    with GlobalMaxPooling1D and PReLU activation for the embedding\n",
    "    layers to improve the speed of training)\n",
    "-   removed Abhishek's features in the attempt to improve generalisation\n",
    "-   **Training data set**: spaCy-treated, with shorties trimmed and stopwords removed\n",
    "-   **Dense dimensions**: 261\n",
    "-   **Dropout rate for hidden layers**: 0.41\n",
    "\n",
    "![img](img/2017-05-29-2059-khet_261_0.41-accuracy_dynamics.png)\n",
    "\n",
    "We pay the price for delegating feature extraction to the model with\n",
    "speed, receiving the score of 0.27044."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing Models and Ensembling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the ways to improve our prediction accuracy is to ensemble the\n",
    "different predictions we have.\n",
    "We resort to GM and AM averaging.\n",
    "First, let's combine all our predictions into one with the help of a\n",
    "geometric mean:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import glob\n",
    "import pandas as pd\n",
    "from scipy.stats.mstats import gmean\n",
    "\n",
    "predPath = \"ensembling/\"\n",
    "models = []\n",
    "\n",
    "for file_name in glob.glob(predPath + '*.csv'):\n",
    "    df = pd.read_csv(file_name, encoding = \"utf-8\")\n",
    "    models.append(df.ix[:, :1])\n",
    "\n",
    "total = pd.concat(models, axis=1)\n",
    "gm_df = pd.DataFrame()\n",
    "gm_df['is_duplicate'] = gmean(total, axis=1)\n",
    "gm_df['test_id'] = df.ix[:,1]\n",
    "gm_df.to_csv(\"gm_ensemble.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This simple procedure got us a score of 0.21736, which is an increase\n",
    "of 0.02624 from the score of Vav, our best model.\n",
    "\n",
    "Now, we attempt rank averaging to improve our result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "GLOB_PREDICTIONS = \"./ensembling/*\"\n",
    "RK_AVG_OUTPUT = \"./rk_avg_ensemble.csv\"\n",
    "\n",
    "from collections import defaultdict\n",
    "from glob import glob\n",
    "\n",
    "def rk_avg_ensemble(glob_files, loc_outfile):\n",
    "  # write to the chosen file\n",
    "  with open(loc_outfile,\"w\") as outfile:\n",
    "    # store the raw ranks\n",
    "    # keys are in the form (<line number>, <id>)\n",
    "    all_ranks = defaultdict(list)\n",
    "    # i is needed to write the header of the output file\n",
    "    for i, glob_file in enumerate( glob(glob_files) ):\n",
    "      # define a container for the ranks found in a file\n",
    "      file_ranks = []\n",
    "      print(\"Parsing now:\", glob_file)\n",
    "      # sort <glob_file> by the predicted probability, ignoring the first line\n",
    "      lines = open(glob_file).readlines()\n",
    "      lines = [lines[0]] + sorted(lines[1:])\n",
    "      for line_number, line in enumerate( lines ):\n",
    "        if line_number == 0 and i == 0:\n",
    "          outfile.write( line )\n",
    "          # if the line is not a header, process it\n",
    "        elif line_number > 0:\n",
    "          # store the row in a list\n",
    "          row = line.strip().split(\",\")\n",
    "          # <row[0]> is the predicted probability\n",
    "          # <row[1]> is the id of the question\n",
    "          file_ranks.append( (float(row[0]), int(row[1])) )\n",
    "          # sort by the initially predicted probability\n",
    "          # and give it a rank\n",
    "      for rank, container in enumerate( sorted(file_ranks) ):\n",
    "        # Store the rank in the dictionary for further processing.\n",
    "        # The key is in the form <id>.\n",
    "        all_ranks[container[1]].append(rank)\n",
    "\n",
    "    print(\"The number of lines with the stored ranks: {}.\".format(len(all_ranks.keys())))\n",
    "    \n",
    "    # define a list as a container for the average ranks\n",
    "    average_ranks = []\n",
    "    # k is in the form (<line number>, <id>);\n",
    "    # first sort by the id\n",
    "    for identifier in sorted(all_ranks):      \n",
    "      # append the average rank together with the identifier\n",
    "      average_ranks.append((sum(all_ranks[identifier])/len(all_ranks[identifier]),\n",
    "                            identifier))\n",
    "      # define a list as a container for the sorted ranks\n",
    "    normalised_ranks = []\n",
    "    # the element of the <average_ranks> is in the format\n",
    "    # (<average_rank>, <identifier>), where\n",
    "    # <identifier> is <test_id>\n",
    "    \n",
    "    for rank, avg_rk_obj in enumerate(sorted(average_ranks)):\n",
    "      # sort <average_ranks> by the average rank and\n",
    "      # append (<id>, <normalized probability>)\n",
    "      # to <sorted_ranks> \n",
    "      normalised_ranks.append((avg_rk_obj[1],\n",
    "                               rank/(len(average_ranks)-1)))\n",
    "      # norm_prob_obj is in the format\n",
    "      # (<id>, <normalised probability>)\n",
    "      # sort <normalised_ranks> by <test_id> and write to the file\n",
    "    for norm_prob_obj in sorted(normalised_ranks):\n",
    "      outfile.write(\"{},{}\\n\".format(norm_prob_obj[1],\n",
    "                                     norm_prob_obj[0]))\n",
    "      print(\"Saved the normalised probabilites to {}.\".format(loc_outfile))\n",
    "\n",
    "rk_avg_ensemble(GLOB_PREDICTIONS, RK_AVG_OUTPUT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, this approach does not yield anything useful, scoring\n",
    "at 0.90852, which is probably due to the fact that the estimated\n",
    "number of duplicates is only a fifth of the entire data\n",
    "set.\n",
    "\n",
    "Now we will try the weighted arithmetic mean.\n",
    "\n",
    "First, let's define a WAM ensembling function which we are going to\n",
    "use throughout our analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def ensemble(weights, output):\n",
    "    import glob\n",
    "    import pandas as pd\n",
    "\n",
    "    weighted_models = []\n",
    "    for file_name in glob.glob(predPath + '*.csv'):\n",
    "        df = pd.read_csv(file_name, encoding = \"utf-8\")\n",
    "        weighted_models.append((df['is_duplicate'],weights[file_name]))\n",
    "    \n",
    "    wam_df = df['is_duplicate'] * 0.0\n",
    "    \n",
    "    for model, weight in weighted_models:\n",
    "        wam_df += model * weight\n",
    "    wam_df /= sum(weights.values())    \n",
    "        \n",
    "    wam_df = pd.concat([df['test_id'], wam_df], axis=1)\n",
    "    wam_df.to_csv(output, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we rank the models which we want to ensemble."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predPath = \"ensembling/\"\n",
    "weights = {\n",
    "# PB Scores are given above the file name\n",
    "#0.25464\n",
    "predPath + \"0.2092_zayin_215_0.40.csv\":2, \n",
    "#0.24360\n",
    "predPath + \"0.2200_vav_198_0.46.csv\":5,\n",
    "#0.26713\n",
    "predPath + \"0.2217_he_199_0.49.csv\":1,\n",
    "#0.24602\n",
    "predPath + \"0.2346_gimmel_214_0.42.csv\":4,\n",
    "#0.24801\n",
    "predPath + \"0.2364_dalet_242_0.32.csv\":3\n",
    "}\n",
    "\n",
    "ensemble(weights, \"wam_ensemble.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Weighted AM turns out to yield the best score so far, giving 0.21623\n",
    " &#x2013; we will try to improve it by building better models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Constructing a Library of Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What can we learn about the data set without embedding the texts into\n",
    "the 300-dimensional vector space and letting the neural net out on the\n",
    "embeddings, which might not be representative of the semantics due to\n",
    "inevitable typos and other oddities of formatting?\n",
    "\n",
    "We will try to find out by building XGBoost models trained on\n",
    "hand-picked features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   built metrics and XGBoost model [inspired by @act444](https://www.kaggle.com/act444/lb-0-158-xgb-handcrafted-leaky)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from custom.features import CustomFeatures\n",
    "cf = CustomFeatures()\n",
    "cf.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   removed Abhishek's features\n",
    "-   **Training data set**: spaCy-treated, with shorties trimmed and stopwords removed\n",
    "-   **Result**: 0.24224\n",
    "-   **Features Importance**:\n",
    "\n",
    "![img](img/2017-05-30-1512-0.222026460989_tet_xgboost-feature_importance.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Yud"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.yud_model import Yud\n",
    "yud = Yud()\n",
    "yud.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Tet\n",
    "-   changed base score to 0.175 from 0.2\n",
    "-   added Abhishek's features\n",
    "-   **Training data set**: spaCy-treated, with shorties trimmed and stopwords removed\n",
    "-   **Training result**: 0.2244, which is worse than the training result\n",
    "    of Tet (0.22437), hence Yud's predictions were not submitted\n",
    "-   **Features Importance**:\n",
    "\n",
    "![img](img/2017-05-30-1553-0.224371846975_yud_xgboost-feature_importance.png)\n",
    "Notice how significant Abhishek's features are, given that they\n",
    "provide vital information about the NLP features of questions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kaf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.kaf_model import Kaf\n",
    "kaf = Kaf()\n",
    "kaf.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Yud\n",
    "-   changed the training data set to include stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from magic import Magic\n",
    "TRAIN_DATA_FILENAME = \"2017-05-24-1818-shorties_trimmed_train\"\n",
    "m = Magic(train_data_filename=TRAIN_DATA_FILENAME)\n",
    "m.spell()\n",
    "\n",
    "from custom.features import CustomFeatures\n",
    "cf = CustomFeatures(train_data_filename=TRAIN_DATA_FILENAME)\n",
    "cf.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   changed `max_depth` from 7 to 10 and `subsample` from 0.6 to 0.55\n",
    "-   **Training data set**: spaCy-treated, with shorties trimmed\n",
    "-   **Result**: 0.24253\n",
    "-   **Features Importance**:\n",
    "\n",
    "![img](img/2017-05-30-1801-0.214863010529_kaf_xgboost-feature_importance.png)\n",
    "Notice how the priority of `tfidf_wm` feature increases with stopwords\n",
    "included. The new settings of the model, however, decrease its\n",
    "predictive power."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lamed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.lamed_model import Lamed\n",
    "lamed = Lamed()\n",
    "lamed.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   trying ensembling with 5 classifiers, inspired by [@emanuele's work](https://github.com/emanuele/kaggle_pbr/blob/master/blend.py)\n",
    "-   **Training data set**: spaCy-treated, with shorties trimmed\n",
    "-   **Result**: 0.24785\n",
    "-   The result is surprisingly good for the time spent training the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attempt at Repeated Ensembling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will try to ensemble again to see whether Lamed gives us any improvement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predPath = \"ensembling/\"\n",
    "weights = {\n",
    "# PB Scores are given above the file name\n",
    "#0.21623\n",
    "predPath + \"wam_ensemble.csv\":8,\n",
    "#0.21736\n",
    "predPath + \"gm_ensemble.csv\":7,\n",
    "#0.24224\n",
    "predPath + \"0.222026460989_tet_xgboost.csv\":6,\n",
    "#0.24253\n",
    "predPath + \"0.214863010529_kaf_xgboost.csv\":5,\n",
    "#0.24360\n",
    "predPath + \"0.2200_vav_198_0.46.csv\":4,\n",
    "#0.24602\n",
    "predPath + \"0.2346_gimmel_214_0.42.csv\":3,\n",
    "#0.24785\n",
    "predPath + \"0.319533794385_lamed_xgboost.csv\":2,\n",
    "#0.24801\n",
    "predPath + \"0.2364_dalet_242_0.32.csv\":1\n",
    "}\n",
    "\n",
    "ensemble(weights, \"wam_ensemble_1.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, the second ensemble scores 0.20882, which is the best score so\n",
    "far. What if we ensemble again?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predPath = \"ensembling/\"\n",
    "weights = {\n",
    "    # PB Scores are given above the file name\n",
    "    #0.20882\n",
    "    predPath + \"wam_ensemble_1.csv\":8,\n",
    "    #0.21623\n",
    "    predPath + \"wam_ensemble.csv\":7,\n",
    "    #0.21736\n",
    "    predPath + \"gm_ensemble.csv\":6,\n",
    "    #0.24224\n",
    "    predPath + \"0.222026460989_tet_xgboost.csv\":5,\n",
    "    #0.24253\n",
    "    predPath + \"0.214863010529_kaf_xgboost.csv\":4,\n",
    "    #0.24360\n",
    "    predPath + \"0.2200_vav_198_0.46.csv\":3,\n",
    "    #0.24602\n",
    "    predPath + \"0.2346_gimmel_214_0.42.csv\":2,\n",
    "    #0.24785\n",
    "    predPath + \"0.319533794385_lamed_xgboost.csv\":1\n",
    "}\n",
    "ensemble(weights, \"wam_ensemble_2.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The third ensemble is worse than the parental second, which is\n",
    "consistent with [the reports](https://mlwave.com/kaggle-ensembling-guide/) that weighted AM ensembling works better\n",
    "with the least correlated predictions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.mem_model import Mem\n",
    "mem = Mem()\n",
    "mem.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Lamed\n",
    "-   added NN MLPClassifier with pyramidal structure of three hidden layers\n",
    "-   removed Lamed's GradientBoostingClassifier\n",
    "-   changed the number of estimators to 180 each\n",
    "-   **Training data set**: spaCy-treated, with shorties trimmed\n",
    "-   **Result**: 0.24785\n",
    "\n",
    "The model received the worst score of 6.55941 after very quick\n",
    "training, which indicates severe overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Nun"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.nun_model import Nun\n",
    "nun = Nun()\n",
    "nun.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on He\n",
    "-   changed the training data set (included stopwords)\n",
    "-   wrote the code for computing similarity measures based on NLTK\n",
    "    WordNet and Brown corpora, but decided against their inclusion due\n",
    "    to the speed of their computation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from custom.similarity import SimilarityFeatures\n",
    "sf = SimilarityFeatures(train_data_filename=\"2017-05-24-1818-shorties_trimmed_train\")\n",
    "sf.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   added Abhishek's features with `euclidean_distance` and\n",
    "    `jaccard_distance` dropped, as suggested by their feature importance\n",
    "-   added Dropout layers to `Q1`, `Q2` and `custom_features` branches\n",
    "    before the merging layer\n",
    "-   swapped places of dropout and batch normalisation layers after merging\n",
    "-   decreased the dropout rate for the question layers\n",
    "-   decreased the dimension of the dense layer for custom features\n",
    "-   added up-down sampling\n",
    "-   changed the optimizer to nadam\n",
    "-   added custom features\n",
    "-   removed reweighing\n",
    "-   **Training data set**: spaCy-treated, with shorties trimmed\n",
    "-   **Base dense dimensions**: 407\n",
    "-   **Base dropout rate for hidden layers**: 0.47\n",
    "\n",
    "![img](img/2017-06-04-1743-nun_407_0.47-accuracy_dynamics.png)\n",
    "The score obtained is 0.24987, which is an improvement from He's\n",
    "0.26713."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Samech"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.samech_model import Samech\n",
    "samech = Samech()\n",
    "samech.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Kaf\n",
    "-   decreased `eta` to 0.01 to prevent overfitting\n",
    "-   increased `max_depth` to 8 to make the model more complex\n",
    "-   increased base score to 0.19\n",
    "-   removed Abhishek's features\n",
    "-   **Training data set**: spaCy-treated, with shorties trimmed\n",
    "\n",
    "![img](img/2017-06-02-2225-0.212772886818_samech_xgboost-feature_importance.png)\n",
    "The score of 0.24505 is worse than Kaf's 0.24253, which is probably\n",
    "due to the lack of good features. We have already seen in Yud's\n",
    "predictions that including Abhishek's can boost the results, so this\n",
    "run was not productive &#x2013; the model does seem to learn more from less,\n",
    "however."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.pe_model import Pe\n",
    "pe = Pe()\n",
    "pe.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Mem without MLPClassifier\n",
    "-   switched to mlxtend StackedClassifier without folding\n",
    "-   **Training data set**: spaCy-treated, with shorties trimmed-\n",
    "-   overfits significantly &#x2013; LB Score of 0.42179"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.phe_model import Phe\n",
    "phe = Phe()\n",
    "phe.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Pe\n",
    "-   removed minimax regularization of predictions\n",
    "-   Increased the dimensions of MLPClassifier and squashed the pyramidal architecture of hidden layers\n",
    "-   Decreased the batch\\_size of MLPClassifier\n",
    "-   **Training data set**: spaCy-treated, with shorties trimmed-\n",
    "-   overfits extremely &#x2013; LB Score of 6.29678, consistent with Mem's\n",
    "    poor result, which is probably due to the addition of MLPClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tsadi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.tsadi_model import Tsadi\n",
    "tsadi = Tsadi()\n",
    "tsadi.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Nun (switched to LSTM)\n",
    "-   removed one Dense layer\n",
    "-   decreased the dropout rate for the question branches\n",
    "-   **Training data set**: spaCy-treated, with shorties trimmed\n",
    "-   was not run due to the computational cost and realisation that\n",
    "    more careful base data selection is necessary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.quf_model import Quf\n",
    "quf = Quf()\n",
    "quf.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Yud\n",
    "-   changed the base dataframe to the train file with minimal preprocessing\n",
    "-   excluded magic2 and kcore features\n",
    "-   added Abhishek's `euclidean_distance` and `jaccard_distance`\n",
    "-   **Training data set**: vanilla"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from magic import Magic\n",
    "TRAIN_DATA_FILENAME = \"vanilla_train\"\n",
    "m = Magic(train_data_filename=TRAIN_DATA_FILENAME)\n",
    "m.spell()\n",
    "\n",
    "from custom.features import CustomFeatures\n",
    "cf = CustomFeatures(train_data_filename=TRAIN_DATA_FILENAME)\n",
    "cf.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, Abhishek's `euclidean_distance` and `jaccard_distance` are the\n",
    "least significant features and may just add a noise to the model. The\n",
    "score received was 0.23158, which is  an improvement from Tet's\n",
    "0.24224. It seems that `vanilla_train` gives better results.\n",
    "\n",
    "![img](img/2017-06-04-1949-0.180881032724_quf_xgboost-feature_importance.png)\n",
    "\n",
    "-"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attempt at Ensembling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We ensemble our best model for the fourth time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predPath = \"ensembling/\"\n",
    "n = 9\n",
    "weights = {\n",
    "    # PB Scores are given above the file name\n",
    "    #0.23158\n",
    "    predPath + \"0.180881032724_quf_xgboost.csv\":n,\n",
    "    #0.24224\n",
    "    predPath + \"0.222026460989_tet_xgboost.csv\":n-1,\n",
    "    #0.24253\n",
    "    predPath + \"0.214863010529_kaf_xgboost.csv\":n-2,\n",
    "    #0.24360\n",
    "    predPath + \"0.2200_vav_198_0.46.csv\":n-3,\n",
    "    #0.24505\n",
    "    predPath + \"0.212772886818_samech_xgboost.csv\":n-4,\n",
    "    #0.24602\n",
    "    predPath + \"0.2346_gimmel_214_0.42.csv\":n-5,\n",
    "    #0.24785\n",
    "    predPath + \"0.319533794385_lamed_xgboost.csv\":n-6,\n",
    "    #0.24801\n",
    "    predPath + \"0.2364_dalet_242_0.32.csv\":n-7,\n",
    "    #0.24987\n",
    "    predPath + \"0.1339_nun_407_0.47.csv\":n-8\n",
    "}\n",
    "ensemble(weights, \"wam_ensemble_3.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, this boosts us to 0.20767, which is an improvement from the\n",
    "`wam_ensemble1.csv` result of 0.20882."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dilettante model meddling did not give satisfactory results. We\n",
    "need more features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NLTK Investigations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   measuring shares of hypernyms and lemmas, as well as linear and\n",
    "    smooth counts of synonyms\n",
    "-   cross path similarity is not computed due to the high computational cost\n",
    "\n",
    "We will plot how good the features are for the discrimination between\n",
    "dupes and non-dupes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "BASE_DIR = 'data/'\n",
    "TRAIN_DATA_FILENAME = \"vanilla_train\"\n",
    "TRAIN_DATA_FILE = BASE_DIR + TRAIN_DATA_FILENAME + '.csv'\n",
    "CUSTOM_FEATURES_TRAIN = 'custom/' + \\\n",
    "                        TRAIN_DATA_FILENAME + \\\n",
    "                        \"-nltk-train.csv\"\n",
    "df_train = pd.read_csv(TRAIN_DATA_FILE, encoding=\"utf-8\")\n",
    "nltk_features = pd.read_csv(CUSTOM_FEATURES_TRAIN, encoding=\"utf-8\")\n",
    "\n",
    "ix_is_dup = np.where(df_train['is_duplicate'] == 1)[0]\n",
    "ix_not_dup = np.where(df_train['is_duplicate'] == 0)[0]\n",
    "\n",
    "\n",
    "def plot_feature_graph(data, fname):\n",
    "    fig = plt.figure()\n",
    "    ax1 = plt.subplot2grid((3, 2), (0, 0), colspan=2)\n",
    "    ax2 = plt.subplot2grid((3, 2), (1, 0), colspan=2)\n",
    "    ax1.set_title('Distribution of %s' % fname, fontsize=16)\n",
    "    sns.distplot(data[fname], \n",
    "                 bins=50, \n",
    "                 ax=ax1)    \n",
    "    sns.distplot(data.loc[ix_is_dup][fname], \n",
    "                 bins=50, \n",
    "                 ax=ax2,\n",
    "                 label='duplicate')    \n",
    "    sns.distplot(data.loc[ix_not_dup][fname], \n",
    "                 bins=50, \n",
    "                 ax=ax2,\n",
    "                 label='non-duplicate')\n",
    "    ax2.legend(loc='upper right', prop={'size': 14})\n",
    "    plt.savefig(\"img/{}.png\".format(fname))\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plot_feature_graph(nltk_features,\"hypernyms_share\")\n",
    "plot_feature_graph(nltk_features,\"linear_synonyms_count\")\n",
    "plot_feature_graph(nltk_features,\"smooth_synonyms_count\")\n",
    "plot_feature_graph(nltk_features,\"lemmas_share\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![img](img/hypernyms_share.png)\n",
    "![img](img/linear_synonyms_count.png)\n",
    "![img](img/smooth_synonyms_count.png)\n",
    "![img](img/lemmas_share.png)\n",
    "\n",
    "We can see that `linear_synonyms_count` and `smooth_synonyms_count`\n",
    "allow almost no differentiation at all, so we exclude them from our\n",
    "models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Properties of words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Texts with similar contents might have similar syllables and character frequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "wordies_features = pd.read_csv('custom/' + TRAIN_DATA_FILENAME + \"-wordies-train.csv\", encoding=\"utf-8\")\n",
    "plot_feature_graph(wordies_features,\"character_freq\")\n",
    "plot_feature_graph(wordies_features,'syllable_similarity')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![img](img/character_freq.png)\n",
    "\n",
    "![img](img/syllable_similarity.png)\n",
    "Here we may discard the character frequencies, since they are almost\n",
    "completely overlapping."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing word and synonym properties"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Resh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from models.resh_model import Resh\n",
    "resh = Resh()\n",
    "resh.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Quf, with magic2 and kcore features included\n",
    "-   **Training data set**: vanilla\n",
    "\n",
    "![img](img/2017-06-05-0748-0.180005762885_resh_xgboost-feature_importance.png)\n",
    "\n",
    "-   Resh model scores 0.23214, which is worse than Quf, and the\n",
    "    `character_freq` seems to be important despite the massive\n",
    "    overlap. Probably `euclidean_distance`, `jaccard_distance` and\n",
    "    `character_freq` contribute to the decline of the model\n",
    "    generalisability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Counts Investigation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we try to measure several set statistics: Wagner-Fischer,\n",
    "MostFreqKSDF, Tversky index, weighted difference, stem frequencies,\n",
    "and stem sharing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from custom.counts_investigation import CountsFeatures\n",
    "c = CountsFeatures()\n",
    "c.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env_features = pd.read_csv('custom/' + TRAIN_DATA_FILENAME + \"-counts-train.csv\", encoding=\"utf-8\")\n",
    "plot_feature_graph(env_features,\"wagner_fischer\")\n",
    "plot_feature_graph(env_features,'wagner_fischer_reverse')\n",
    "plot_feature_graph(env_features,'most_freq_2_sdf')\n",
    "plot_feature_graph(env_features,'most_freq_2_sdf_reverse')\n",
    "plot_feature_graph(env_features,'most_freq_2_sdf_reverse')\n",
    "plot_feature_graph(env_features,'stems_freq')\n",
    "plot_feature_graph(env_features,'stems_share')\n",
    "plot_feature_graph(env_features,'stems_weighted_difference')\n",
    "plot_feature_graph(env_features,'stems_tversky_index')\n",
    "del env_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![img](img/wagner_fischer.png)\n",
    "\n",
    "![img](img/wagner_fischer_reverse.png)\n",
    "\n",
    " ![img](img/most_freq_2_sdf_reverse.png)\n",
    " ![img](img/most_freq_2_sdf_reverse.png)\n",
    " ![img](img/most_freq_2_sdf_reverse.png)\n",
    "  ![img](img/stems_freq.png)\n",
    "  ![img](img/stems_share.png)\n",
    "  ![img](img/stems_weighted_difference.png)\n",
    "  ![img](img/stems_tversky_index.png)\n",
    "Only the last four features seem to be worthy to be included in our next model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Environment Investigation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's represent each sentence as one continuous string of stemmed\n",
    "words and calculate several similarity statistics on them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from custom.environment_investigation import EnvFeatures\n",
    "e = EnvFeatures()\n",
    "e.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env_features = pd.read_csv('custom/' + TRAIN_DATA_FILENAME + \"-env-train.csv\", encoding=\"utf-8\").fillna(0)\n",
    "plot_feature_graph(env_features,'kendall_tau')\n",
    "plot_feature_graph(env_features,'kendall_p_value')\n",
    "plot_feature_graph(env_features,'string_similarity')\n",
    "plot_feature_graph(env_features,'subset_count')\n",
    "del env_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![img](img/kendall_tau.png)\n",
    "\n",
    "![img](img/kendall_p_value.png)\n",
    "\n",
    "![img](img/string_similarity.png)\n",
    "\n",
    "![img](img/subset_count.png)\n",
    "`kendall_p_value` and `string_similarity` seem to be worthy to be included into the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing word, synonym, counts and environment properties"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Shin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from shin_model import Shin\n",
    "s = Shin()\n",
    "s.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Resh, with counts and environment features included\n",
    "-   **Training data set**: vanilla\n",
    "\n",
    "![img](img/2017-06-06-0147-0.178758220484_shin_xgboost-feature_importance.png)\n",
    "The added features seem to be important and blend well with the rest,\n",
    "giving a score of 0.22808, which is an improvement from Quf and Resh."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Az Investigation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Questions with similar places mentioned seem to be more likely to\n",
    "share the same intent. We use the Movehub City Rankings [data](https://www.kaggle.com/blitzr/movehub-city-rankings) to fetch\n",
    "the country names together with some cities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from custom.investigation_az import AzFeatures\n",
    "az = AzFeatures()\n",
    "az.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "az_features = pd.read_csv('custom/' + TRAIN_DATA_FILENAME + \"-az-train.csv\", encoding=\"utf-8\").fillna(0.0)\n",
    "plot_feature_graph(az_features,'places_share')\n",
    "# All zeroes.\n",
    "#plot_feature_graph(az_features,'places_difference')\n",
    "plot_feature_graph(az_features,'places_prevalence')\n",
    "del az_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![img](img/places_share.png)\n",
    "\n",
    "![img](img/places_prevalence.png)\n",
    "There is some differentiation, but not that much &#x2013; not many questions mention any countries at all."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Buky Investigation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What can Word2Vec embeddings trained on Google News tell us?  We\n",
    "introduce multiple features, all based on the vectors corresponding to\n",
    "each word in the questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from custom.investigation_buky import BukyFeatures\n",
    "buky = BukyFeatures()\n",
    "buky.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collins & Duffy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We feed [Collins and Duffy's data](https://www.kaggle.com/c/quora-question-pairs/discussion/32334) in the hopes of retrieving some\n",
    "information affecting the duplicity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from custom.duffy import Duffy\n",
    "d = Duffy()\n",
    "d.build()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pagerank"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We compute pageranks, [inspired by @ZFTurbo](https://www.kaggle.com/zfturbo/pagerank-on-quora-feature-file-generator/code)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from custom import pagerank"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to combine the features and feed them to our model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tav"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tav_model import Tav\n",
    "t = Tav()\n",
    "t.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Shin, with the computed features included\n",
    "-   **Training data set**: vanilla\n",
    "\n",
    "![img](img/2017-06-06-2327-0.174476734017_tav_xgboost-feature_importance.png)\n",
    "Luckily, we have found features which are more important than the ones used previously. Tav scores us 0.20662."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With not much time under our belts, let's train something more\n",
    "lightweight than LSTM but hopefully more accurate than vanilla NN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sin_model import Sin\n",
    "s = Sin()\n",
    "s.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-   based on Khet (Convolutional1D with the custom features fed through a hidden layer)\n",
    "-   all the best features included\n",
    "-   trained for 5 total epochs due to time constraints, with a break\n",
    "    in-between to change the number of epochs\n",
    "-   **Training data set**: vanilla\n",
    "\n",
    "![img](img/2017-06-07-0229-sin_323_0.46-accuracy_dynamics.png)\n",
    "\n",
    "The model seems to be very stable, with some overfitting occuring by\n",
    "the end of the third epoch. Sin yields 0.22727, which is worse than\n",
    "Tav."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### WAM Ensemble 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While Sin is being evaluated, let's ensemble what we have so far and\n",
    "see what we get."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def ensemble(weights, output):\n",
    "    import glob\n",
    "    import pandas as pd\n",
    "\n",
    "    weighted_models = []\n",
    "    for file_name in glob.glob(predPath + '*.csv'):\n",
    "        df = pd.read_csv(file_name, encoding = \"utf-8\")\n",
    "        weighted_models.append((df['is_duplicate'],weights[file_name]))\n",
    "    \n",
    "    wam_df = df['is_duplicate'] * 0.0\n",
    "    \n",
    "    for model, weight in weighted_models:\n",
    "        wam_df += model * weight\n",
    "    wam_df /= sum(weights.values())    \n",
    "        \n",
    "    wam_df = pd.concat([df['test_id'], wam_df], axis=1)\n",
    "    wam_df.to_csv(output, index=False)\n",
    "predPath = \"ensembling/\"\n",
    "n = 9\n",
    "weights = {\n",
    "    # PB Scores are given above the file name\n",
    "    #0.20662\n",
    "    predPath + \"0.174476734017_tav_xgboost.csv\":n+5,\n",
    "    #0.20767\n",
    "    predPath + \"wam_ensemble_3.csv\":n+4,\n",
    "    #0.21736\n",
    "    predPath + \"gm_ensemble.csv\":n+3,\n",
    "    #0.22808\n",
    "    predPath + \"0.178758220484_shin_xgboost.csv\":n+2,\n",
    "    #0.23158\n",
    "    predPath + \"0.180881032724_quf_xgboost.csv\":n+1,\n",
    "    #0.23214\n",
    "    predPath + \"0.180005762885_resh_xgboost.csv\":n,\n",
    "    #0.24224\n",
    "    predPath + \"0.222026460989_tet_xgboost.csv\":n-1,\n",
    "    #0.24253\n",
    "    predPath + \"0.214863010529_kaf_xgboost.csv\":n-2,\n",
    "    #0.24360\n",
    "    predPath + \"0.2200_vav_198_0.46.csv\":n-3,\n",
    "    #0.24505\n",
    "    predPath + \"0.212772886818_samech_xgboost.csv\":n-4,\n",
    "    #0.24602\n",
    "    predPath + \"0.2346_gimmel_214_0.42.csv\":n-5,\n",
    "    #0.24785\n",
    "    predPath + \"0.319533794385_lamed_xgboost.csv\":n-6,\n",
    "    #0.24801\n",
    "    predPath + \"0.2364_dalet_242_0.32.csv\":n-7,\n",
    "    #0.24987\n",
    "    predPath + \"0.1339_nun_407_0.47.csv\":n-8\n",
    "}\n",
    "ensemble(weights, \"wam_ensemble_4.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ensembling gives us an improvement of more than 0.005."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### WAM Ensemble 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's apply the trick which we used in our first ensembling.\n",
    "First, let's compute the GM ensemble of all the models which scored under 0.25."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import glob\n",
    "import pandas as pd\n",
    "from scipy.stats.mstats import gmean\n",
    "\n",
    "predPath = \"ensembling/\"\n",
    "models = []\n",
    "\n",
    "for file_name in glob.glob(predPath + '*.csv'):\n",
    "    df = pd.read_csv(file_name, encoding = \"utf-8\")\n",
    "    models.append(df['is_duplicate'])\n",
    "\n",
    "total = pd.concat(models, axis=1)\n",
    "gm_df = pd.DataFrame()\n",
    "gm_df['is_duplicate'] = gmean(total, axis=1)\n",
    "gm_df['test_id'] = df['test_id']\n",
    "gm_df.to_csv(\"gm_ensemble_1.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we combine the three best models and take a weighted mean:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def ensemble(weights, output):\n",
    "    import glob\n",
    "    import pandas as pd\n",
    "\n",
    "    weighted_models = []\n",
    "    for file_name in glob.glob(predPath + '*.csv'):\n",
    "        df = pd.read_csv(file_name, encoding = \"utf-8\")\n",
    "        weighted_models.append((df['is_duplicate'],weights[file_name]))\n",
    "    \n",
    "    wam_df = df['is_duplicate'] * 0.0\n",
    "    \n",
    "    for model, weight in weighted_models:\n",
    "        wam_df += model * weight\n",
    "    wam_df /= sum(weights.values())    \n",
    "        \n",
    "    wam_df = pd.concat([df['test_id'], wam_df], axis=1)\n",
    "    wam_df.to_csv(output, index=False)\n",
    "predPath = \"ensembling/\"\n",
    "n = 3\n",
    "weights = {\n",
    "    # PB Scores are given above the file name\n",
    "    #0.20662\n",
    "    predPath + \"0.174476734017_tav_xgboost.csv\":n,\n",
    "    #0.20767\n",
    "    predPath + \"wam_ensemble_4.csv\":n-1,\n",
    "    #0.21067\n",
    "    predPath + \"gm_ensemble_1.csv\":n-2\n",
    "}\n",
    "ensemble(weights, \"wam_ensemble_5.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final ensemble scores 0.19638 on the public LB and 0.20098 on the private LB."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "QQP was a superb experience for me &#x2013; data, tips and knowledge\n",
    "shared by the Kaggle community and all the other researchers working\n",
    "on NLP were indispensable in the present investigation of the data\n",
    "set. Given the fact that the only experience of machine learning I had\n",
    "before was limited to using automatic translators, the competition\n",
    "turned out to give valuable lessons in natural language processing and\n",
    "powerful prediction methods.\n",
    "Although the final score is not the best, it evidently shows that a\n",
    "dilettante with some free time, programming experience and wonderful\n",
    "receptive community can indeed beat the three quarters of\n",
    "participants.\n",
    "Key aspects of analysis which I would like to\n",
    "improve in the future include the following:\n",
    "\n",
    "-   some graphs look awful and not particularly informative, especially\n",
    "    the diagrams on the feature importance &#x2013; using violin or scatter\n",
    "    plots might be more useful\n",
    "-   there must be a way to extract features found by different neural\n",
    "    networks and inject them into the new model to boost accuracy\n",
    "-   a diverse range of models should be tested &#x2013; only three frameworks\n",
    "    (XGBoost, Keras NN, Scikit stacking) were tested in the present\n",
    "    analysis &#x2013; and looking at, for example, H2o and LightGBM may help\n",
    "-   methodical approach to tweaking the model parameters should be\n",
    "    adopted &#x2013; predictions here were fine-tuned based on the shaky\n",
    "    understanding of the skimmed documentation, and a lot of them were\n",
    "    added in bulk\n",
    "\n",
    "There is a lot to learn, produce and share."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  },
  "org": null
 },
 "nbformat": 4,
 "nbformat_minor": 1
}