t81_558_class11_text_nlp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 11: Natural Language Processing and Speech Recognition**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), School of Engineering and Applied Science, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module Video Material\n",
    "\n",
    "Main video lecture:\n",
    "\n",
    "* [Module 11, Part 1: Chatbots and NLP](https://www.youtube.com/watch?v=bv_iVVrlfbU)\n",
    "* [Module 11, Part 2: End to End Networks](https://www.youtube.com/watch?v=qN9hHlZKIL4)\n",
    "* [Module 11, Part 3: Word2Vec](https://www.youtube.com/watch?v=Ae3GVw5nTYU)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Helpful Functions\n",
    "\n",
    "You will see these at the top of every module.  These are simply a set of reusable functions that we will make use of.  Each of them will be explained as the semester progresses.  They are explained in greater detail as the course progresses.  Class 4 contains a complete overview of these functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import base64\n",
    "import os\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import requests\n",
    "from sklearn import preprocessing\n",
    "\n",
    "\n",
    "# Encode text values to dummy variables(i.e. [1,0,0],[0,1,0],[0,0,1] for red,green,blue)\n",
    "def encode_text_dummy(df, name):\n",
    "    dummies = pd.get_dummies(df[name])\n",
    "    for x in dummies.columns:\n",
    "        dummy_name = f\"{name}-{x}\"\n",
    "        df[dummy_name] = dummies[x]\n",
    "    df.drop(name, axis=1, inplace=True)\n",
    "\n",
    "\n",
    "# Encode text values to a single dummy variable.  The new columns (which do not replace the old) will have a 1\n",
    "# at every location where the original column (name) matches each of the target_values.  One column is added for\n",
    "# each target value.\n",
    "def encode_text_single_dummy(df, name, target_values):\n",
    "    for tv in target_values:\n",
    "        l = list(df[name].astype(str))\n",
    "        l = [1 if str(x) == str(tv) else 0 for x in l]\n",
    "        name2 = f\"{name}-{tv}\"\n",
    "        df[name2] = l\n",
    "\n",
    "\n",
    "# Encode text values to indexes(i.e. [1],[2],[3] for red,green,blue).\n",
    "def encode_text_index(df, name):\n",
    "    le = preprocessing.LabelEncoder()\n",
    "    df[name] = le.fit_transform(df[name])\n",
    "    return le.classes_\n",
    "\n",
    "\n",
    "# Encode a numeric column as zscores\n",
    "def encode_numeric_zscore(df, name, mean=None, sd=None):\n",
    "    if mean is None:\n",
    "        mean = df[name].mean()\n",
    "\n",
    "    if sd is None:\n",
    "        sd = df[name].std()\n",
    "\n",
    "    df[name] = (df[name] - mean) / sd\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the median\n",
    "def missing_median(df, name):\n",
    "    med = df[name].median()\n",
    "    df[name] = df[name].fillna(med)\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the default\n",
    "def missing_default(df, name, default_value):\n",
    "    df[name] = df[name].fillna(default_value)\n",
    "\n",
    "\n",
    "# Convert a Pandas dataframe to the x,y inputs that TensorFlow needs\n",
    "def to_xy(df, target):\n",
    "    result = []\n",
    "    for x in df.columns:\n",
    "        if x != target:\n",
    "            result.append(x)\n",
    "    # find out the type of the target column.  Is it really this hard? :(\n",
    "    target_type = df[target].dtypes\n",
    "    target_type = target_type[0] if hasattr(\n",
    "        target_type, '__iter__') else target_type\n",
    "    # Encode to int for classification, float otherwise. TensorFlow likes 32 bits.\n",
    "    if target_type in (np.int64, np.int32):\n",
    "        # Classification\n",
    "        dummies = pd.get_dummies(df[target])\n",
    "        return df[result].values.astype(np.float32), dummies.values.astype(np.float32)\n",
    "    # Regression\n",
    "    return df[result].values.astype(np.float32), df[[target]].values.astype(np.float32)\n",
    "\n",
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return f\"{h}:{m:>02}:{s:>05.2f}\"\n",
    "\n",
    "\n",
    "# Regression chart.\n",
    "def chart_regression(pred, y, sort=True):\n",
    "    t = pd.DataFrame({'pred': pred, 'y': y.flatten()})\n",
    "    if sort:\n",
    "        t.sort_values(by=['y'], inplace=True)\n",
    "    plt.plot(t['y'].tolist(), label='expected')\n",
    "    plt.plot(t['pred'].tolist(), label='prediction')\n",
    "    plt.ylabel('output')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "# Remove all rows where the specified column is +/- sd standard deviations\n",
    "def remove_outliers(df, name, sd):\n",
    "    drop_rows = df.index[(np.abs(df[name] - df[name].mean())\n",
    "                          >= (sd * df[name].std()))]\n",
    "    df.drop(drop_rows, axis=0, inplace=True)\n",
    "\n",
    "\n",
    "# Encode a column to a range between normalized_low and normalized_high.\n",
    "def encode_numeric_range(df, name, normalized_low=-1, normalized_high=1,\n",
    "                         data_low=None, data_high=None):\n",
    "    if data_low is None:\n",
    "        data_low = min(df[name])\n",
    "        data_high = max(df[name])\n",
    "\n",
    "    df[name] = ((df[name] - data_low) / (data_high - data_low)) \\\n",
    "        * (normalized_high - normalized_low) + normalized_low\n",
    "\n",
    "\n",
    "# This function submits an assignment.  You can submit an assignment as much as you like, only the final\n",
    "# submission counts.  The paramaters are as follows:\n",
    "# data - Pandas dataframe output.\n",
    "# key - Your student key that was emailed to you.\n",
    "# no - The assignment class number, should be 1 through 1.\n",
    "# source_file - The full path to your Python or IPYNB file.  This must have \"_class1\" as part of its name.  \n",
    "# .             The number must match your assignment number.  For example \"_class2\" for class assignment #2.\n",
    "def submit(data,key,no,source_file=None):\n",
    "    if source_file is None and '__file__' not in globals(): raise Exception('Must specify a filename when a Jupyter notebook.')\n",
    "    if source_file is None: source_file = __file__\n",
    "    suffix = '_class{}'.format(no)\n",
    "    if suffix not in source_file: raise Exception('{} must be part of the filename.'.format(suffix))\n",
    "    with open(source_file, \"rb\") as image_file:\n",
    "        encoded_python = base64.b64encode(image_file.read()).decode('ascii')\n",
    "    ext = os.path.splitext(source_file)[-1].lower()\n",
    "    if ext not in ['.ipynb','.py']: raise Exception(\"Source file is {} must be .py or .ipynb\".format(ext))\n",
    "    r = requests.post(\"https://api.heatonresearch.com/assignment-submit\",\n",
    "        headers={'x-api-key':key}, json={'csv':base64.b64encode(data.to_csv(index=False).encode('ascii')).decode(\"ascii\"),\n",
    "        'assignment': no, 'ext':ext, 'py':encoded_python})\n",
    "    if r.status_code == 200:\n",
    "        print(\"Success: {}\".format(r.text))\n",
    "    else: print(\"Failure: {}\".format(r.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chat Bots\n",
    "\n",
    "Using the above code you can create your own primitive chat bots.  A some what famous video on Youtube from Cornell University shows what happens [when two chat bots converse](https://www.youtube.com/watch?v=WnzlbyTZsQY).  Other interesting chat bot type technology:\n",
    "\n",
    "* [CleverBot](http://www.cleverbot.com/)\n",
    "* [Computer Science Paper Generator](https://pdos.csail.mit.edu/archive/scigen/)\n",
    "\n",
    "### Other Resources\n",
    "* [Word Net](http://wordnet.princeton.edu/)\n",
    "* [bAbI Datasets](https://research.fb.com/downloads/babi/)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# End-To-End Memory Networks\n",
    "\n",
    "The origional source papers for End-to-End Memory Networks:\n",
    "\n",
    "* Jason Weston, Antoine Bordes, Sumit Chopra, Tomas Mikolov, Alexander M. Rush, [\"Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks\"](http://arxiv.org/abs/1502.05698)\n",
    "* Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, Rob Fergus, [\"End-To-End Memory Networks\"](http://arxiv.org/abs/1503.08895)\n",
    "\n",
    "Other useful links for End-To-End Memory Networks\n",
    "\n",
    "* [bAbI Datasets](https://research.fb.com/downloads/babi/)\n",
    "* [Keras End-To-End Memory Networks](https://github.com/fchollet/keras/blob/master/examples/babi_memnn.py)\n",
    "* [Online JavaScript Demo of End-to-End Memory Networks](http://yerevann.com/dmn-ui/#/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Imports and Utility Functions\n",
    "\n",
    "The following imports are needed to create the end-to-end memory network. Neither Keras nor TensorFlow directly support End-to-End Memory Networks (yet), so it is necessary to create them using existing tools.  Several functions are needed defined here to read the bAbI dataset that we are using to train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential, Model\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.layers import Input, Activation, Dense, Permute, Dropout, add, dot, concatenate\n",
    "from keras.layers import LSTM\n",
    "from keras.utils.data_utils import get_file\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.models import load_model\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn import metrics\n",
    "from functools import reduce\n",
    "import pickle\n",
    "import tarfile\n",
    "import numpy as np\n",
    "import re\n",
    "import os\n",
    "import time\n",
    "\n",
    "\n",
    "def tokenize(sent):\n",
    "    '''Return the tokens of a sentence including punctuation.\n",
    "    >>> tokenize('Bob dropped the apple. Where is the apple?')\n",
    "    ['Bob', 'dropped', 'the', 'apple', '.', 'Where', 'is', 'the', 'apple', '?']\n",
    "    '''\n",
    "    return [x.strip() for x in re.split('(\\W+)', sent) if x.strip()]\n",
    "\n",
    "\n",
    "def parse_stories(lines, only_supporting=False):\n",
    "    '''Parse stories provided in the bAbi tasks format\n",
    "    If only_supporting is true, only the sentences\n",
    "    that support the answer are kept.\n",
    "    '''\n",
    "    data = []\n",
    "    story = []\n",
    "    for line in lines:\n",
    "        line = line.decode('utf-8').strip()\n",
    "        nid, line = line.split(' ', 1)\n",
    "        nid = int(nid)\n",
    "        if nid == 1:\n",
    "            story = []\n",
    "        if '\\t' in line:\n",
    "            q, a, supporting = line.split('\\t')\n",
    "            q = tokenize(q)\n",
    "            substory = None\n",
    "            if only_supporting:\n",
    "                # Only select the related substory\n",
    "                supporting = map(int, supporting.split())\n",
    "                substory = [story[i - 1] for i in supporting]\n",
    "            else:\n",
    "                # Provide all the substories\n",
    "                substory = [x for x in story if x]\n",
    "            data.append((substory, q, a))\n",
    "            story.append('')\n",
    "        else:\n",
    "            sent = tokenize(line)\n",
    "            story.append(sent)\n",
    "    return data\n",
    "\n",
    "\n",
    "def get_stories(f, only_supporting=False, max_length=None):\n",
    "    '''Given a file name, read the file,\n",
    "    retrieve the stories,\n",
    "    and then convert the sentences into a single story.\n",
    "    If max_length is supplied,\n",
    "    any stories longer than max_length tokens will be discarded.\n",
    "    '''\n",
    "    data = parse_stories(f.readlines(), only_supporting=only_supporting)\n",
    "    flatten = lambda data: reduce(lambda x, y: x + y, data)\n",
    "    data = [(flatten(story), q, answer) for story, q, answer in data if not max_length or len(flatten(story)) < max_length]\n",
    "    return data\n",
    "\n",
    "\n",
    "def vectorize_stories(data):\n",
    "    inputs, queries, answers = [], [], []\n",
    "    for story, query, answer in data:\n",
    "        inputs.append([word_idx[w] for w in story])\n",
    "        queries.append([word_idx[w] for w in query])\n",
    "        answers.append(word_idx[answer])\n",
    "    return (pad_sequences(inputs, maxlen=story_maxlen),\n",
    "            pad_sequences(queries, maxlen=query_maxlen),\n",
    "            np.array(answers))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Getting the Data\n",
    "\n",
    "The data is downloaded from the Internet, if needed.\n",
    "\n",
    "As you can see (below), this dataset contains stories and questions about those stories.  The computer is not learning these specific stories below, but rather how to read a story and answer a question about that story.  Consider the first story \"Mary moved to the bathroom. John went to the hallway.\" the computer is not learning that Mary is in the bathroom or John is in the hallway, this changes per story.  Rather, the computer is learning to parse the story and extract information about individual people and their locations.\n",
    "\n",
    "The computer is learning to read, at least in a limited sense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/text-datasets/babi_tasks_1-20_v1-2.tar.gz\n",
      "11747328/11745123 [==============================] - 6s 1us/step\n",
      "Extracting stories for the challenge: single_supporting_fact_10k\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    path = get_file('babi-tasks-v1-2.tar.gz', origin='https://s3.amazonaws.com/text-datasets/babi_tasks_1-20_v1-2.tar.gz')\n",
    "except:\n",
    "    print('Error downloading dataset, please download it manually:\\n'\n",
    "          '$ wget http://www.thespermwhale.com/jaseweston/babi/tasks_1-20_v1-2.tar.gz\\n'\n",
    "          '$ mv tasks_1-20_v1-2.tar.gz ~/.keras/datasets/babi-tasks-v1-2.tar.gz')\n",
    "    raise\n",
    "tar = tarfile.open(path)\n",
    "\n",
    "challenges = {\n",
    "    # QA1 with 10,000 samples\n",
    "    'single_supporting_fact_10k': 'tasks_1-20_v1-2/en-10k/qa1_single-supporting-fact_{}.txt',\n",
    "    # QA2 with 10,000 samples\n",
    "    'two_supporting_facts_10k': 'tasks_1-20_v1-2/en-10k/qa2_two-supporting-facts_{}.txt',\n",
    "}\n",
    "challenge_type = 'single_supporting_fact_10k'\n",
    "challenge = challenges[challenge_type]\n",
    "\n",
    "print('Extracting stories for the challenge:', challenge_type)\n",
    "train_stories = get_stories(tar.extractfile(challenge.format('train')))\n",
    "test_stories = get_stories(tar.extractfile(challenge.format('test')))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Story: Mary moved to the bathroom . John went to the hallway .\n",
      "Query: Where is Mary ?\n",
      "Answer: bathroom\n",
      "---\n",
      "Story: Mary moved to the bathroom . John went to the hallway . Daniel went back to the hallway . Sandra moved to the garden .\n",
      "Query: Where is Daniel ?\n",
      "Answer: hallway\n",
      "---\n",
      "Story: Mary moved to the bathroom . John went to the hallway . Daniel went back to the hallway . Sandra moved to the garden . John moved to the office . Sandra journeyed to the bathroom .\n",
      "Query: Where is Daniel ?\n",
      "Answer: hallway\n",
      "---\n",
      "Story: Mary moved to the bathroom . John went to the hallway . Daniel went back to the hallway . Sandra moved to the garden . John moved to the office . Sandra journeyed to the bathroom . Mary moved to the hallway . Daniel travelled to the office .\n",
      "Query: Where is Daniel ?\n",
      "Answer: office\n",
      "---\n",
      "Story: Mary moved to the bathroom . John went to the hallway . Daniel went back to the hallway . Sandra moved to the garden . John moved to the office . Sandra journeyed to the bathroom . Mary moved to the hallway . Daniel travelled to the office . John went back to the garden . John moved to the bedroom .\n",
      "Query: Where is Sandra ?\n",
      "Answer: bathroom\n",
      "---\n"
     ]
    }
   ],
   "source": [
    "# See what the data looks like\n",
    "\n",
    "for i in range(5):\n",
    "    print(\"Story: {}\".format(' '.join(train_stories[i][0])))\n",
    "    print(\"Query: {}\".format(' '.join(train_stories[i][1])))\n",
    "    print(\"Answer: {}\".format(train_stories[i][2]))\n",
    "    print(\"---\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Building the Vocabulary\n",
    "\n",
    "This type of neural network can only deal with a set vocabulary.  The words are indexed and each becomes a number.  Words not in the training vocabulary will not be recognized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-\n",
      "Vocab size: 22 unique words\n",
      "Story max length: 68 words\n",
      "Query max length: 4 words\n",
      "Number of training stories: 10000\n",
      "Number of test stories: 1000\n",
      "-\n",
      "Here's what a \"story\" tuple looks like (input, query, answer):\n",
      "(['Mary', 'moved', 'to', 'the', 'bathroom', '.', 'John', 'went', 'to', 'the', 'hallway', '.'], ['Where', 'is', 'Mary', '?'], 'bathroom')\n",
      "-\n",
      "(0, '.')\n",
      "(1, '?')\n",
      "(2, 'Daniel')\n",
      "(3, 'John')\n",
      "(4, 'Mary')\n",
      "(5, 'Sandra')\n",
      "(6, 'Where')\n",
      "(7, 'back')\n",
      "(8, 'bathroom')\n",
      "(9, 'bedroom')\n",
      "(10, 'garden')\n",
      "(11, 'hallway')\n",
      "(12, 'is')\n",
      "(13, 'journeyed')\n",
      "(14, 'kitchen')\n",
      "(15, 'moved')\n",
      "(16, 'office')\n",
      "(17, 'the')\n",
      "(18, 'to')\n",
      "(19, 'travelled')\n",
      "(20, 'went')\n"
     ]
    }
   ],
   "source": [
    "vocab = set()\n",
    "for story, q, answer in train_stories + test_stories:\n",
    "    vocab |= set(story + q + [answer])\n",
    "vocab = sorted(vocab)\n",
    "\n",
    "# Reserve 0 for masking via pad_sequences\n",
    "vocab_size = len(vocab) + 1\n",
    "story_maxlen = max(map(len, (x for x, _, _ in train_stories + test_stories)))\n",
    "query_maxlen = max(map(len, (x for _, x, _ in train_stories + test_stories)))\n",
    "\n",
    "print('-')\n",
    "print('Vocab size:', vocab_size, 'unique words')\n",
    "print('Story max length:', story_maxlen, 'words')\n",
    "print('Query max length:', query_maxlen, 'words')\n",
    "print('Number of training stories:', len(train_stories))\n",
    "print('Number of test stories:', len(test_stories))\n",
    "print('-')\n",
    "print('Here\\'s what a \"story\" tuple looks like (input, query, answer):')\n",
    "print(train_stories[0])\n",
    "print('-')\n",
    "\n",
    "\n",
    "for s in list(enumerate(vocab)):\n",
    "    print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Building the Training and Test Data\n",
    "\n",
    "The training data that is actually sent to the neural network is the vectorized representation of the sentences.  Each word is replaced by its vocab number.  Additionally there are two parts to the input (x) data: story and query.  The answer (x) is a always a single vocab word number.  This is a classification network.  Any of the vocab words could potentially be the answer.  Stories can be at most 68 words and questions at most 4.  Both of these limits are automatically determined from the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vectorizing the word sequences...\n",
      "-\n",
      "inputs: integer tensor of shape (samples, max_length)\n",
      "inputs_train shape: (10000, 68)\n",
      "inputs_test shape: (1000, 68)\n",
      "-\n",
      "queries: integer tensor of shape (samples, max_length)\n",
      "queries_train shape: (10000, 4)\n",
      "queries_test shape: (1000, 4)\n",
      "-\n",
      "answers: binary (1 or 0) tensor of shape (samples, vocab_size)\n",
      "answers_train shape: (10000,)\n",
      "answers_test shape: (1000,)\n",
      "-\n"
     ]
    }
   ],
   "source": [
    "print('Vectorizing the word sequences...')\n",
    "word_idx = dict((c, i + 1) for i, c in enumerate(vocab))\n",
    "inputs_train, queries_train, answers_train = vectorize_stories(train_stories)\n",
    "inputs_test, queries_test, answers_test = vectorize_stories(test_stories)\n",
    "\n",
    "print('-')\n",
    "print('inputs: integer tensor of shape (samples, max_length)')\n",
    "print('inputs_train shape:', inputs_train.shape)\n",
    "print('inputs_test shape:', inputs_test.shape)\n",
    "print('-')\n",
    "print('queries: integer tensor of shape (samples, max_length)')\n",
    "print('queries_train shape:', queries_train.shape)\n",
    "print('queries_test shape:', queries_test.shape)\n",
    "print('-')\n",
    "print('answers: binary (1 or 0) tensor of shape (samples, vocab_size)')\n",
    "print('answers_train shape:', answers_train.shape)\n",
    "print('answers_test shape:', answers_test.shape)\n",
    "print('-')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Story (x): [ 0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "  0  0  0  0  0  0  0  0  5 16 19 18  9  1  4 21 19 18 12  1]\n",
      "Question (x): [ 7 13  5  2]\n",
      "Answer: 9\n"
     ]
    }
   ],
   "source": [
    "# See individual training element.\n",
    "\n",
    "print(\"Story (x): {}\".format(inputs_train[0]))\n",
    "print(\"Question (x): {}\".format(queries_train[0]))\n",
    "print(\"Answer: {}\".format(answers_train[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compile the Neural Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Compiling...\n",
      "Done.\n"
     ]
    }
   ],
   "source": [
    "print('Compiling...')\n",
    "\n",
    "# placeholders\n",
    "input_sequence = Input((story_maxlen,))\n",
    "question = Input((query_maxlen,))\n",
    "\n",
    "# encoders\n",
    "# embed the input sequence into a sequence of vectors\n",
    "input_encoder_m = Sequential()\n",
    "input_encoder_m.add(Embedding(input_dim=vocab_size,\n",
    "                              output_dim=64))\n",
    "input_encoder_m.add(Dropout(0.3))\n",
    "# output: (samples, story_maxlen, embedding_dim)\n",
    "\n",
    "# embed the input into a sequence of vectors of size query_maxlen\n",
    "input_encoder_c = Sequential()\n",
    "input_encoder_c.add(Embedding(input_dim=vocab_size,\n",
    "                              output_dim=query_maxlen))\n",
    "input_encoder_c.add(Dropout(0.3))\n",
    "# output: (samples, story_maxlen, query_maxlen)\n",
    "\n",
    "# embed the question into a sequence of vectors\n",
    "question_encoder = Sequential()\n",
    "question_encoder.add(Embedding(input_dim=vocab_size,\n",
    "                               output_dim=64,\n",
    "                               input_length=query_maxlen))\n",
    "question_encoder.add(Dropout(0.3))\n",
    "# output: (samples, query_maxlen, embedding_dim)\n",
    "\n",
    "# encode input sequence and questions (which are indices)\n",
    "# to sequences of dense vectors\n",
    "input_encoded_m = input_encoder_m(input_sequence)\n",
    "input_encoded_c = input_encoder_c(input_sequence)\n",
    "question_encoded = question_encoder(question)\n",
    "\n",
    "# compute a 'match' between the first input vector sequence\n",
    "# and the question vector sequence\n",
    "# shape: `(samples, story_maxlen, query_maxlen)`\n",
    "match = dot([input_encoded_m, question_encoded], axes=(2, 2))\n",
    "match = Activation('softmax')(match)\n",
    "\n",
    "# add the match matrix with the second input vector sequence\n",
    "response = add([match, input_encoded_c])  # (samples, story_maxlen, query_maxlen)\n",
    "response = Permute((2, 1))(response)  # (samples, query_maxlen, story_maxlen)\n",
    "\n",
    "# concatenate the match matrix with the question vector sequence\n",
    "answer = concatenate([response, question_encoded])\n",
    "\n",
    "# the original paper uses a matrix multiplication for this reduction step.\n",
    "# we choose to use a RNN instead.\n",
    "answer = LSTM(32)(answer)  # (samples, 32)\n",
    "\n",
    "# one regularization layer -- more would probably be needed.\n",
    "answer = Dropout(0.3)(answer)\n",
    "answer = Dense(vocab_size)(answer)  # (samples, vocab_size)\n",
    "# we output a probability distribution over the vocabulary\n",
    "answer = Activation('softmax')(answer)\n",
    "\n",
    "# build the final model\n",
    "model = Model([input_sequence, question], answer)\n",
    "model.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "print(\"Done.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train the Neural Network\n",
    "\n",
    "It will take some time (probably up to 1/2 hour) to train this network on a CPU.  The network is saved.  If you've previously saved the neural network, you can skip this step and load the neural network in the next step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 10000 samples, validate on 1000 samples\n",
      "Epoch 1/120\n",
      "10000/10000 [==============================] - 4s 364us/step - loss: 1.9371 - acc: 0.1704 - val_loss: 1.7936 - val_acc: 0.1710\n",
      "Epoch 2/120\n",
      "10000/10000 [==============================] - 3s 300us/step - loss: 1.7189 - acc: 0.2442 - val_loss: 1.6447 - val_acc: 0.2790\n",
      "Epoch 3/120\n",
      "10000/10000 [==============================] - 3s 331us/step - loss: 1.6440 - acc: 0.2982 - val_loss: 1.5707 - val_acc: 0.3710\n",
      "Epoch 4/120\n",
      "10000/10000 [==============================] - 3s 340us/step - loss: 1.5479 - acc: 0.3660 - val_loss: 1.4928 - val_acc: 0.3680\n",
      "Epoch 5/120\n",
      "10000/10000 [==============================] - 3s 265us/step - loss: 1.5126 - acc: 0.3817 - val_loss: 1.4806 - val_acc: 0.4050\n",
      "Epoch 6/120\n",
      "10000/10000 [==============================] - 2s 236us/step - loss: 1.4904 - acc: 0.3978 - val_loss: 1.4607 - val_acc: 0.4110\n",
      "Epoch 7/120\n",
      "10000/10000 [==============================] - 2s 238us/step - loss: 1.4813 - acc: 0.4044 - val_loss: 1.4465 - val_acc: 0.4320\n",
      "Epoch 8/120\n",
      "10000/10000 [==============================] - 2s 233us/step - loss: 1.4523 - acc: 0.4278 - val_loss: 1.4110 - val_acc: 0.4620\n",
      "Epoch 9/120\n",
      "10000/10000 [==============================] - 3s 264us/step - loss: 1.4149 - acc: 0.4446 - val_loss: 1.3629 - val_acc: 0.4700\n",
      "Epoch 10/120\n",
      "10000/10000 [==============================] - 3s 268us/step - loss: 1.3857 - acc: 0.4548 - val_loss: 1.3466 - val_acc: 0.4600\n",
      "Epoch 11/120\n",
      "10000/10000 [==============================] - 3s 338us/step - loss: 1.3657 - acc: 0.4646 - val_loss: 1.3431 - val_acc: 0.4830\n",
      "Epoch 12/120\n",
      "10000/10000 [==============================] - 4s 383us/step - loss: 1.3585 - acc: 0.4688 - val_loss: 1.3179 - val_acc: 0.4820\n",
      "Epoch 13/120\n",
      "10000/10000 [==============================] - 4s 376us/step - loss: 1.3353 - acc: 0.4830 - val_loss: 1.2918 - val_acc: 0.5080\n",
      "Epoch 14/120\n",
      "10000/10000 [==============================] - 3s 304us/step - loss: 1.3206 - acc: 0.4924 - val_loss: 1.2758 - val_acc: 0.5070\n",
      "Epoch 15/120\n",
      "10000/10000 [==============================] - 3s 256us/step - loss: 1.3040 - acc: 0.4949 - val_loss: 1.2761 - val_acc: 0.5090\n",
      "Epoch 16/120\n",
      "10000/10000 [==============================] - 2s 248us/step - loss: 1.2797 - acc: 0.5023 - val_loss: 1.2440 - val_acc: 0.5010\n",
      "Epoch 17/120\n",
      "10000/10000 [==============================] - 2s 241us/step - loss: 1.2680 - acc: 0.5016 - val_loss: 1.2485 - val_acc: 0.5160\n",
      "Epoch 18/120\n",
      "10000/10000 [==============================] - 2s 249us/step - loss: 1.2554 - acc: 0.5023 - val_loss: 1.2346 - val_acc: 0.5080\n",
      "Epoch 19/120\n",
      "10000/10000 [==============================] - 2s 224us/step - loss: 1.2472 - acc: 0.5053 - val_loss: 1.2264 - val_acc: 0.5160\n",
      "Epoch 20/120\n",
      "10000/10000 [==============================] - 2s 227us/step - loss: 1.2239 - acc: 0.5125 - val_loss: 1.2101 - val_acc: 0.5110\n",
      "Epoch 21/120\n",
      "10000/10000 [==============================] - 2s 233us/step - loss: 1.2136 - acc: 0.5186 - val_loss: 1.1851 - val_acc: 0.5190\n",
      "Epoch 22/120\n",
      "10000/10000 [==============================] - 2s 244us/step - loss: 1.2110 - acc: 0.5132 - val_loss: 1.2197 - val_acc: 0.5220\n",
      "Epoch 23/120\n",
      "10000/10000 [==============================] - 2s 232us/step - loss: 1.2042 - acc: 0.5196 - val_loss: 1.1844 - val_acc: 0.5170\n",
      "Epoch 24/120\n",
      "10000/10000 [==============================] - 2s 242us/step - loss: 1.2077 - acc: 0.5141 - val_loss: 1.1926 - val_acc: 0.5150\n",
      "Epoch 25/120\n",
      "10000/10000 [==============================] - 2s 244us/step - loss: 1.1938 - acc: 0.5217 - val_loss: 1.1800 - val_acc: 0.5230\n",
      "Epoch 26/120\n",
      "10000/10000 [==============================] - 2s 225us/step - loss: 1.1941 - acc: 0.5178 - val_loss: 1.1796 - val_acc: 0.5150\n",
      "Epoch 27/120\n",
      "10000/10000 [==============================] - 2s 224us/step - loss: 1.1891 - acc: 0.5183 - val_loss: 1.1737 - val_acc: 0.5140\n",
      "Epoch 28/120\n",
      "10000/10000 [==============================] - 2s 231us/step - loss: 1.1796 - acc: 0.5206 - val_loss: 1.1773 - val_acc: 0.5250\n",
      "Epoch 29/120\n",
      "10000/10000 [==============================] - 2s 223us/step - loss: 1.1723 - acc: 0.5223 - val_loss: 1.1903 - val_acc: 0.5080\n",
      "Epoch 30/120\n",
      "10000/10000 [==============================] - 2s 226us/step - loss: 1.1617 - acc: 0.5246 - val_loss: 1.1873 - val_acc: 0.5140\n",
      "Epoch 31/120\n",
      "10000/10000 [==============================] - 3s 283us/step - loss: 1.1624 - acc: 0.5247 - val_loss: 1.1754 - val_acc: 0.4990\n",
      "Epoch 32/120\n",
      "10000/10000 [==============================] - 3s 266us/step - loss: 1.1627 - acc: 0.5216 - val_loss: 1.1679 - val_acc: 0.5140\n",
      "Epoch 33/120\n",
      "10000/10000 [==============================] - 3s 268us/step - loss: 1.1588 - acc: 0.5284 - val_loss: 1.1763 - val_acc: 0.5140\n",
      "Epoch 34/120\n",
      "10000/10000 [==============================] - 2s 235us/step - loss: 1.1564 - acc: 0.5307 - val_loss: 1.1606 - val_acc: 0.5130\n",
      "Epoch 35/120\n",
      "10000/10000 [==============================] - 2s 230us/step - loss: 1.1365 - acc: 0.5404 - val_loss: 1.1590 - val_acc: 0.5240\n",
      "Epoch 36/120\n",
      "10000/10000 [==============================] - 2s 221us/step - loss: 1.1374 - acc: 0.5355 - val_loss: 1.1630 - val_acc: 0.5360\n",
      "Epoch 37/120\n",
      "10000/10000 [==============================] - 2s 220us/step - loss: 1.1277 - acc: 0.5459 - val_loss: 1.1332 - val_acc: 0.5460\n",
      "Epoch 38/120\n",
      "10000/10000 [==============================] - 2s 219us/step - loss: 1.0910 - acc: 0.5630 - val_loss: 1.0831 - val_acc: 0.5740\n",
      "Epoch 39/120\n",
      "10000/10000 [==============================] - 2s 220us/step - loss: 1.0372 - acc: 0.5959 - val_loss: 0.9797 - val_acc: 0.6340\n",
      "Epoch 40/120\n",
      "10000/10000 [==============================] - 2s 218us/step - loss: 0.9338 - acc: 0.6507 - val_loss: 0.8545 - val_acc: 0.6830\n",
      "Epoch 41/120\n",
      "10000/10000 [==============================] - 2s 213us/step - loss: 0.8302 - acc: 0.7006 - val_loss: 0.7644 - val_acc: 0.7270\n",
      "Epoch 42/120\n",
      "10000/10000 [==============================] - 2s 215us/step - loss: 0.7561 - acc: 0.7286 - val_loss: 0.7032 - val_acc: 0.7390\n",
      "Epoch 43/120\n",
      "10000/10000 [==============================] - 2s 218us/step - loss: 0.7089 - acc: 0.7414 - val_loss: 0.6817 - val_acc: 0.7360\n",
      "Epoch 44/120\n",
      "10000/10000 [==============================] - 2s 218us/step - loss: 0.6721 - acc: 0.7560 - val_loss: 0.6663 - val_acc: 0.7340\n",
      "Epoch 45/120\n",
      "10000/10000 [==============================] - 2s 223us/step - loss: 0.6433 - acc: 0.7632 - val_loss: 0.6267 - val_acc: 0.7360\n",
      "Epoch 46/120\n",
      "10000/10000 [==============================] - 3s 330us/step - loss: 0.6176 - acc: 0.7723 - val_loss: 0.6112 - val_acc: 0.7530\n",
      "Epoch 47/120\n",
      "10000/10000 [==============================] - 3s 286us/step - loss: 0.5839 - acc: 0.7846 - val_loss: 0.5854 - val_acc: 0.7560\n",
      "Epoch 48/120\n",
      "10000/10000 [==============================] - 2s 231us/step - loss: 0.5655 - acc: 0.7881 - val_loss: 0.5633 - val_acc: 0.7650\n",
      "Epoch 49/120\n",
      "10000/10000 [==============================] - 2s 230us/step - loss: 0.5215 - acc: 0.8033 - val_loss: 0.5285 - val_acc: 0.7830\n",
      "Epoch 50/120\n",
      "10000/10000 [==============================] - 2s 237us/step - loss: 0.4938 - acc: 0.8165 - val_loss: 0.5102 - val_acc: 0.8040\n",
      "Epoch 51/120\n",
      "10000/10000 [==============================] - 2s 215us/step - loss: 0.4625 - acc: 0.8256 - val_loss: 0.4596 - val_acc: 0.8210\n",
      "Epoch 52/120\n",
      "10000/10000 [==============================] - 2s 220us/step - loss: 0.4315 - acc: 0.8426 - val_loss: 0.4673 - val_acc: 0.8180\n",
      "Epoch 53/120\n",
      "10000/10000 [==============================] - 2s 221us/step - loss: 0.4080 - acc: 0.8530 - val_loss: 0.4252 - val_acc: 0.8420\n",
      "Epoch 54/120\n",
      "10000/10000 [==============================] - 2s 219us/step - loss: 0.3845 - acc: 0.8570 - val_loss: 0.4272 - val_acc: 0.8380\n",
      "Epoch 55/120\n",
      "10000/10000 [==============================] - 2s 225us/step - loss: 0.3761 - acc: 0.8628 - val_loss: 0.4038 - val_acc: 0.8470\n",
      "Epoch 56/120\n",
      "10000/10000 [==============================] - 2s 228us/step - loss: 0.3620 - acc: 0.8693 - val_loss: 0.3794 - val_acc: 0.8640\n",
      "Epoch 57/120\n",
      "10000/10000 [==============================] - 2s 237us/step - loss: 0.3469 - acc: 0.8726 - val_loss: 0.3810 - val_acc: 0.8560\n",
      "Epoch 58/120\n",
      "10000/10000 [==============================] - 2s 228us/step - loss: 0.3375 - acc: 0.8744 - val_loss: 0.3977 - val_acc: 0.8460\n",
      "Epoch 59/120\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 2s 234us/step - loss: 0.3234 - acc: 0.8763 - val_loss: 0.3587 - val_acc: 0.8620\n",
      "Epoch 60/120\n",
      "10000/10000 [==============================] - 2s 237us/step - loss: 0.3134 - acc: 0.8834 - val_loss: 0.3605 - val_acc: 0.8530\n",
      "Epoch 61/120\n",
      "10000/10000 [==============================] - 2s 234us/step - loss: 0.3103 - acc: 0.8842 - val_loss: 0.3440 - val_acc: 0.8640\n",
      "Epoch 62/120\n",
      "10000/10000 [==============================] - 2s 232us/step - loss: 0.2978 - acc: 0.8895 - val_loss: 0.3318 - val_acc: 0.8670\n",
      "Epoch 63/120\n",
      "10000/10000 [==============================] - 2s 223us/step - loss: 0.2970 - acc: 0.8909 - val_loss: 0.3695 - val_acc: 0.8530\n",
      "Epoch 64/120\n",
      "10000/10000 [==============================] - 2s 217us/step - loss: 0.2916 - acc: 0.8913 - val_loss: 0.3389 - val_acc: 0.8680\n",
      "Epoch 65/120\n",
      "10000/10000 [==============================] - 2s 217us/step - loss: 0.2683 - acc: 0.9028 - val_loss: 0.3424 - val_acc: 0.8730\n",
      "Epoch 66/120\n",
      "10000/10000 [==============================] - 2s 219us/step - loss: 0.2592 - acc: 0.9042 - val_loss: 0.3269 - val_acc: 0.8800\n",
      "Epoch 67/120\n",
      "10000/10000 [==============================] - 2s 217us/step - loss: 0.2557 - acc: 0.9093 - val_loss: 0.3120 - val_acc: 0.8840\n",
      "Epoch 68/120\n",
      "10000/10000 [==============================] - 2s 225us/step - loss: 0.2518 - acc: 0.9055 - val_loss: 0.3310 - val_acc: 0.8880\n",
      "Epoch 69/120\n",
      "10000/10000 [==============================] - 3s 251us/step - loss: 0.2302 - acc: 0.9131 - val_loss: 0.2761 - val_acc: 0.8980\n",
      "Epoch 70/120\n",
      "10000/10000 [==============================] - 3s 272us/step - loss: 0.2245 - acc: 0.9203 - val_loss: 0.2854 - val_acc: 0.8970\n",
      "Epoch 71/120\n",
      "10000/10000 [==============================] - 2s 235us/step - loss: 0.2156 - acc: 0.9214 - val_loss: 0.2842 - val_acc: 0.9020\n",
      "Epoch 72/120\n",
      "10000/10000 [==============================] - 3s 266us/step - loss: 0.2023 - acc: 0.9259 - val_loss: 0.2571 - val_acc: 0.9050\n",
      "Epoch 73/120\n",
      "10000/10000 [==============================] - 3s 289us/step - loss: 0.1961 - acc: 0.9296 - val_loss: 0.2681 - val_acc: 0.9040\n",
      "Epoch 74/120\n",
      "10000/10000 [==============================] - 3s 250us/step - loss: 0.1900 - acc: 0.9324 - val_loss: 0.2316 - val_acc: 0.9180\n",
      "Epoch 75/120\n",
      "10000/10000 [==============================] - 3s 281us/step - loss: 0.1842 - acc: 0.9341 - val_loss: 0.2453 - val_acc: 0.9100\n",
      "Epoch 76/120\n",
      "10000/10000 [==============================] - 3s 277us/step - loss: 0.1784 - acc: 0.9368 - val_loss: 0.2521 - val_acc: 0.9080\n",
      "Epoch 77/120\n",
      "10000/10000 [==============================] - 2s 242us/step - loss: 0.1658 - acc: 0.9408 - val_loss: 0.2389 - val_acc: 0.9190\n",
      "Epoch 78/120\n",
      "10000/10000 [==============================] - 2s 226us/step - loss: 0.1736 - acc: 0.9406 - val_loss: 0.2087 - val_acc: 0.9300\n",
      "Epoch 79/120\n",
      "10000/10000 [==============================] - 2s 234us/step - loss: 0.1532 - acc: 0.9461 - val_loss: 0.2303 - val_acc: 0.9270\n",
      "Epoch 80/120\n",
      "10000/10000 [==============================] - 2s 235us/step - loss: 0.1561 - acc: 0.9443 - val_loss: 0.1968 - val_acc: 0.9360\n",
      "Epoch 81/120\n",
      "10000/10000 [==============================] - 2s 246us/step - loss: 0.1479 - acc: 0.9487 - val_loss: 0.2192 - val_acc: 0.9190\n",
      "Epoch 82/120\n",
      "10000/10000 [==============================] - 2s 226us/step - loss: 0.1433 - acc: 0.9490 - val_loss: 0.1930 - val_acc: 0.9380\n",
      "Epoch 83/120\n",
      "10000/10000 [==============================] - 2s 225us/step - loss: 0.1430 - acc: 0.9518 - val_loss: 0.1953 - val_acc: 0.9330\n",
      "Epoch 84/120\n",
      "10000/10000 [==============================] - 2s 229us/step - loss: 0.1354 - acc: 0.9519 - val_loss: 0.1948 - val_acc: 0.9300\n",
      "Epoch 85/120\n",
      "10000/10000 [==============================] - 2s 227us/step - loss: 0.1331 - acc: 0.9546 - val_loss: 0.1844 - val_acc: 0.9390\n",
      "Epoch 86/120\n",
      "10000/10000 [==============================] - 2s 228us/step - loss: 0.1299 - acc: 0.9518 - val_loss: 0.1649 - val_acc: 0.9380\n",
      "Epoch 87/120\n",
      "10000/10000 [==============================] - 2s 246us/step - loss: 0.1276 - acc: 0.9549 - val_loss: 0.2127 - val_acc: 0.9240\n",
      "Epoch 88/120\n",
      "10000/10000 [==============================] - 2s 238us/step - loss: 0.1181 - acc: 0.9580 - val_loss: 0.2052 - val_acc: 0.9280\n",
      "Epoch 89/120\n",
      "10000/10000 [==============================] - 2s 242us/step - loss: 0.1177 - acc: 0.9572 - val_loss: 0.1822 - val_acc: 0.9450\n",
      "Epoch 90/120\n",
      "10000/10000 [==============================] - 2s 232us/step - loss: 0.1163 - acc: 0.9595 - val_loss: 0.1718 - val_acc: 0.9430\n",
      "Epoch 91/120\n",
      "10000/10000 [==============================] - 2s 247us/step - loss: 0.1110 - acc: 0.9586 - val_loss: 0.1707 - val_acc: 0.9430\n",
      "Epoch 92/120\n",
      "10000/10000 [==============================] - 3s 250us/step - loss: 0.1103 - acc: 0.9608 - val_loss: 0.1985 - val_acc: 0.9370\n",
      "Epoch 93/120\n",
      "10000/10000 [==============================] - 2s 240us/step - loss: 0.1019 - acc: 0.9626 - val_loss: 0.1733 - val_acc: 0.9390\n",
      "Epoch 94/120\n",
      "10000/10000 [==============================] - 2s 241us/step - loss: 0.1043 - acc: 0.9624 - val_loss: 0.1745 - val_acc: 0.9430\n",
      "Epoch 95/120\n",
      "10000/10000 [==============================] - 2s 234us/step - loss: 0.0931 - acc: 0.9656 - val_loss: 0.1667 - val_acc: 0.9500\n",
      "Epoch 96/120\n",
      "10000/10000 [==============================] - 2s 248us/step - loss: 0.0992 - acc: 0.9637 - val_loss: 0.1559 - val_acc: 0.9480\n",
      "Epoch 97/120\n",
      "10000/10000 [==============================] - 2s 236us/step - loss: 0.0996 - acc: 0.9654 - val_loss: 0.1592 - val_acc: 0.9500\n",
      "Epoch 98/120\n",
      "10000/10000 [==============================] - 3s 254us/step - loss: 0.0967 - acc: 0.9659 - val_loss: 0.1650 - val_acc: 0.9500\n",
      "Epoch 99/120\n",
      "10000/10000 [==============================] - 2s 237us/step - loss: 0.0885 - acc: 0.9694 - val_loss: 0.1448 - val_acc: 0.9580\n",
      "Epoch 100/120\n",
      "10000/10000 [==============================] - 2s 233us/step - loss: 0.0962 - acc: 0.9693 - val_loss: 0.1435 - val_acc: 0.9530\n",
      "Epoch 101/120\n",
      "10000/10000 [==============================] - 2s 239us/step - loss: 0.0802 - acc: 0.9729 - val_loss: 0.1386 - val_acc: 0.9620\n",
      "Epoch 102/120\n",
      "10000/10000 [==============================] - 2s 237us/step - loss: 0.0814 - acc: 0.9720 - val_loss: 0.1810 - val_acc: 0.9400\n",
      "Epoch 103/120\n",
      "10000/10000 [==============================] - 2s 240us/step - loss: 0.0859 - acc: 0.9709 - val_loss: 0.1473 - val_acc: 0.9580\n",
      "Epoch 104/120\n",
      "10000/10000 [==============================] - 2s 238us/step - loss: 0.0808 - acc: 0.9722 - val_loss: 0.1404 - val_acc: 0.9620\n",
      "Epoch 105/120\n",
      "10000/10000 [==============================] - 2s 232us/step - loss: 0.0843 - acc: 0.9728 - val_loss: 0.1544 - val_acc: 0.9420\n",
      "Epoch 106/120\n",
      "10000/10000 [==============================] - 2s 230us/step - loss: 0.0750 - acc: 0.9741 - val_loss: 0.1311 - val_acc: 0.9610\n",
      "Epoch 107/120\n",
      "10000/10000 [==============================] - 3s 251us/step - loss: 0.0808 - acc: 0.9722 - val_loss: 0.1270 - val_acc: 0.9630\n",
      "Epoch 108/120\n",
      "10000/10000 [==============================] - 2s 238us/step - loss: 0.0736 - acc: 0.9746 - val_loss: 0.1372 - val_acc: 0.9530\n",
      "Epoch 109/120\n",
      "10000/10000 [==============================] - 2s 231us/step - loss: 0.0718 - acc: 0.9766 - val_loss: 0.1554 - val_acc: 0.9530\n",
      "Epoch 110/120\n",
      "10000/10000 [==============================] - 2s 237us/step - loss: 0.0778 - acc: 0.9738 - val_loss: 0.1569 - val_acc: 0.9470\n",
      "Epoch 111/120\n",
      "10000/10000 [==============================] - 2s 247us/step - loss: 0.0721 - acc: 0.9738 - val_loss: 0.1239 - val_acc: 0.9600\n",
      "Epoch 112/120\n",
      "10000/10000 [==============================] - 2s 241us/step - loss: 0.0657 - acc: 0.9777 - val_loss: 0.1278 - val_acc: 0.9590\n",
      "Epoch 113/120\n",
      "10000/10000 [==============================] - 3s 252us/step - loss: 0.0717 - acc: 0.9760 - val_loss: 0.1372 - val_acc: 0.9610\n",
      "Epoch 114/120\n",
      "10000/10000 [==============================] - 3s 254us/step - loss: 0.0694 - acc: 0.9767 - val_loss: 0.1389 - val_acc: 0.9610\n",
      "Epoch 115/120\n",
      "10000/10000 [==============================] - 3s 260us/step - loss: 0.0633 - acc: 0.9775 - val_loss: 0.1205 - val_acc: 0.9610\n",
      "Epoch 116/120\n",
      "10000/10000 [==============================] - 2s 232us/step - loss: 0.0692 - acc: 0.9774 - val_loss: 0.1246 - val_acc: 0.9670\n",
      "Epoch 117/120\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 2s 236us/step - loss: 0.0642 - acc: 0.9782 - val_loss: 0.1912 - val_acc: 0.9380\n",
      "Epoch 118/120\n",
      "10000/10000 [==============================] - 2s 242us/step - loss: 0.0616 - acc: 0.9798 - val_loss: 0.1243 - val_acc: 0.9600\n",
      "Epoch 119/120\n",
      "10000/10000 [==============================] - 3s 284us/step - loss: 0.0668 - acc: 0.9768 - val_loss: 0.1030 - val_acc: 0.9680\n",
      "Epoch 120/120\n",
      "10000/10000 [==============================] - 3s 288us/step - loss: 0.0593 - acc: 0.9803 - val_loss: 0.1074 - val_acc: 0.9680\n",
      "Elapsed time: 0:04:56.90\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "# train\n",
    "model.fit([inputs_train, queries_train], answers_train,\n",
    "          batch_size=32,\n",
    "          epochs=120,\n",
    "          validation_data=([inputs_test, queries_test], answers_test))\n",
    "\n",
    "# save\n",
    "save_path = \"./dnn/\"\n",
    "# save entire network to HDF5 (save everything, suggested)\n",
    "model.save(os.path.join(save_path,\"chatbot.h5\"))\n",
    "# save the vocab too, indexes must be the same\n",
    "pickle.dump( vocab, open( os.path.join(save_path,\"vocab.pkl\"), \"wb\" ) )\n",
    "\n",
    "elapsed_time = time.time() - start_time\n",
    "print(\"Elapsed time: {}\".format(hms_string(elapsed_time)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the model, if it exists, load vocab too\n",
    "save_path = \"./dnn/\"\n",
    "model = load_model(os.path.join(save_path,\"chatbot.h5\"))\n",
    "vocab = pickle.load( open( os.path.join(save_path,\"vocab.pkl\"), \"rb\" ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate Accuracy\n",
    "\n",
    "We evaluate the accuracy, using the same technique as previous classification networks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[2.3238423e-18 2.5143602e-18 2.3133702e-18 ... 2.6530610e-18\n",
      "  1.8533650e-18 2.2625758e-18]\n",
      " [8.0050475e-14 7.3438549e-14 6.4761103e-14 ... 6.8537176e-14\n",
      "  6.4089196e-14 7.0876431e-14]\n",
      " [1.2593846e-18 1.3367013e-18 1.2563471e-18 ... 1.1518428e-18\n",
      "  1.0292255e-18 1.3966357e-18]\n",
      " ...\n",
      " [1.4936653e-16 1.5909874e-16 1.3077923e-16 ... 1.5915703e-16\n",
      "  1.3617577e-16 1.6359896e-16]\n",
      " [3.8541100e-14 3.0556165e-14 3.4044074e-14 ... 3.0966989e-14\n",
      "  3.1383096e-14 2.7715989e-14]\n",
      " [6.4671424e-12 6.8727970e-12 5.1056368e-12 ... 5.5267995e-12\n",
      "  5.7186747e-12 6.8221743e-12]]\n"
     ]
    }
   ],
   "source": [
    "pred = model.predict([inputs_test, queries_test])\n",
    "# See what the predictions look like, they are just probabilities of each class.\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12  9 15 12 15 12 11 12 17 11 15 11 17 10 10 15 11 15 17 15 12 17 17 12\n",
      " 10 10 10 15 15 15 12 17 15 15  9 17  9 11 15 10  9 11  9 12 11 15 12  9\n",
      " 17 10 10 17 11 11 15 10 15 12 11 12 12 17  9 11  9 15 15  9 17 17 11 12\n",
      " 11 10  9 15 15 15 12 17 11  9 10 15 17 11 11 12 15 15  9  9 12  9  9 15\n",
      " 15 15 10 10 10  9 17 11 11 12 10  9 17 10  9  9 15 12 15 11 17 15  9 17\n",
      " 10  9 10 12 10 17 11 17 11  9 11 12  9 11 17  9 17 15  9  9 17 12 12 17\n",
      " 10 15 15 15 15 15 11 11  9 17 10  9 11 10 17 12 10 10 10 11  9 11 11 11\n",
      " 15 10 17 17 11  9  9 17 15 10 17 17 12 12 12 12 10  9 17 11 15 17 12 15\n",
      " 11 12 12 11 17 15  9 10 12 17 17 10 12  9 15 12  9 12  9 10 12 10  9 10\n",
      "  9 12 12  9  9 12 15  9 10 17 12 11  9  9 17 11 12 17 12 10 10 10 10 17\n",
      " 12 15 11 11 15 15 11 11 11 11 10 17  9  9 10 10 17 17 15 11 11 17 12 17\n",
      " 17  9 11 10  9 10 15 17 17 15  9 17 15 12 17  9  9 11 17 17 11 17  9 17\n",
      " 11  9 15 15 10 15 15 11 11 12 12  9  9  9 15 17 12 11  9 17 17 15 11 11\n",
      " 17 10 17 17 10 17 17 11 11 15 15 11 11  9 10 11 11 10 15 17 15 10 17 17\n",
      " 12 11 12 12 12 11 17 15 11 17 12 12 10 15 10 11 11 10 10  9 17 12 11 12\n",
      " 11  9 11 11 11 11 11 12 12 12 12 12 11 17 17 10 15  9 15 10  9 10 10 17\n",
      " 17 15 17 17 17 15 15 12  9 10 17 17 10 17  9 11 10 11 10 15 17 15 12 12\n",
      " 17 15 10 11  9 12 17 10 10 15 12  9 12 12  9 12 10 15 15 17 10 10 17 15\n",
      " 15 15 17 11 15 12 10 11 10 12  9 11 10 11 10 12 11 15 15 15 15 11  9 15\n",
      " 10 12 11 12  9 11 10 17 11 11 10 10 10 17 12 17 17  9 15 11  9 15 11  9\n",
      " 10 10 12 12 11 10  9 12 10  9 15 10 10 11 11 15 15 10 12 12 15 11  9 12\n",
      "  9 15 10 17 17 10 10 17 10 10 17  9  9 10 12 11 17 17 12 12 11 12 17 11\n",
      " 11 12 11 11 11 17  9 11 10 12 12 10 11 17 12 12 15 11 15 15 12 10 10  9\n",
      " 17 10 15 12 15 10 10 10 11 10 11 15 10  9  9 15 10 12 10 10 15 15 17  9\n",
      " 10 12 10 10 12 12 11 15 10 10 11 12 10 10 17 12 17 11  9  9 17 17 11 10\n",
      " 10 10 15 15 17 11 15 12 17 10 15 17  9 11  9 17 15 12 15 17 11 17 17 15\n",
      "  9 10  9 10 10 10  9 10 10 15 11 11 11 15 12  9 11 11 10 17 11 11 12 17\n",
      " 15 15  9 11 15 15 11  9 15 15 11 15 15 15 11 10 17  9 17 17 17 17 10 10\n",
      " 17  9 15  9 17 12 12 12 15 15 15 17  9 11  9 17  9 12 11  9  9 17 11 11\n",
      " 11 10 15  9 12 12  9  9 17 12 12  9 17 12 11 11 15 15 12 11 12 17 12 17\n",
      " 11 12 15 17 17 11 17  9 11  9 10 10 12 17 12 12  9  9  9 15 11 11 15 17\n",
      "  9 10  9 12 17 15 17 17 12 10 12 11 11 17 15  9 17 10  9 11 12 15  9 15\n",
      " 12 10  9 10 17 17 17 15  9  9  9 17 10 17 17  9 12 17 17 12 12 11 11 11\n",
      "  9 17 11 17 17 17 17 11 12  9 17 10 17 11 10 10 12 12 11 11 17 12 12 17\n",
      " 17 15 11 17 11 11  9 12  9 17  9 10  9 15 11  9 15 15 10  9  9  9 17 11\n",
      " 11 11 17 15 15 10 10 11 11 17 10 12  9 10 10 10 10 10 15 17  9 11 17 15\n",
      " 12 11 10 10 15 12 10 11 12 12 11 11 11 15 12 17 10 12 17 15 12 10 10 12\n",
      " 17 10 11 17 11 12 17 17 11 12 11 17  9  9 11 17 11 11 15 15 11 17 10 10\n",
      " 17 12 11 15 12 11 11 10  9 12 12 15 10  9 17 15 17  9 17 17 10 17 15 12\n",
      " 12 11 10 10 10 17 17 15 12 17  9  9 15 10 12 15 15 15 17 17 15 11 15 10\n",
      " 12  9  9 10 11 11  9  9 12 15  9 15  9 12 12 10 17 17 10 12 15  9 10 17\n",
      " 11 11 11 17  9  9 11 15 11 11 17 15 12 17  9 11]\n"
     ]
    }
   ],
   "source": [
    "# Use argmax to turn those into actual predictions.  The class (word) with the highest\n",
    "# probability is the answer.\n",
    "\n",
    "pred = np.argmax(pred,axis=1)\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final accuracy: 0.968\n"
     ]
    }
   ],
   "source": [
    "score = metrics.accuracy_score(answers_test, pred)\n",
    "print(\"Final accuracy: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adhoc Query\n",
    "\n",
    "You might want to create your own stories and questions.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Remember, I only know these words: ['.', '?', 'Daniel', 'John', 'Mary', 'Sandra', 'Where', 'back', 'bathroom', 'bedroom', 'garden', 'hallway', 'is', 'journeyed', 'kitchen', 'moved', 'office', 'the', 'to', 'travelled', 'went']\n",
      "\n",
      "[1.13837134e-11 1.20584836e-11 7.97362263e-12 1.01629026e-11\n",
      " 1.16442056e-11 9.60991963e-12 8.67680667e-12 1.20000234e-11\n",
      " 8.34373456e-12 1.12214297e-01 9.30538226e-05 8.48266482e-01\n",
      " 2.91703129e-03 1.13392681e-11 9.72282758e-12 2.62911152e-02\n",
      " 8.84859894e-12 1.02181016e-02 9.39766060e-12 8.41918982e-12\n",
      " 9.12226111e-12 9.85704574e-12]\n",
      "Answer: garden([11])\n"
     ]
    }
   ],
   "source": [
    "print(\"Remember, I only know these words: {}\".format(vocab))\n",
    "print()\n",
    "story = \"Daniel went to the hallway. Mary went to the bathroom.  Daniel went to the bedroom.\"\n",
    "query = \"Where is Sandra?\"\n",
    "\n",
    "adhoc_stories = (tokenize(story), tokenize(query), '?')\n",
    "\n",
    "adhoc_train, adhoc_query, adhoc_answer = vectorize_stories([adhoc_stories])\n",
    "\n",
    "pred = model.predict([adhoc_train, adhoc_query])\n",
    "print(pred[0])\n",
    "pred = np.argmax(pred,axis=1)\n",
    "print(\"Answer: {}({})\".format(vocab[pred[0]-1],pred))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word2Vec\n",
    "\n",
    "Word2vec is a group of related models that are used to produce word embeddings. These models are shallow, two-layer neural networks that are trained to reconstruct linguistic contexts of words. Word2vec takes as its input a large corpus of text and produces a vector space, typically of several hundred dimensions, with each unique word in the corpus being assigned a corresponding vector in the space. Word vectors are positioned in the vector space such that words that share common contexts in the corpus are located in close proximity to one another in the space.\n",
    "\n",
    "Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). [Efficient estimation of word representations in vector space](https://arxiv.org/abs/1301.3781). arXiv preprint arXiv:1301.3781.\n",
    "\n",
    "![Word2Vec](https://pbs.twimg.com/media/C7jJxIjWkAA8E_s.jpg)\n",
    "[Trust Word2Vec](https://twitter.com/DanilBaibak/status/844647217885581312)\n",
    "\n",
    "### Suggested Software for Word2Vec\n",
    "\n",
    "* [GoogleNews Vectors](https://code.google.com/archive/p/word2vec/), [GitHub Mirror](https://github.com/mmihaltz/word2vec-GoogleNews-vectors)\n",
    "* [Python Gensim](https://radimrehurek.com/gensim/)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz\n",
      "1647050752/1647046227 [==============================] - 706s 0us/step\n",
      "/Users/jheaton/.keras/datasets/GoogleNews-vectors-negative300.bin.gz\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    path = get_file('GoogleNews-vectors-negative300.bin.gz', origin='https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz')\n",
    "except:\n",
    "    print('Error downloading')\n",
    "    raise\n",
    "    \n",
    "print(path)    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'gensim'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-16-8ffa5f126234>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mgensim\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;31m# Not that the path below refers to a location on my hard drive.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;31m# You should download GoogleNews Vectors (see suggested software above)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgensim\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodels\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mKeyedVectors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_word2vec_format\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbinary\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'gensim'"
     ]
    }
   ],
   "source": [
    "import gensim\n",
    "\n",
    "# Not that the path below refers to a location on my hard drive.\n",
    "# You should download GoogleNews Vectors (see suggested software above)\n",
    "model = gensim.models.KeyedVectors.load_word2vec_format(path, binary=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Word2vec makes each word a vector.  We are using the 300-number vector, which can be seen for the word \"hello\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w = model['hello']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(w))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below shows the distance between two words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "w1 = model['king']\n",
    "w2 = model['queen']\n",
    "\n",
    "dist = np.linalg.norm(w1-w2)\n",
    "\n",
    "print(dist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows the classic word2vec equation of **queen = (king - man) + female**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.most_similar(positive=['woman', 'king'], negative=['man'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The following code shows which item does not belong with the others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.doesnt_match(\"house garage store dog\".split())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code shows the similarity between two words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.similarity('iphone', 'android')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code shows which words are most similar to the given one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.most_similar('dog')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More on LSTM\n",
    "\n",
    "* [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)\n",
    "* [LSTM Music](https://www.youtube.com/watch?v=0VTI1BBLydE)\n",
    "* [Natural Language Processing from Scratch](https://arxiv.org/abs/1103.0398)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (wustl)",
   "language": "python",
   "name": "wustl"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}