udacity-deep-learning/6_lstm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8tQJd2YSCfWR"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D7tqLMoKF6uq"
   },
   "source": [
    "Deep Learning\n",
    "=============\n",
    "\n",
    "Assignment 6\n",
    "------------\n",
    "\n",
    "After training a skip-gram model in `5_word2vec.ipynb`, the goal of this notebook is to train a LSTM character model over [Text8](http://mattmahoney.net/dc/textdata) data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "MvEblsgEXxrd"
   },
   "outputs": [],
   "source": [
    "# These are all the modules we'll be using later. Make sure you can import them\n",
    "# before proceeding further.\n",
    "from __future__ import print_function\n",
    "import os\n",
    "import numpy as np\n",
    "import random\n",
    "import string\n",
    "import tensorflow as tf\n",
    "import zipfile\n",
    "from six.moves import range\n",
    "from six.moves.urllib.request import urlretrieve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "collapsed": false,
    "executionInfo": {
     "elapsed": 5993,
     "status": "ok",
     "timestamp": 1445965582896,
     "user": {
      "color": "#1FA15D",
      "displayName": "Vincent Vanhoucke",
      "isAnonymous": false,
      "isMe": true,
      "permissionId": "05076109866853157986",
      "photoUrl": "//lh6.googleusercontent.com/-cCJa7dTDcgQ/AAAAAAAAAAI/AAAAAAAACgw/r2EZ_8oYer4/s50-c-k-no/photo.jpg",
      "sessionId": "6f6f07b359200c46",
      "userId": "102167687554210253930"
     },
     "user_tz": 420
    },
    "id": "RJ-o3UBUFtCw",
    "outputId": "d530534e-0791-4a94-ca6d-1c8f1b908a9e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found and verified text8.zip\n"
     ]
    }
   ],
   "source": [
    "url = 'http://mattmahoney.net/dc/'\n",
    "\n",
    "def maybe_download(filename, expected_bytes):\n",
    "  \"\"\"Download a file if not present, and make sure it's the right size.\"\"\"\n",
    "  if not os.path.exists(filename):\n",
    "    filename, _ = urlretrieve(url + filename, filename)\n",
    "  statinfo = os.stat(filename)\n",
    "  if statinfo.st_size == expected_bytes:\n",
    "    print('Found and verified %s' % filename)\n",
    "  else:\n",
    "    print(statinfo.st_size)\n",
    "    raise Exception(\n",
    "      'Failed to verify ' + filename + '. Can you get to it with a browser?')\n",
    "  return filename\n",
    "\n",
    "filename = maybe_download('text8.zip', 31344016)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "collapsed": false,
    "executionInfo": {
     "elapsed": 5982,
     "status": "ok",
     "timestamp": 1445965582916,
     "user": {
      "color": "#1FA15D",
      "displayName": "Vincent Vanhoucke",
      "isAnonymous": false,
      "isMe": true,
      "permissionId": "05076109866853157986",
      "photoUrl": "//lh6.googleusercontent.com/-cCJa7dTDcgQ/AAAAAAAAAAI/AAAAAAAACgw/r2EZ_8oYer4/s50-c-k-no/photo.jpg",
      "sessionId": "6f6f07b359200c46",
      "userId": "102167687554210253930"
     },
     "user_tz": 420
    },
    "id": "Mvf09fjugFU_",
    "outputId": "8f75db58-3862-404b-a0c3-799380597390"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data size 100000000\n"
     ]
    }
   ],
   "source": [
    "def read_data(filename):\n",
    "  with zipfile.ZipFile(filename) as f:\n",
    "    name = f.namelist()[0]\n",
    "    data = tf.compat.as_str(f.read(name))\n",
    "  return data\n",
    "  \n",
    "text = read_data(filename)\n",
    "print('Data size %d' % len(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ga2CYACE-ghb"
   },
   "source": [
    "Create a small validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "collapsed": false,
    "executionInfo": {
     "elapsed": 6184,
     "status": "ok",
     "timestamp": 1445965583138,
     "user": {
      "color": "#1FA15D",
      "displayName": "Vincent Vanhoucke",
      "isAnonymous": false,
      "isMe": true,
      "permissionId": "05076109866853157986",
      "photoUrl": "//lh6.googleusercontent.com/-cCJa7dTDcgQ/AAAAAAAAAAI/AAAAAAAACgw/r2EZ_8oYer4/s50-c-k-no/photo.jpg",
      "sessionId": "6f6f07b359200c46",
      "userId": "102167687554210253930"
     },
     "user_tz": 420
    },
    "id": "w-oBpfFG-j43",
    "outputId": "bdb96002-d021-4379-f6de-a977924f0d02"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "99999000 ons anarchists advocate social relations based upon voluntary association of autonomous individuals mutual aid and self governance while anarchism is most easily defined by what it is against anarchis\n",
      "1000  anarchism originated as a term of abuse first used against early working class radicals including t\n"
     ]
    }
   ],
   "source": [
    "valid_size = 1000\n",
    "valid_text = text[:valid_size]\n",
    "train_text = text[valid_size:]\n",
    "train_size = len(train_text)\n",
    "print(train_size, train_text[:200])\n",
    "print(valid_size, valid_text[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Zdw6i4F8glpp"
   },
   "source": [
    "Utility functions to map characters to vocabulary IDs and back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "collapsed": false,
    "executionInfo": {
     "elapsed": 6276,
     "status": "ok",
     "timestamp": 1445965583249,
     "user": {
      "color": "#1FA15D",
      "displayName": "Vincent Vanhoucke",
      "isAnonymous": false,
      "isMe": true,
      "permissionId": "05076109866853157986",
      "photoUrl": "//lh6.googleusercontent.com/-cCJa7dTDcgQ/AAAAAAAAAAI/AAAAAAAACgw/r2EZ_8oYer4/s50-c-k-no/photo.jpg",
      "sessionId": "6f6f07b359200c46",
      "userId": "102167687554210253930"
     },
     "user_tz": 420
    },
    "id": "gAL1EECXeZsD",
    "outputId": "88fc9032-feb9-45ff-a9a0-a26759cc1f2e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unexpected character: ï\n",
      "1 26 0 0\n",
      "a z  \n"
     ]
    }
   ],
   "source": [
    "# 将字符转为id\n",
    "vocabulary_size = len(string.ascii_lowercase) + 1 # [a-z] + ' '\n",
    "first_letter = ord(string.ascii_lowercase[0])\n",
    "\n",
    "def char2id(char):\n",
    "  if char in string.ascii_lowercase:\n",
    "    return ord(char) - first_letter + 1\n",
    "  elif char == ' ':\n",
    "    return 0\n",
    "  else:\n",
    "    print('Unexpected character: %s' % char)\n",
    "    return 0\n",
    "  \n",
    "def id2char(dictid):\n",
    "  if dictid > 0:\n",
    "    return chr(dictid + first_letter - 1)\n",
    "  else:\n",
    "    return ' '\n",
    "\n",
    "print(char2id('a'), char2id('z'), char2id(' '), char2id('ï'))\n",
    "print(id2char(1), id2char(26), id2char(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lFwoyygOmWsL"
   },
   "source": [
    "Function to generate a training batch for the LSTM model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "collapsed": false,
    "executionInfo": {
     "elapsed": 6473,
     "status": "ok",
     "timestamp": 1445965583467,
     "user": {
      "color": "#1FA15D",
      "displayName": "Vincent Vanhoucke",
      "isAnonymous": false,
      "isMe": true,
      "permissionId": "05076109866853157986",
      "photoUrl": "//lh6.googleusercontent.com/-cCJa7dTDcgQ/AAAAAAAAAAI/AAAAAAAACgw/r2EZ_8oYer4/s50-c-k-no/photo.jpg",
      "sessionId": "6f6f07b359200c46",
      "userId": "102167687554210253930"
     },
     "user_tz": 420
    },
    "id": "d9wMtjy5hCj9",
    "outputId": "3dd79c80-454a-4be0-8b71-4a4a357b3367"
   },
   "outputs": [],
   "source": [
    "batch_size=64\n",
    "num_unrollings=10\n",
    "\n",
    "class BatchGenerator(object):\n",
    "  def __init__(self, text, batch_size, num_unrollings):\n",
    "    self._text = text\n",
    "    self._text_size = len(text)\n",
    "    self._batch_size = batch_size\n",
    "    self._num_unrollings = num_unrollings\n",
    "    segment = self._text_size // batch_size\n",
    "    self._cursor = [ offset * segment for offset in range(batch_size)]\n",
    "    self._last_batch = self._next_batch()\n",
    "  \n",
    "  def _next_batch(self):\n",
    "    \"\"\"Generate a single batch from the current cursor position in the data.\"\"\"\n",
    "    batch = np.zeros(shape=(self._batch_size, vocabulary_size), dtype=np.float)\n",
    "    for b in range(self._batch_size):\n",
    "      batch[b, char2id(self._text[self._cursor[b]])] = 1.0\n",
    "      self._cursor[b] = (self._cursor[b] + 1) % self._text_size\n",
    "    return batch\n",
    "  \n",
    "  def next(self):\n",
    "    \"\"\"Generate the next array of batches from the data. The array consists of\n",
    "    the last batch of the previous array, followed by num_unrollings new ones.\n",
    "    \"\"\"\n",
    "    batches = [self._last_batch]\n",
    "    for step in range(self._num_unrollings):\n",
    "      batches.append(self._next_batch())\n",
    "    self._last_batch = batches[-1]\n",
    "    return batches\n",
    "\n",
    "def characters(probabilities):\n",
    "  \"\"\"Turn a 1-hot encoding or a probability distribution over the possible\n",
    "  characters back into its (most likely) character representation.\"\"\"\n",
    "  return [id2char(c) for c in np.argmax(probabilities, 1)]\n",
    "\n",
    "def batches2string(batches):\n",
    "  \"\"\"Convert a sequence of batches back into their (most likely) string\n",
    "  representation.\"\"\"\n",
    "  s = [''] * batches[0].shape[0]\n",
    "  for b in batches:\n",
    "    s = [''.join(x) for x in zip(s, characters(b))]\n",
    "  return s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ons anarchi', 'when milita', 'lleria arch', ' abbeys and', 'married urr', 'hel and ric', 'y and litur', 'ay opened f', 'tion from t', 'migration t', 'new york ot', 'he boeing s', 'e listed wi', 'eber has pr', 'o be made t', 'yer who rec', 'ore signifi', 'a fierce cr', ' two six ei', 'aristotle s', 'ity can be ', ' and intrac', 'tion of the', 'dy to pass ', 'f certain d', 'at it will ', 'e convince ', 'ent told hi', 'ampaign and', 'rver side s', 'ious texts ', 'o capitaliz', 'a duplicate', 'gh ann es d', 'ine january', 'ross zero t', 'cal theorie', 'ast instanc', ' dimensiona', 'most holy m', 't s support', 'u is still ', 'e oscillati', 'o eight sub', 'of italy la', 's the tower', 'klahoma pre', 'erprise lin', 'ws becomes ', 'et in a naz', 'the fabian ', 'etchy to re', ' sharman ne', 'ised empero', 'ting in pol', 'd neo latin', 'th risky ri', 'encyclopedi', 'fense the a', 'duating fro', 'treet grid ', 'ations more', 'appeal of d', 'si have mad']\n",
      "['ists advoca', 'ary governm', 'hes nationa', 'd monasteri', 'raca prince', 'chard baer ', 'rgical lang', 'for passeng', 'the nationa', 'took place ', 'ther well k', 'seven six s', 'ith a gloss', 'robably bee', 'to recogniz', 'ceived the ', 'icant than ', 'ritic of th', 'ight in sig', 's uncaused ', ' lost as in', 'cellular ic', 'e size of t', ' him a stic', 'drugs confu', ' take to co', ' the priest', 'im to name ', 'd barred at', 'standard fo', ' such as es', 'ze on the g', 'e of the or', 'd hiver one', 'y eight mar', 'the lead ch', 'es classica', 'ce the non ', 'al analysis', 'mormons bel', 't or at lea', ' disagreed ', 'ing system ', 'btypes base', 'anguages th', 'r commissio', 'ess one nin', 'nux suse li', ' the first ', 'zi concentr', ' society ne', 'elatively s', 'etworks sha', 'or hirohito', 'litical ini', 'n most of t', 'iskerdoo ri', 'ic overview', 'air compone', 'om acnm acc', ' centerline', 'e than any ', 'devotional ', 'de such dev']\n",
      "[' a']\n",
      "['an']\n"
     ]
    }
   ],
   "source": [
    "train_batches = BatchGenerator(train_text, batch_size, num_unrollings)\n",
    "valid_batches = BatchGenerator(valid_text, 1, 1)\n",
    "\n",
    "print(batches2string(train_batches.next()))\n",
    "print(batches2string(train_batches.next()))\n",
    "print(batches2string(valid_batches.next()))\n",
    "print(batches2string(valid_batches.next()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "KyVd8FxT5QBc"
   },
   "outputs": [],
   "source": [
    "def logprob(predictions, labels):\n",
    "  \"\"\"Log-probability of the true labels in a predicted batch.\"\"\"\n",
    "  predictions[predictions < 1e-10] = 1e-10\n",
    "  return np.sum(np.multiply(labels, -np.log(predictions))) / labels.shape[0]\n",
    "\n",
    "def sample_distribution(distribution):\n",
    "  \"\"\"Sample one element from a distribution assumed to be an array of normalized\n",
    "  probabilities.\n",
    "  \"\"\"\n",
    "  r = random.uniform(0, 1)\n",
    "  s = 0\n",
    "  for i in range(len(distribution)):\n",
    "    s += distribution[i]\n",
    "    if s >= r:\n",
    "      return i\n",
    "  return len(distribution) - 1\n",
    "\n",
    "def sample(prediction):\n",
    "  \"\"\"Turn a (column) prediction into 1-hot encoded samples.\"\"\"\n",
    "  p = np.zeros(shape=[1, vocabulary_size], dtype=np.float)\n",
    "  p[0, sample_distribution(prediction[0])] = 1.0\n",
    "  return p\n",
    "\n",
    "def random_distribution():\n",
    "  \"\"\"Generate a random column of probabilities.\"\"\"\n",
    "  b = np.random.uniform(0.0, 1.0, size=[1, vocabulary_size])\n",
    "  return b/np.sum(b, 1)[:,None]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "K8f67YXaDr4C"
   },
   "source": [
    "Simple LSTM Model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "Q5rxZK6RDuGe"
   },
   "outputs": [],
   "source": [
    "num_nodes = 64\n",
    "\n",
    "graph = tf.Graph()\n",
    "with graph.as_default():\n",
    "  \n",
    "  # Parameters:\n",
    "  # Input gate: input, previous output, and bias.\n",
    "  ix = tf.Variable(tf.truncated_normal([vocabulary_size, num_nodes], -0.1, 0.1))\n",
    "  im = tf.Variable(tf.truncated_normal([num_nodes, num_nodes], -0.1, 0.1))\n",
    "  ib = tf.Variable(tf.zeros([1, num_nodes]))\n",
    "  # Forget gate: input, previous output, and bias.\n",
    "  fx = tf.Variable(tf.truncated_normal([vocabulary_size, num_nodes], -0.1, 0.1))\n",
    "  fm = tf.Variable(tf.truncated_normal([num_nodes, num_nodes], -0.1, 0.1))\n",
    "  fb = tf.Variable(tf.zeros([1, num_nodes]))\n",
    "  # Memory cell: input, state and bias.                             \n",
    "  cx = tf.Variable(tf.truncated_normal([vocabulary_size, num_nodes], -0.1, 0.1))\n",
    "  cm = tf.Variable(tf.truncated_normal([num_nodes, num_nodes], -0.1, 0.1))\n",
    "  cb = tf.Variable(tf.zeros([1, num_nodes]))\n",
    "  # Output gate: input, previous output, and bias.\n",
    "  ox = tf.Variable(tf.truncated_normal([vocabulary_size, num_nodes], -0.1, 0.1))\n",
    "  om = tf.Variable(tf.truncated_normal([num_nodes, num_nodes], -0.1, 0.1))\n",
    "  ob = tf.Variable(tf.zeros([1, num_nodes]))\n",
    "  # Variables saving state across unrollings.\n",
    "  saved_output = tf.Variable(tf.zeros([batch_size, num_nodes]), trainable=False)\n",
    "  saved_state = tf.Variable(tf.zeros([batch_size, num_nodes]), trainable=False)\n",
    "  # Classifier weights and biases.\n",
    "  w = tf.Variable(tf.truncated_normal([num_nodes, vocabulary_size], -0.1, 0.1))\n",
    "  b = tf.Variable(tf.zeros([vocabulary_size]))\n",
    "  \n",
    "  # Definition of the cell computation.\n",
    "  def lstm_cell(i, o, state):\n",
    "    \"\"\"Create a LSTM cell. See e.g.: http://arxiv.org/pdf/1402.1128v1.pdf\n",
    "    Note that in this formulation, we omit the various connections between the\n",
    "    previous state and the gates.\"\"\"\n",
    "    input_gate = tf.sigmoid(tf.matmul(i, ix) + tf.matmul(o, im) + ib)\n",
    "    forget_gate = tf.sigmoid(tf.matmul(i, fx) + tf.matmul(o, fm) + fb)\n",
    "    update = tf.matmul(i, cx) + tf.matmul(o, cm) + cb\n",
    "    state = forget_gate * state + input_gate * tf.tanh(update)\n",
    "    output_gate = tf.sigmoid(tf.matmul(i, ox) + tf.matmul(o, om) + ob)\n",
    "    return output_gate * tf.tanh(state), state\n",
    "\n",
    "  # Input data.\n",
    "  train_data = list()\n",
    "  for _ in range(num_unrollings + 1):\n",
    "    train_data.append(\n",
    "      tf.placeholder(tf.float32, shape=[batch_size,vocabulary_size]))\n",
    "  # 一次10个数据，11个字母，train_inputs: “ons anarch” 而 train_labels:“ns anarchi”\n",
    "  train_inputs = train_data[:num_unrollings]\n",
    "  train_labels = train_data[1:]  # labels are inputs shifted by one time step.\n",
    "\n",
    "\n",
    "  # Unrolled LSTM loop.\n",
    "  outputs = list()\n",
    "  output = saved_output\n",
    "  state = saved_state\n",
    "  # 利用lstm预测input获得outputs\n",
    "  for i in train_inputs:\n",
    "    output, state = lstm_cell(i, output, state)\n",
    "    outputs.append(output)\n",
    "\n",
    "  # State saving across unrollings.\n",
    "  with tf.control_dependencies([saved_output.assign(output),\n",
    "                                saved_state.assign(state)]):\n",
    "    # Classifier.\n",
    "    logits = tf.nn.xw_plus_b(tf.concat(outputs, 0), w, b)\n",
    "    loss = tf.reduce_mean(\n",
    "      tf.nn.softmax_cross_entropy_with_logits(\n",
    "        labels=tf.concat(train_labels, 0), logits=logits))\n",
    "\n",
    "  # Optimizer.\n",
    "  global_step = tf.Variable(0)\n",
    "  learning_rate = tf.train.exponential_decay(\n",
    "    10.0, global_step, 5000, 0.1, staircase=True)\n",
    "  optimizer = tf.train.GradientDescentOptimizer(learning_rate)\n",
    "  gradients, v = zip(*optimizer.compute_gradients(loss))\n",
    "  gradients, _ = tf.clip_by_global_norm(gradients, 1.25)\n",
    "  optimizer = optimizer.apply_gradients(\n",
    "    zip(gradients, v), global_step=global_step)\n",
    "\n",
    "  # Predictions.\n",
    "  train_prediction = tf.nn.softmax(logits)\n",
    "  \n",
    "  # Sampling and validation eval: batch 1, no unrolling.\n",
    "  sample_input = tf.placeholder(tf.float32, shape=[1, vocabulary_size])\n",
    "  saved_sample_output = tf.Variable(tf.zeros([1, num_nodes]))\n",
    "  saved_sample_state = tf.Variable(tf.zeros([1, num_nodes]))\n",
    "  reset_sample_state = tf.group(\n",
    "    saved_sample_output.assign(tf.zeros([1, num_nodes])),\n",
    "    saved_sample_state.assign(tf.zeros([1, num_nodes])))\n",
    "  sample_output, sample_state = lstm_cell(\n",
    "    sample_input, saved_sample_output, saved_sample_state)\n",
    "  with tf.control_dependencies([saved_sample_output.assign(sample_output),\n",
    "                                saved_sample_state.assign(sample_state)]):\n",
    "    sample_prediction = tf.nn.softmax(tf.nn.xw_plus_b(sample_output, w, b))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "cellView": "both",
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "output_extras": [
      {
       "item_id": 41
      },
      {
       "item_id": 80
      },
      {
       "item_id": 126
      },
      {
       "item_id": 144
      }
     ]
    },
    "colab_type": "code",
    "collapsed": false,
    "executionInfo": {
     "elapsed": 199909,
     "status": "ok",
     "timestamp": 1445965877333,
     "user": {
      "color": "#1FA15D",
      "displayName": "Vincent Vanhoucke",
      "isAnonymous": false,
      "isMe": true,
      "permissionId": "05076109866853157986",
      "photoUrl": "//lh6.googleusercontent.com/-cCJa7dTDcgQ/AAAAAAAAAAI/AAAAAAAACgw/r2EZ_8oYer4/s50-c-k-no/photo.jpg",
      "sessionId": "6f6f07b359200c46",
      "userId": "102167687554210253930"
     },
     "user_tz": 420
    },
    "id": "RD9zQCZTEaEm",
    "outputId": "5e868466-2532-4545-ce35-b403cf5d9de6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initialized\n",
      "Average loss at step 0: 3.297466 learning rate: 10.000000\n",
      "Minibatch perplexity: 27.04\n",
      "================================================================================\n",
      "prcoitz nbtmxdpluzcsj yczes  ywr qju seywetlug cvcrmev oxrihglfldomhsc btgz johl\n",
      "rugan  nof  xcahrj  xoimhzkie xmwvsv lpye  re r axmecm  t qqe gina dbuvaecnjopjn\n",
      "e kwyxe v thd  mnftiritohdwslf li qxdgwba sckxtrnaamy seqeofl ulah rpjei vrnrtfo\n",
      "z cjrutq ye vzoxgiyoidsmd hp mmtpehus ah t cpjhhe ttfei cptz kopl skecbussani  e\n",
      "dqazme m flgvaqimaaemnrsxp nosqgpeeurckxzhqoogysadt pnx q  sf s g enfnegj hh dss\n",
      "================================================================================\n",
      "Validation set perplexity: 20.30\n",
      "Average loss at step 100: 2.593902 learning rate: 10.000000\n",
      "Minibatch perplexity: 11.10\n",
      "Validation set perplexity: 10.42\n",
      "Average loss at step 200: 2.260314 learning rate: 10.000000\n",
      "Minibatch perplexity: 8.63\n",
      "Validation set perplexity: 8.69\n",
      "Average loss at step 300: 2.103249 learning rate: 10.000000\n",
      "Minibatch perplexity: 7.39\n",
      "Validation set perplexity: 8.12\n",
      "Average loss at step 400: 1.995193 learning rate: 10.000000\n",
      "Minibatch perplexity: 7.48\n",
      "Validation set perplexity: 7.95\n",
      "Average loss at step 500: 1.933473 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.41\n",
      "Validation set perplexity: 6.97\n",
      "Average loss at step 600: 1.907060 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.36\n",
      "Validation set perplexity: 6.90\n",
      "Average loss at step 700: 1.861398 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.38\n",
      "Validation set perplexity: 6.62\n",
      "Average loss at step 800: 1.820803 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.82\n",
      "Validation set perplexity: 6.41\n",
      "Average loss at step 900: 1.830661 learning rate: 10.000000\n",
      "Minibatch perplexity: 7.00\n",
      "Validation set perplexity: 6.19\n",
      "Average loss at step 1000: 1.825107 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.56\n",
      "================================================================================\n",
      "buranign his receizitism inficidal riben one eight lessoud one of nine he hume s\n",
      " of hemen govermer surfces one five smarly ffur the prisced of incompetibe of ze\n",
      "mer veru consict sjanalf and dipple kinstand proper of vilture of the consudevan\n",
      "rago helo sen he cleate snon pulliw into cont hivarder the one one finild decese\n",
      "hictile becapilic seven perides of the magena with and the radelt hixlian one hi\n",
      "================================================================================\n",
      "Validation set perplexity: 6.12\n",
      "Average loss at step 1100: 1.779447 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.52\n",
      "Validation set perplexity: 5.88\n",
      "Average loss at step 1200: 1.756202 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.93\n",
      "Validation set perplexity: 5.60\n",
      "Average loss at step 1300: 1.736041 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.69\n",
      "Validation set perplexity: 5.56\n",
      "Average loss at step 1400: 1.747829 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.99\n",
      "Validation set perplexity: 5.46\n",
      "Average loss at step 1500: 1.739305 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.72\n",
      "Validation set perplexity: 5.38\n",
      "Average loss at step 1600: 1.747943 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.47\n",
      "Validation set perplexity: 5.51\n",
      "Average loss at step 1700: 1.716449 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.56\n",
      "Validation set perplexity: 5.44\n",
      "Average loss at step 1800: 1.678055 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.39\n",
      "Validation set perplexity: 5.29\n",
      "Average loss at step 1900: 1.649635 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.09\n",
      "Validation set perplexity: 5.19\n",
      "Average loss at step 2000: 1.699116 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.70\n",
      "================================================================================\n",
      "winne the end fornating thema is and incendently cantiastary lasen it profucally\n",
      "renssasimen whitern alludly main ars moliremall by bill emporting deffenser mako\n",
      "ka barse one one nine excension of these putlanchn theoublis in stone underse to\n",
      " shovi doondy thei one knikns prodineties fluet botm to seven eight seven one ni\n",
      "orial in the idsints in importicies on two one wither de the calarching whileshe\n",
      "================================================================================\n",
      "Validation set perplexity: 5.34\n",
      "Average loss at step 2100: 1.690451 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.24\n",
      "Validation set perplexity: 4.94\n",
      "Average loss at step 2200: 1.685421 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.58\n",
      "Validation set perplexity: 5.01\n",
      "Average loss at step 2300: 1.642907 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.88\n",
      "Validation set perplexity: 4.80\n",
      "Average loss at step 2400: 1.661219 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.16\n",
      "Validation set perplexity: 4.81\n",
      "Average loss at step 2500: 1.677324 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.20\n",
      "Validation set perplexity: 4.64\n",
      "Average loss at step 2600: 1.653231 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.78\n",
      "Validation set perplexity: 4.64\n",
      "Average loss at step 2700: 1.656223 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.53\n",
      "Validation set perplexity: 4.62\n",
      "Average loss at step 2800: 1.650554 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.66\n",
      "Validation set perplexity: 4.64\n",
      "Average loss at step 2900: 1.649697 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.60\n",
      "Validation set perplexity: 4.71\n",
      "Average loss at step 3000: 1.649148 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.12\n",
      "================================================================================\n",
      "kenifalling with canocyar in zero two kneralays pracrifer in yrangquctile in in \n",
      "ted to official combunist and imponsy all hobil continus control societ spead an\n",
      "d whose hussince malaway emperopated from lifing suplants aritagary timeh matan \n",
      "nor acries one gned and boiq south aflian pridisiting all at anaterity statgr ec\n",
      "ure inneger american inflyzing consive pattral ld tedr daision of independer alv\n",
      "================================================================================\n",
      "Validation set perplexity: 4.66\n",
      "Average loss at step 3100: 1.633971 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.79\n",
      "Validation set perplexity: 4.61\n",
      "Average loss at step 3200: 1.651917 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.72\n",
      "Validation set perplexity: 4.63\n",
      "Average loss at step 3300: 1.639248 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.96\n",
      "Validation set perplexity: 4.59\n",
      "Average loss at step 3400: 1.671412 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.43\n",
      "Validation set perplexity: 4.57\n",
      "Average loss at step 3500: 1.658427 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.53\n",
      "Validation set perplexity: 4.66\n",
      "Average loss at step 3600: 1.667742 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.54\n",
      "Validation set perplexity: 4.57\n",
      "Average loss at step 3700: 1.647332 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.07\n",
      "Validation set perplexity: 4.57\n",
      "Average loss at step 3800: 1.643315 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.55\n",
      "Validation set perplexity: 4.61\n",
      "Average loss at step 3900: 1.633747 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.21\n",
      "Validation set perplexity: 4.63\n",
      "Average loss at step 4000: 1.651066 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.64\n",
      "================================================================================\n",
      "gether elussion geesthies cause roted the rake is is hels with whose of a satic \n",
      "dases for gene remorredilibe by cubss aphicter notwicic is i mmalling insecutati\n",
      "p only numbers abter hive benker citucal higntered of mansitasional printicalito\n",
      "lynaguamer stills the formphtleroll ruther in samicanity the solihigatively ther\n",
      "urb ingluent of ymartraqe the manding inceatus entrets doeket ovenkazies halgam \n",
      "================================================================================\n",
      "Validation set perplexity: 4.60\n",
      "Average loss at step 4100: 1.632685 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.21\n",
      "Validation set perplexity: 4.78\n",
      "Average loss at step 4200: 1.633376 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.22\n",
      "Validation set perplexity: 4.55\n",
      "Average loss at step 4300: 1.611644 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.06\n",
      "Validation set perplexity: 4.52\n",
      "Average loss at step 4400: 1.607315 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.86\n",
      "Validation set perplexity: 4.46\n",
      "Average loss at step 4500: 1.615705 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.21\n",
      "Validation set perplexity: 4.58\n",
      "Average loss at step 4600: 1.614903 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.99\n",
      "Validation set perplexity: 4.65\n",
      "Average loss at step 4700: 1.621680 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.22\n",
      "Validation set perplexity: 4.55\n",
      "Average loss at step 4800: 1.627614 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.39\n",
      "Validation set perplexity: 4.58\n",
      "Average loss at step 4900: 1.633121 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.18\n",
      "Validation set perplexity: 4.58\n",
      "Average loss at step 5000: 1.603882 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.49\n",
      "================================================================================\n",
      "uda in s rall govent inding the was ottomy gates of r shopl pire and the used an\n",
      "isa simples he to cially linkery facabah two linker and ejre that has one edohro\n",
      "met moss lay on the world man the aram elx a p hingosoly one one one eight eight\n",
      "dd posity of mility are pare or maa est reficience parease by author of altomaic\n",
      "ching the henga of than two palsed cateral of cityer place way people from given\n",
      "================================================================================\n",
      "Validation set perplexity: 4.72\n",
      "Average loss at step 5100: 1.603216 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.03\n",
      "Validation set perplexity: 4.49\n",
      "Average loss at step 5200: 1.587077 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.65\n",
      "Validation set perplexity: 4.43\n",
      "Average loss at step 5300: 1.578445 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.67\n",
      "Validation set perplexity: 4.41\n",
      "Average loss at step 5400: 1.576484 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.95\n",
      "Validation set perplexity: 4.39\n",
      "Average loss at step 5500: 1.566184 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.73\n",
      "Validation set perplexity: 4.35\n",
      "Average loss at step 5600: 1.578901 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.84\n",
      "Validation set perplexity: 4.35\n",
      "Average loss at step 5700: 1.566950 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.38\n",
      "Validation set perplexity: 4.36\n",
      "Average loss at step 5800: 1.578459 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.93\n",
      "Validation set perplexity: 4.38\n",
      "Average loss at step 5900: 1.570264 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.06\n",
      "Validation set perplexity: 4.35\n",
      "Average loss at step 6000: 1.545577 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.93\n",
      "================================================================================\n",
      "wakzations of the onlogesh gable he the qurity connaich area mexication briting \n",
      "ar srived atlease have in captayer was cakernegic charinar furt under povited by\n",
      "xille propurents infantically including decaduly leadic reved of economict incon\n",
      "dession gotherom also them was reprovers debie there of queed to intertably at d\n",
      "ques most uslam some as characte of nearn the helamon agains rerional there is v\n",
      "================================================================================\n",
      "Validation set perplexity: 4.35\n",
      "Average loss at step 6100: 1.562849 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.07\n",
      "Validation set perplexity: 4.33\n",
      "Average loss at step 6200: 1.533720 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.93\n",
      "Validation set perplexity: 4.33\n",
      "Average loss at step 6300: 1.545136 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.03\n",
      "Validation set perplexity: 4.33\n",
      "Average loss at step 6400: 1.538162 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.43\n",
      "Validation set perplexity: 4.33\n",
      "Average loss at step 6500: 1.554771 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.61\n",
      "Validation set perplexity: 4.32\n",
      "Average loss at step 6600: 1.594018 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.70\n",
      "Validation set perplexity: 4.30\n",
      "Average loss at step 6700: 1.578919 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.18\n",
      "Validation set perplexity: 4.31\n",
      "Average loss at step 6800: 1.602105 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.76\n",
      "Validation set perplexity: 4.32\n",
      "Average loss at step 6900: 1.580424 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.67\n",
      "Validation set perplexity: 4.33\n",
      "Average loss at step 7000: 1.578265 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.05\n",
      "================================================================================\n",
      "ast afrecon alandeggodom and eny musty by paged be etwoctory his paramms moctcec\n",
      "ady in polos were that controduction in holdy soupte for anthals reguest from do\n",
      "xing to sereser stets the respend instative in two zero zero four zero zero over\n",
      " the five one nine nine nine strams wassoric rar some of there move includes dur\n",
      "ant prilded would compitties ror paronity by barraused seven three zero zero wit\n",
      "================================================================================\n",
      "Validation set perplexity: 4.30\n"
     ]
    }
   ],
   "source": [
    "num_steps = 7001\n",
    "summary_frequency = 100\n",
    "\n",
    "with tf.Session(graph=graph) as session:\n",
    "  tf.global_variables_initializer().run()\n",
    "  print('Initialized')\n",
    "  mean_loss = 0\n",
    "  for step in range(num_steps):\n",
    "    batches = train_batches.next()\n",
    "    feed_dict = dict()\n",
    "    for i in range(num_unrollings + 1):\n",
    "      feed_dict[train_data[i]] = batches[i]\n",
    "    _, l, predictions, lr = session.run(\n",
    "      [optimizer, loss, train_prediction, learning_rate], feed_dict=feed_dict)\n",
    "    mean_loss += l\n",
    "    if step % summary_frequency == 0:\n",
    "      if step > 0:\n",
    "        mean_loss = mean_loss / summary_frequency\n",
    "      # The mean loss is an estimate of the loss over the last few batches.\n",
    "      print(\n",
    "        'Average loss at step %d: %f learning rate: %f' % (step, mean_loss, lr))\n",
    "      mean_loss = 0\n",
    "      labels = np.concatenate(list(batches)[1:])\n",
    "      print('Minibatch perplexity: %.2f' % float(\n",
    "        np.exp(logprob(predictions, labels))))\n",
    "      if step % (summary_frequency * 10) == 0:\n",
    "        # Generate some samples.\n",
    "        print('=' * 80)\n",
    "        for _ in range(5):\n",
    "          feed = sample(random_distribution())\n",
    "          sentence = characters(feed)[0]\n",
    "          reset_sample_state.run()\n",
    "          for _ in range(79):\n",
    "            prediction = sample_prediction.eval({sample_input: feed})\n",
    "            feed = sample(prediction)\n",
    "            sentence += characters(feed)[0]\n",
    "          print(sentence)\n",
    "        print('=' * 80)\n",
    "      # Measure validation set perplexity.\n",
    "      reset_sample_state.run()\n",
    "      valid_logprob = 0\n",
    "      for _ in range(valid_size):\n",
    "        b = valid_batches.next()\n",
    "        predictions = sample_prediction.eval({sample_input: b[0]})\n",
    "        valid_logprob = valid_logprob + logprob(predictions, b[1])\n",
    "      print('Validation set perplexity: %.2f' % float(np.exp(\n",
    "        valid_logprob / valid_size)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pl4vtmFfa5nn"
   },
   "source": [
    "---\n",
    "Problem 1\n",
    "---------\n",
    "\n",
    "You might have noticed that the definition of the LSTM cell involves 4 matrix multiplications with the input, and 4 matrix multiplications with the output. Simplify the expression by using a single matrix multiply for each, and variables that are 4 times larger.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_nodes = 64\n",
    "\n",
    "graph = tf.Graph()\n",
    "with graph.as_default():\n",
    "  \n",
    "  # Parameters:\n",
    "  # Input gate,Forget gate,Memory cell,Output gate: input, previous output, and bias.\n",
    "  input_x = tf.Variable(tf.truncated_normal([vocabulary_size, num_nodes*4], -0.1, 0.1))\n",
    "  output_m = tf.Variable(tf.truncated_normal([num_nodes, num_nodes*4], -0.1, 0.1))\n",
    "  bias = tf.Variable(tf.zeros([1, num_nodes*4]))\n",
    "\n",
    "  # Variables saving state across unrollings.\n",
    "  saved_output = tf.Variable(tf.zeros([batch_size, num_nodes]), trainable=False)\n",
    "  saved_state = tf.Variable(tf.zeros([batch_size, num_nodes]), trainable=False)\n",
    "  # Classifier weights and biases.\n",
    "  w = tf.Variable(tf.truncated_normal([num_nodes, vocabulary_size], -0.1, 0.1))\n",
    "  b = tf.Variable(tf.zeros([vocabulary_size]))\n",
    "  \n",
    "  # Definition of the cell computation.\n",
    "  def lstm_cell(i, o, state):\n",
    "    tmp = tf.matmul(i, input_x) + tf.matmul(o, output_m) + bias;\n",
    "    tmp_input, tmp_forget, update, tmp_output = tf.split(tmp, 4, 1)\n",
    "    input_gate = tf.sigmoid(tmp_input)\n",
    "    forget_gate = tf.sigmoid(tmp_forget)\n",
    "    state = forget_gate * state + input_gate * tf.tanh(update)\n",
    "    output_gate = tf.sigmoid(tmp_output)\n",
    "    return output_gate * tf.tanh(state), state\n",
    "\n",
    "  # Input data.\n",
    "  train_data = list()\n",
    "  for _ in range(num_unrollings + 1):\n",
    "    train_data.append(\n",
    "      tf.placeholder(tf.float32, shape=[batch_size,vocabulary_size]))\n",
    "  train_inputs = train_data[:num_unrollings]\n",
    "  train_labels = train_data[1:]  # labels are inputs shifted by one time step.\n",
    "\n",
    "\n",
    "  # Unrolled LSTM loop.\n",
    "  outputs = list()\n",
    "  output = saved_output\n",
    "  state = saved_state\n",
    "  # 利用lstm预测input获得outputs\n",
    "  for i in train_inputs:\n",
    "    output, state = lstm_cell(i, output, state)\n",
    "    outputs.append(output)\n",
    "\n",
    "  # State saving across unrollings.\n",
    "  with tf.control_dependencies([saved_output.assign(output),\n",
    "                                saved_state.assign(state)]):\n",
    "    # Classifier.\n",
    "    logits = tf.nn.xw_plus_b(tf.concat(outputs, 0), w, b)\n",
    "    loss = tf.reduce_mean(\n",
    "      tf.nn.softmax_cross_entropy_with_logits(\n",
    "        labels=tf.concat(train_labels, 0), logits=logits))\n",
    "\n",
    "  # Optimizer.\n",
    "  global_step = tf.Variable(0)\n",
    "  learning_rate = tf.train.exponential_decay(\n",
    "    10.0, global_step, 5000, 0.1, staircase=True)\n",
    "  optimizer = tf.train.GradientDescentOptimizer(learning_rate)\n",
    "  gradients, v = zip(*optimizer.compute_gradients(loss))\n",
    "  gradients, _ = tf.clip_by_global_norm(gradients, 1.25)\n",
    "  optimizer = optimizer.apply_gradients(\n",
    "    zip(gradients, v), global_step=global_step)\n",
    "\n",
    "  # Predictions.\n",
    "  train_prediction = tf.nn.softmax(logits)\n",
    "  \n",
    "  # Sampling and validation eval: batch 1, no unrolling.\n",
    "  sample_input = tf.placeholder(tf.float32, shape=[1, vocabulary_size])\n",
    "  saved_sample_output = tf.Variable(tf.zeros([1, num_nodes]))\n",
    "  saved_sample_state = tf.Variable(tf.zeros([1, num_nodes]))\n",
    "  reset_sample_state = tf.group(\n",
    "    saved_sample_output.assign(tf.zeros([1, num_nodes])),\n",
    "    saved_sample_state.assign(tf.zeros([1, num_nodes])))\n",
    "  sample_output, sample_state = lstm_cell(\n",
    "    sample_input, saved_sample_output, saved_sample_state)\n",
    "  with tf.control_dependencies([saved_sample_output.assign(sample_output),\n",
    "                                saved_sample_state.assign(sample_state)]):\n",
    "    sample_prediction = tf.nn.softmax(tf.nn.xw_plus_b(sample_output, w, b))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initialized\n",
      "Average loss at step 0: 3.292466 learning rate: 10.000000\n",
      "Minibatch perplexity: 26.91\n",
      "================================================================================\n",
      "y serkcyenocqwgcr yk o qj banwlakqetki wok ra tux bv mvuosjrir qubczt sotzo hjoj\n",
      "zf   bxipcihenzln psa btmaokoyhirytz niguu  zr t eeojo o oiwuy  f ifprepncsbesxf\n",
      "xayopl jvihxrfilaxeue s jcotru toieqhcteydgpvdc jeceaqa tjnwt buaoehf sshrshma v\n",
      "mtbemezt lab  rd  a c mpanjz  jpate contczwdetoralo lo sip sndgrsxc rt snfno gqp\n",
      "k  slwawg rnst rdseytict of  armvpe f echeh  dunueohwxrpodfovw wa xceeznwcddyhjw\n",
      "================================================================================\n",
      "Validation set perplexity: 19.79\n",
      "Average loss at step 100: 2.590879 learning rate: 10.000000\n",
      "Minibatch perplexity: 12.26\n",
      "Validation set perplexity: 10.59\n",
      "Average loss at step 200: 2.243920 learning rate: 10.000000\n",
      "Minibatch perplexity: 8.24\n",
      "Validation set perplexity: 8.83\n",
      "Average loss at step 300: 2.074710 learning rate: 10.000000\n",
      "Minibatch perplexity: 7.05\n",
      "Validation set perplexity: 8.00\n",
      "Average loss at step 400: 1.987416 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.80\n",
      "Validation set perplexity: 7.91\n",
      "Average loss at step 500: 1.989852 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.41\n",
      "Validation set perplexity: 7.31\n",
      "Average loss at step 600: 1.916649 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.42\n",
      "Validation set perplexity: 6.88\n",
      "Average loss at step 700: 1.889061 learning rate: 10.000000\n",
      "Minibatch perplexity: 7.02\n",
      "Validation set perplexity: 6.68\n",
      "Average loss at step 800: 1.866487 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.98\n",
      "Validation set perplexity: 6.39\n",
      "Average loss at step 900: 1.854887 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.09\n",
      "Validation set perplexity: 6.37\n",
      "Average loss at step 1000: 1.787673 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.73\n",
      "================================================================================\n",
      "recanations it is not nut is poperned the pollogiemaliqut were is and mutinals d\n",
      " of ascommensy the boisting pro ledle altemply by eagopimally miloth a souch stu\n",
      "zer mulas reserment on twoneciny a the cathubuin the us which in the vars mis in\n",
      "cortious were the rore sever aijing of booded by fereuning the ferozed homemy on\n",
      "tern the fibt he time zeation peoopladua fremoog manuntical light of there one g\n",
      "================================================================================\n",
      "Validation set perplexity: 6.06\n",
      "Average loss at step 1100: 1.761482 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.26\n",
      "Validation set perplexity: 6.13\n",
      "Average loss at step 1200: 1.786632 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.08\n",
      "Validation set perplexity: 6.14\n",
      "Average loss at step 1300: 1.767339 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.66\n",
      "Validation set perplexity: 5.85\n",
      "Average loss at step 1400: 1.739369 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.68\n",
      "Validation set perplexity: 5.79\n",
      "Average loss at step 1500: 1.727741 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.35\n",
      "Validation set perplexity: 5.57\n",
      "Average loss at step 1600: 1.718162 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.78\n",
      "Validation set perplexity: 5.53\n",
      "Average loss at step 1700: 1.738716 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.19\n",
      "Validation set perplexity: 5.52\n",
      "Average loss at step 1800: 1.702531 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.66\n",
      "Validation set perplexity: 5.43\n",
      "Average loss at step 1900: 1.705240 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.27\n",
      "Validation set perplexity: 5.31\n",
      "Average loss at step 2000: 1.719138 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.05\n",
      "================================================================================\n",
      "jan jover mayeerationally the everystemes and come stowed myrded energewartanuef\n",
      "site of the theoake of the cour a havchism and cesture with the ender the was tr\n",
      "ist movidia vences almorealitarihebolisies and recentrage coerty were it bleasta\n",
      "wal keogets for used imsrowary londs pewary and all detreate the feed olascesses\n",
      "ce dewere iroliaphes brateraluation alusess as internated of the subsitions a on\n",
      "================================================================================\n",
      "Validation set perplexity: 5.49\n",
      "Average loss at step 2100: 1.694689 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.20\n",
      "Validation set perplexity: 5.46\n",
      "Average loss at step 2200: 1.670641 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.54\n",
      "Validation set perplexity: 5.19\n",
      "Average loss at step 2300: 1.682676 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.53\n",
      "Validation set perplexity: 5.16\n",
      "Average loss at step 2400: 1.682404 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.73\n",
      "Validation set perplexity: 5.29\n",
      "Average loss at step 2500: 1.703302 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.72\n",
      "Validation set perplexity: 5.20\n",
      "Average loss at step 2600: 1.673023 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.14\n",
      "Validation set perplexity: 5.01\n",
      "Average loss at step 2700: 1.692625 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.47\n",
      "Validation set perplexity: 5.21\n",
      "Average loss at step 2800: 1.653630 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.96\n",
      "Validation set perplexity: 5.10\n",
      "Average loss at step 2900: 1.658832 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.80\n",
      "Validation set perplexity: 4.92\n",
      "Average loss at step 3000: 1.660388 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.55\n",
      "================================================================================\n",
      "zation the kciment not backs overob is the six for luminaric as anows userman as\n",
      "leittion with mision malosonratel remounts out belianly or secment has of the ow\n",
      "ters he governming suspate soxting mone whter georeghn beads expore modelshist o\n",
      "ing four he fezrep again farcr lithern thefratists severes loging detan leder to\n",
      "zer azerisan one he two zero clead to foring lqued killed liwh true with charch \n",
      "================================================================================\n",
      "Validation set perplexity: 4.90\n",
      "Average loss at step 3100: 1.656019 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.61\n",
      "Validation set perplexity: 4.94\n",
      "Average loss at step 3200: 1.653441 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.10\n",
      "Validation set perplexity: 5.03\n",
      "Average loss at step 3300: 1.633479 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.11\n",
      "Validation set perplexity: 4.96\n",
      "Average loss at step 3400: 1.640911 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.37\n",
      "Validation set perplexity: 4.96\n",
      "Average loss at step 3500: 1.634015 learning rate: 10.000000\n",
      "Minibatch perplexity: 6.03\n",
      "Validation set perplexity: 4.95\n",
      "Average loss at step 3600: 1.632863 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.35\n",
      "Validation set perplexity: 4.89\n",
      "Average loss at step 3700: 1.632302 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.69\n",
      "Validation set perplexity: 5.01\n",
      "Average loss at step 3800: 1.630289 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.13\n",
      "Validation set perplexity: 4.82\n",
      "Average loss at step 3900: 1.616608 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.13\n",
      "Validation set perplexity: 4.97\n",
      "Average loss at step 4000: 1.623383 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.06\n",
      "================================================================================\n",
      "cessisty sayos armments the decapers either upustendes the to million requered t\n",
      "riazza and by the hridestrue thouralitiels for warginuanial spact on interssinat\n",
      "ger christions anasti compazham wased as inclusions to canat surrell pitised gur\n",
      "d incodoment such the rotas mullliytishalled which great harstate and view was s\n",
      "ed in the stitution ambly benallo distilide lapks of fucutionsis real the calsth\n",
      "================================================================================\n",
      "Validation set perplexity: 4.83\n",
      "Average loss at step 4100: 1.625341 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.24\n",
      "Validation set perplexity: 5.07\n",
      "Average loss at step 4200: 1.610531 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.06\n",
      "Validation set perplexity: 5.06\n",
      "Average loss at step 4300: 1.593464 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.65\n",
      "Validation set perplexity: 5.07\n",
      "Average loss at step 4400: 1.621188 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.01\n",
      "Validation set perplexity: 5.02\n",
      "Average loss at step 4500: 1.629941 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.99\n",
      "Validation set perplexity: 5.00\n",
      "Average loss at step 4600: 1.631912 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.27\n",
      "Validation set perplexity: 4.85\n",
      "Average loss at step 4700: 1.603258 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.45\n",
      "Validation set perplexity: 4.90\n",
      "Average loss at step 4800: 1.586957 learning rate: 10.000000\n",
      "Minibatch perplexity: 5.18\n",
      "Validation set perplexity: 5.08\n",
      "Average loss at step 4900: 1.595864 learning rate: 10.000000\n",
      "Minibatch perplexity: 4.93\n",
      "Validation set perplexity: 4.95\n",
      "Average loss at step 5000: 1.621852 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.19\n",
      "================================================================================\n",
      "ri who he only on the riels approkine astore most even are reed in three one now\n",
      "t deducess to h take kypiner over to zero siter hooristic competer has tiil to l\n",
      "visors highly in the hald offer that ok slomisies world nambited on a paet speci\n",
      "ess accoble yosentsy be fiesh amplesies by sapis buid samulo as indesilt comyras\n",
      "udy out us resignetisters praitein hequined storg increasences by playes well th\n",
      "================================================================================\n",
      "Validation set perplexity: 4.90\n",
      "Average loss at step 5100: 1.632392 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.25\n",
      "Validation set perplexity: 4.75\n",
      "Average loss at step 5200: 1.627188 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.39\n",
      "Validation set perplexity: 4.68\n",
      "Average loss at step 5300: 1.596091 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.79\n",
      "Validation set perplexity: 4.68\n",
      "Average loss at step 5400: 1.589065 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.86\n",
      "Validation set perplexity: 4.67\n",
      "Average loss at step 5500: 1.581539 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.91\n",
      "Validation set perplexity: 4.67\n",
      "Average loss at step 5600: 1.609742 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.42\n",
      "Validation set perplexity: 4.64\n",
      "Average loss at step 5700: 1.566965 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.11\n",
      "Validation set perplexity: 4.67\n",
      "Average loss at step 5800: 1.576415 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.85\n",
      "Validation set perplexity: 4.61\n",
      "Average loss at step 5900: 1.588339 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.78\n",
      "Validation set perplexity: 4.61\n",
      "Average loss at step 6000: 1.559600 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.24\n",
      "================================================================================\n",
      "hed wind justion a can speal includes of its and d harz they but starm catally h\n",
      "kes evaluas one nine seven five commna races stap at has and reaura had flue sti\n",
      "pellein in the same edite the eldents end bael telsry in gas problem structly lo\n",
      "quisioning fiese in dida ameractihat approatic checked the sunsition with shouch\n",
      " the frawer to are yetrales have from abecary not and chaes howeralif tradet son\n",
      "================================================================================\n",
      "Validation set perplexity: 4.61\n",
      "Average loss at step 6100: 1.585554 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.42\n",
      "Validation set perplexity: 4.59\n",
      "Average loss at step 6200: 1.595075 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.80\n",
      "Validation set perplexity: 4.61\n",
      "Average loss at step 6300: 1.607254 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.31\n",
      "Validation set perplexity: 4.61\n",
      "Average loss at step 6400: 1.637948 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.51\n",
      "Validation set perplexity: 4.59\n",
      "Average loss at step 6500: 1.635205 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.73\n",
      "Validation set perplexity: 4.58\n",
      "Average loss at step 6600: 1.602841 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.14\n",
      "Validation set perplexity: 4.54\n",
      "Average loss at step 6700: 1.592709 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.11\n",
      "Validation set perplexity: 4.54\n",
      "Average loss at step 6800: 1.574989 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.37\n",
      "Validation set perplexity: 4.56\n",
      "Average loss at step 6900: 1.574789 learning rate: 1.000000\n",
      "Minibatch perplexity: 5.46\n",
      "Validation set perplexity: 4.54\n",
      "Average loss at step 7000: 1.582733 learning rate: 1.000000\n",
      "Minibatch perplexity: 4.70\n",
      "================================================================================\n",
      "es the pro still they reflucion stage pribe racition since for a lawn clattory c\n",
      "w the such are indeastruble acceptes a carler illiten lock woulding stank to the\n",
      "zedy kalsbalth was since only yeyle syffern the litken ranaloscal four zentsed a\n",
      "t two zero five gaclli treechrate mateement letarticl football potese dehands an\n",
      "tacture or seit again of being inchares among are of two to one nine seven zasiv\n",
      "================================================================================\n",
      "Validation set perplexity: 4.54\n"
     ]
    }
   ],
   "source": [
    "num_steps = 7001\n",
    "summary_frequency = 100\n",
    "\n",
    "with tf.Session(graph=graph) as session:\n",
    "  tf.global_variables_initializer().run()\n",
    "  print('Initialized')\n",
    "  mean_loss = 0\n",
    "  for step in range(num_steps):\n",
    "    batches = train_batches.next()\n",
    "    feed_dict = dict()\n",
    "    for i in range(num_unrollings + 1):\n",
    "      feed_dict[train_data[i]] = batches[i]\n",
    "    _, l, predictions, lr = session.run(\n",
    "      [optimizer, loss, train_prediction, learning_rate], feed_dict=feed_dict)\n",
    "    mean_loss += l\n",
    "    if step % summary_frequency == 0:\n",
    "      if step > 0:\n",
    "        mean_loss = mean_loss / summary_frequency\n",
    "      # The mean loss is an estimate of the loss over the last few batches.\n",
    "      print(\n",
    "        'Average loss at step %d: %f learning rate: %f' % (step, mean_loss, lr))\n",
    "      mean_loss = 0\n",
    "      labels = np.concatenate(list(batches)[1:])\n",
    "      print('Minibatch perplexity: %.2f' % float(\n",
    "        np.exp(logprob(predictions, labels))))\n",
    "      if step % (summary_frequency * 10) == 0:\n",
    "        # Generate some samples.\n",
    "        print('=' * 80)\n",
    "        for _ in range(5):\n",
    "          feed = sample(random_distribution())\n",
    "          sentence = characters(feed)[0]\n",
    "          reset_sample_state.run()\n",
    "          for _ in range(79):\n",
    "            prediction = sample_prediction.eval({sample_input: feed})\n",
    "            feed = sample(prediction)\n",
    "            sentence += characters(feed)[0]\n",
    "          print(sentence)\n",
    "        print('=' * 80)\n",
    "      # Measure validation set perplexity.\n",
    "      reset_sample_state.run()\n",
    "      valid_logprob = 0\n",
    "      for _ in range(valid_size):\n",
    "        b = valid_batches.next()\n",
    "        predictions = sample_prediction.eval({sample_input: b[0]})\n",
    "        valid_logprob = valid_logprob + logprob(predictions, b[1])\n",
    "      print('Validation set perplexity: %.2f' % float(np.exp(\n",
    "        valid_logprob / valid_size)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4eErTCTybtph"
   },
   "source": [
    "---\n",
    "Problem 2\n",
    "---------\n",
    "\n",
    "We want to train a LSTM over bigrams, that is pairs of consecutive characters like 'ab' instead of single characters like 'a'. Since the number of possible bigrams is large, feeding them directly to the LSTM using 1-hot encodings will lead to a very sparse representation that is very wasteful computationally.\n",
    "\n",
    "a- Introduce an embedding lookup on the inputs, and feed the embeddings to the LSTM cell instead of the inputs themselves.\n",
    "\n",
    "b- Write a bigram-based LSTM, modeled on the character LSTM above.\n",
    "\n",
    "c- Introduce Dropout. For best practices on how to use Dropout in LSTMs, refer to this [article](http://arxiv.org/abs/1409.2329).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Y5tapX3kpcqZ"
   },
   "source": [
    "---\n",
    "Problem 3\n",
    "---------\n",
    "\n",
    "(difficult!)\n",
    "\n",
    "Write a sequence-to-sequence LSTM which mirrors all the words in a sentence. For example, if your input is:\n",
    "\n",
    "    the quick brown fox\n",
    "    \n",
    "the model should attempt to output:\n",
    "\n",
    "    eht kciuq nworb xof\n",
    "    \n",
    "Refer to the lecture on how to put together a sequence-to-sequence model, as well as [this article](http://arxiv.org/abs/1409.3215) for best practices.\n",
    "\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "default_view": {},
   "name": "6_lstm.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}