03_CNN_Glove_Keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modeling Stock Market Sentiment with CNNs and Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we will build a CNN Network to predict the stock market sentiment based on a comment about the market."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the following libraries for our analysis:\n",
    "\n",
    "* numpy - numerical computing library used to work with our data\n",
    "* pandas - data analysis library used to read in our data from csv\n",
    "* tensorflow - a lower level deep learning framework used for modeling\n",
    "* keras - a higher level deep learning library that absracts away a lot of DL details. Keras will use Tensforflow in the background"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also be using the python Counter object for counting our vocabulary items and we have a util module that extracts away a lot of the details of our data processing. Please read through the util.py to get a better understanding of how to preprocess the data for analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "import utils as utl\n",
    "from collections import Counter\n",
    "\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.utils import to_categorical\n",
    "\n",
    "from keras.layers import Dense, Input, Flatten, Dropout, Merge\n",
    "from keras.layers import Conv1D, MaxPooling1D, Embedding\n",
    "from keras.models import Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Processing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will train the model using messages tagged with SPY, the S&P 500 index fund, from [StockTwits.com](https://www.stocktwits.com). StockTwits is a social media network for traders and investors to share their views about the stock market. When a user posts a message, they tag the relevant stock ticker ($SPY in our case) and have the option to tag the messages with their sentiment – “bullish” if they believe the stock will go up and “bearish” if they believe the stock will go down.\n",
    "\n",
    "Our dataset consists of approximately 100,000 messages posted in 2017 that are tagged with $SPY where the user indicated their sentiment. Before we get to our CNN Network we have to perform some processing on our data to get it ready for modeling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read and View Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we simply read in our data using pandas, pull out our message and sentiment data into numpy arrays. Let's also take a look at a few samples to get familiar with the data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Messages: $SPY crazy day so far!... Sentiment: bearish\n",
      "Messages: $SPY Will make a new ATH this week. Watch it!... Sentiment: bullish\n",
      "Messages: $SPY $DJIA white elephant in room is $AAPL. Up 14% since election. Strong headwinds w/Trump trade & Strong dollar. How many 7's do you see?... Sentiment: bearish\n",
      "Messages: $SPY blocks above. We break above them We should push to double top... Sentiment: bullish\n",
      "Messages: $SPY Nothing happening in the market today, guess I'll go to the store and spend some $.... Sentiment: bearish\n",
      "Messages: $SPY What an easy call. Good jobs report: good economy, markets go up.  Bad jobs report: no more rate hikes, markets go up.  Win-win.... Sentiment: bullish\n",
      "Messages: $SPY BS market.... Sentiment: bullish\n",
      "Messages: $SPY this rally all the cheerleaders were screaming about this morning is pretty weak. I keep adding 2 my short at all spikes... Sentiment: bearish\n",
      "Messages: $SPY Dollar ripping higher!... Sentiment: bearish\n",
      "Messages: $SPY no reason to go down !... Sentiment: bullish\n"
     ]
    }
   ],
   "source": [
    "# read data from csv file\n",
    "data = pd.read_csv(\"data/StockTwits_SPY_Sentiment_2017.gz\",\n",
    "                   encoding=\"utf-8\",\n",
    "                   compression=\"gzip\",\n",
    "                   index_col=0)\n",
    "\n",
    "# get messages and sentiment labels\n",
    "messages = data.message.values\n",
    "labels = data.sentiment.values\n",
    "\n",
    "# View sample of messages with sentiment\n",
    "\n",
    "for i in range(10):\n",
    "    print(\"Messages: {}...\".format(messages[i]),\n",
    "          \"Sentiment: {}\".format(labels[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check Message Lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also want to get a sense of the distribution of the length of our inputs. We check for the longest and average messages. We will need to make our input length uniform to feed the data into our model so later we will have some decisions to make about possibly truncating some of the longer messages if they are too long. We also notice that one message has no content remaining after we preprocessed the data, so we will remove this message from our data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Zero-length messages: 0\n",
      "Maximum message length: 635\n",
      "Average message length: 75.64462136603174\n"
     ]
    }
   ],
   "source": [
    "messages_lens = Counter([len(x) for x in messages])\n",
    "print(\"Zero-length messages: {}\".format(messages_lens[0]))\n",
    "print(\"Maximum message length: {}\".format(max(messages_lens)))\n",
    "print(\"Average message length: {}\".format(np.mean([len(x) for x in messages])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "messages, labels = utl.drop_empty_messages(messages, labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preprocess Messages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Working with raw text data often requires preprocessing the text in some fashion to normalize for context. In our case we want to normalize for known unique \"entities\" that appear within messages that carry a similar contextual meaning when analyzing sentiment. This means we want to replace references to specific stock tickers, user names, url links or numbers with a special token identifying the \"entity\". Here we will also make everything lower case and remove punctuation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "messages = np.array([utl.preprocess_ST_message(message) for message in messages])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generate Vocab to Index Mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use a Keras `Tokenizer` in order to generate our word index. The tockenizer takes our vocabulary and assigns each word a unique index from 1 to *VOCAB_SIZE*. Zero is reserved for padding which we will get to in a bit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "VOCAB = sorted(list(set(messages)))\n",
    "VOCAB_SIZE = len(VOCAB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer(num_words=VOCAB_SIZE)\n",
    "tokenizer.fit_on_texts(messages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 31975 unique tokens.\n"
     ]
    }
   ],
   "source": [
    "word_index = tokenizer.word_index\n",
    "print('Found %s unique tokens.' % len(word_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Encode Messages and Labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to \"translate\" our text to number for our algorithm to take in as inputs. We call this translation an encoding. We encode our messages to sequences of numbers where each nummber is the word index from the mapping we made earlier. The phrase \"I am bullish\" would now look something like [1, 234, 5345] where each number is the index for the respective word in the message. We can do this very easily with our tokenizer by calling the `text_to_sequences` method. For our sentiment labels we will simply encode \"bearish\" as 0 and \"bullish\" as 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "sequences = tokenizer.texts_to_sequences(messages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = to_categorical(utl.encode_ST_labels(labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Pad Messages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last thing we need to do is make our message inputs the same length. In our case, the average message length is 78 words so we will use a max length of around this amount. We need to Zero Pad the rest of the messages that are shorter. We will use a left padding that will pad all of the messages that are shorter than 244 words with 0s at the beginning. So our encoded \"I am bullish\" messages goes from [1, 234, 5345] (length 3) to [0, 0, 0, 0, 0, 0, ... , 0, 0, 1, 234, 5345] (length 80). Keras has a build in processing function called `pad_sequences` to do this for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_SEQUENCE_LENGTH = 80\n",
    "cnn_data = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train and Validation Split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last thing we do is split our data into tranining and validation sets. Typically we will want a test set as well but we will skip this for demonstration purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "VALIDATION_SPLIT = .2\n",
    "num_validation_samples = int(VALIDATION_SPLIT * cnn_data.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "indices = np.arange(cnn_data.shape[0])\n",
    "np.random.shuffle(indices)\n",
    "cnn_data = cnn_data[indices]\n",
    "labels = labels[indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = cnn_data[:-num_validation_samples]\n",
    "y_train = labels[:-num_validation_samples]\n",
    "x_val = cnn_data[-num_validation_samples:]\n",
    "y_val = labels[-num_validation_samples:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building and Training our CNN Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we will load our pretrained word embeddings and build out CNN Model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Glove Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this example we will use the Twitter GloVe embeddings that can be found here https://nlp.stanford.edu/projects/glove/. We have our embeddings saved in a text file in our data directory so first we load parse these and load them in to a dictionary.\n",
    "\n",
    "Next we get the mean and standard devation of all embedding values. The pretrained GloVe embeddings won't contain all of the words in our vocabularly so we will seed our embedding matrix for our vocabularly with random draws from a normal distribution with mean *emb_mean* and standard deviation *emb_std*. Next we iterate through all of the words in our vocabulary and set our word embeddings to the GloVe vectors where they are available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "EMBEDDING_DIM = 50\n",
    "EMBEDDING_FILE = 'data/glove.twitter.27B.50d.txt'\n",
    "\n",
    "def get_embed_coefs(word, *arr): \n",
    "    return word, np.asarray(arr, dtype='float32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings_index = dict(get_embed_coefs(*o.rstrip().rsplit(' ')) for o in open(EMBEDDING_FILE))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings_values=list(embeddings_index.values())\n",
    "all_embs = np.stack(embeddings_values)\n",
    "emb_mean,emb_std = all_embs.mean(), all_embs.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_matrix = np.random.normal(emb_mean, emb_std, (len(word_index) + 1, EMBEDDING_DIM))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "for word, i in word_index.items():\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if embedding_vector is not None: \n",
    "        embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(31976, 50)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_matrix.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model and Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we build our Convoluational Neural Network using Keras. We will use the architecture from the Yoon Kim model (https://arxiv.org/abs/1408.5882) with some adjustments. We first start with our embeddings layer and then have 3 parallel 1D convulational layers with 128 filters and sizes [3, 4, 5] respectively. We concatenate these results, pass to a dropout layer for regularization, and then to a fully connected layer with relu activation and finally a softmax layer to make out predictions.\n",
    "\n",
    "Here we also define that we will use a categorical crossentropy loss funcation and an Adam optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ConvNet(embeddings, max_sequence_length, num_words, embedding_dim, labels_index, trainable=False, extra_conv=True):\n",
    "    \n",
    "    embedding_layer = Embedding(num_words,\n",
    "                            embedding_dim,\n",
    "                            weights=[embeddings],\n",
    "                            input_length=max_sequence_length,\n",
    "                            trainable=trainable)\n",
    "\n",
    "    sequence_input = Input(shape=(max_sequence_length,), dtype='int32')\n",
    "    embedded_sequences = embedding_layer(sequence_input)\n",
    "\n",
    "    # Yoon Kim model (https://arxiv.org/abs/1408.5882)\n",
    "    convs = []\n",
    "    filter_sizes = [3,4,5]\n",
    "\n",
    "    for filter_size in filter_sizes:\n",
    "        l_conv = Conv1D(filters=128, kernel_size=filter_size, activation='relu')(embedded_sequences)\n",
    "        l_pool = MaxPooling1D(pool_size=3)(l_conv)\n",
    "        convs.append(l_pool)\n",
    "\n",
    "    l_merge = Merge(mode='concat', concat_axis=1)(convs)\n",
    "\n",
    "    # add a 1D convnet with global maxpooling, instead of Yoon Kim model\n",
    "    conv = Conv1D(filters=128, kernel_size=3, activation='relu')(embedded_sequences)\n",
    "    pool = MaxPooling1D(pool_size=3)(conv)\n",
    "\n",
    "    if extra_conv==True:\n",
    "        x = Dropout(0.5)(l_merge)  \n",
    "    else:\n",
    "        # Original Yoon Kim model\n",
    "        x = Dropout(0.5)(pool)\n",
    "    x = Flatten()(x)\n",
    "    x = Dense(128, activation='relu')(x)\n",
    "    #x = Dropout(0.5)(x)\n",
    "\n",
    "    preds = Dense(labels_index, activation='softmax')(x)\n",
    "\n",
    "    model = Model(sequence_input, preds)\n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "                  optimizer='adam',\n",
    "                  metrics=['acc'])\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.5/dist-packages/ipykernel_launcher.py:21: UserWarning: The `Merge` layer is deprecated and will be removed after 08/2017. Use instead layers from `keras.layers.merge`, e.g. `add`, `concatenate`, etc.\n"
     ]
    }
   ],
   "source": [
    "model = ConvNet(embedding_matrix, MAX_SEQUENCE_LENGTH, len(word_index)+1, EMBEDDING_DIM, \n",
    "                len(labels[0]), False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and now we train!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 77574 samples, validate on 19393 samples\n",
      "Epoch 1/50\n",
      "77574/77574 [==============================] - 12s - loss: 0.6753 - acc: 0.5859 - val_loss: 0.6544 - val_acc: 0.6047\n",
      "Epoch 2/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.6345 - acc: 0.6348 - val_loss: 0.6146 - val_acc: 0.6532\n",
      "Epoch 3/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.6028 - acc: 0.6689 - val_loss: 0.5894 - val_acc: 0.6806\n",
      "Epoch 4/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.5756 - acc: 0.6942 - val_loss: 0.5797 - val_acc: 0.6917\n",
      "Epoch 5/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.5539 - acc: 0.7101 - val_loss: 0.5636 - val_acc: 0.6999\n",
      "Epoch 6/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.5345 - acc: 0.7219 - val_loss: 0.5570 - val_acc: 0.7073\n",
      "Epoch 7/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.5184 - acc: 0.7345 - val_loss: 0.5548 - val_acc: 0.7085\n",
      "Epoch 8/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.5021 - acc: 0.7469 - val_loss: 0.5630 - val_acc: 0.7026\n",
      "Epoch 9/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.4874 - acc: 0.7565 - val_loss: 0.5496 - val_acc: 0.7124\n",
      "Epoch 10/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.4739 - acc: 0.7652 - val_loss: 0.5515 - val_acc: 0.7129\n",
      "Epoch 11/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.4613 - acc: 0.7736 - val_loss: 0.5528 - val_acc: 0.7099\n",
      "Epoch 12/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.4490 - acc: 0.7805 - val_loss: 0.5545 - val_acc: 0.7110\n",
      "Epoch 13/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.4388 - acc: 0.7867 - val_loss: 0.5598 - val_acc: 0.7110\n",
      "Epoch 14/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.4262 - acc: 0.7957 - val_loss: 0.5577 - val_acc: 0.7103\n",
      "Epoch 15/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.4147 - acc: 0.8016 - val_loss: 0.5626 - val_acc: 0.7091\n",
      "Epoch 16/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.4066 - acc: 0.8061 - val_loss: 0.5721 - val_acc: 0.7082\n",
      "Epoch 17/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3935 - acc: 0.8117 - val_loss: 0.5746 - val_acc: 0.7129\n",
      "Epoch 18/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3832 - acc: 0.8180 - val_loss: 0.5828 - val_acc: 0.7137\n",
      "Epoch 19/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3772 - acc: 0.8221 - val_loss: 0.5809 - val_acc: 0.7141\n",
      "Epoch 20/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3720 - acc: 0.8254 - val_loss: 0.5847 - val_acc: 0.7085\n",
      "Epoch 21/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3646 - acc: 0.8298 - val_loss: 0.5838 - val_acc: 0.7073\n",
      "Epoch 22/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3575 - acc: 0.8341 - val_loss: 0.5841 - val_acc: 0.7103\n",
      "Epoch 23/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3514 - acc: 0.8380 - val_loss: 0.5921 - val_acc: 0.7118\n",
      "Epoch 24/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3466 - acc: 0.8396 - val_loss: 0.5975 - val_acc: 0.7066\n",
      "Epoch 25/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3424 - acc: 0.8423 - val_loss: 0.5915 - val_acc: 0.7112\n",
      "Epoch 26/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3368 - acc: 0.8449 - val_loss: 0.6010 - val_acc: 0.7091\n",
      "Epoch 27/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3331 - acc: 0.8474 - val_loss: 0.6044 - val_acc: 0.7111\n",
      "Epoch 28/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3263 - acc: 0.8495 - val_loss: 0.6139 - val_acc: 0.7096\n",
      "Epoch 29/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3217 - acc: 0.8543 - val_loss: 0.6137 - val_acc: 0.7047\n",
      "Epoch 30/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3163 - acc: 0.8550 - val_loss: 0.6122 - val_acc: 0.7059\n",
      "Epoch 31/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3172 - acc: 0.8552 - val_loss: 0.6215 - val_acc: 0.7074\n",
      "Epoch 32/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3096 - acc: 0.8585 - val_loss: 0.6237 - val_acc: 0.7090\n",
      "Epoch 33/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3030 - acc: 0.8628 - val_loss: 0.6231 - val_acc: 0.7080\n",
      "Epoch 34/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.3026 - acc: 0.8621 - val_loss: 0.6233 - val_acc: 0.7102\n",
      "Epoch 35/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.2999 - acc: 0.8643 - val_loss: 0.6312 - val_acc: 0.7129\n",
      "Epoch 36/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.2992 - acc: 0.8646 - val_loss: 0.6212 - val_acc: 0.7113\n",
      "Epoch 37/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.2923 - acc: 0.8695 - val_loss: 0.6390 - val_acc: 0.7023\n",
      "Epoch 38/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.2921 - acc: 0.8683 - val_loss: 0.6325 - val_acc: 0.7132\n",
      "Epoch 39/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.2904 - acc: 0.8700 - val_loss: 0.6335 - val_acc: 0.7070\n",
      "Epoch 40/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.2864 - acc: 0.8711 - val_loss: 0.6445 - val_acc: 0.7093\n",
      "Epoch 41/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.2833 - acc: 0.8720 - val_loss: 0.6386 - val_acc: 0.7074\n",
      "Epoch 42/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.2815 - acc: 0.8721 - val_loss: 0.6494 - val_acc: 0.7045\n",
      "Epoch 43/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.2768 - acc: 0.8752 - val_loss: 0.6433 - val_acc: 0.7026\n",
      "Epoch 44/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.2790 - acc: 0.8740 - val_loss: 0.6509 - val_acc: 0.7007\n",
      "Epoch 45/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.2750 - acc: 0.8764 - val_loss: 0.6492 - val_acc: 0.7077\n",
      "Epoch 46/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.2709 - acc: 0.8787 - val_loss: 0.6567 - val_acc: 0.7113\n",
      "Epoch 47/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.2699 - acc: 0.8785 - val_loss: 0.6500 - val_acc: 0.7094\n",
      "Epoch 48/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.2679 - acc: 0.8791 - val_loss: 0.6596 - val_acc: 0.7059\n",
      "Epoch 49/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.2660 - acc: 0.8801 - val_loss: 0.6665 - val_acc: 0.7064\n",
      "Epoch 50/50\n",
      "77574/77574 [==============================] - 9s - loss: 0.2662 - acc: 0.8799 - val_loss: 0.6540 - val_acc: 0.7039\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f2f2f562f28>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x_train, y_train, validation_data=(x_val, y_val), epochs=50, batch_size=128)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}