courses/machine_learning/deepdive/09_sequence/txtcls1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1> Text Classification using TensorFlow on Cloud ML Engine </h1>\n",
    "\n",
    "This notebook illustrates:\n",
    "<ol>\n",
    "<li> Creating datasets for Machine Learning using BigQuery\n",
    "<li> Creating a text classification model using a CNN and custom Estimator \n",
    "<li> Training on Cloud ML Engine\n",
    "<li> Deploying model\n",
    "<li> Predicting with model\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# change these to try this notebook out\n",
    "BUCKET = 'cloud-training-demos-ml'\n",
    "PROJECT = 'cloud-training-demos'\n",
    "REGION = 'us-central1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import os\n",
    "os.environ['BUCKET'] = BUCKET\n",
    "os.environ['PROJECT'] = PROJECT\n",
    "os.environ['REGION'] = REGION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%datalab project set -p $PROJECT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.2.1\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print tf.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The idea is to look at the title of a newspaper article and figure out whether the article came from the New York Times or from TechCrunch. There are very sophisticated approaches that we can try, but for now, let's go with something very simple.\n",
    "\n",
    "<h2> Data exploration and preprocessing in BigQuery </h2>\n",
    "<p>\n",
    "What does the Hacker News dataset look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div class=\"bqtv\" id=\"1_151508482315\"><table><tr><th>url</th><th>title</th><th>score</th></tr><tr><td></td><td>Ask HN: What's your speciality, and what's your \"FizzBuzz\" equivalent?</td><td>260</td></tr><tr><td></td><td>Ask HN: Books with a high signal to noise ratio?</td><td>260</td></tr><tr><td></td><td>Ask HN: Why can't I make as much as I make?</td><td>262</td></tr><tr><td></td><td>Ask HN: Is it OK to submit job postings on HN?</td><td>11</td></tr><tr><td></td><td>Ask HN: Can I help you be more awesome today? (No strings. Inquire within.)</td><td>11</td></tr><tr><td></td><td>Show HN: Free and Open Source book to teach Firefox OS app development</td><td>11</td></tr><tr><td>http://empowerunited.com/</td><td>Could this be the solution for the 99%?</td><td>11</td></tr><tr><td>https://github.com/Groundworkstech/Submicron</td><td>Deep-Submicron Backdoors</td><td>11</td></tr><tr><td>http://vancouver.en.craigslist.ca/van/roo/2035580033.html</td><td>Best Roommate Ad Ever</td><td>11</td></tr><tr><td>https://www.kickstarter.com/projects/carlosxcl/code-cards</td><td>Show HN: Code Cards, Like Texas hold 'em for people who want to code</td><td>11</td></tr></table></div>\n",
       "    <br />(rows: 10, time: 1.6s,   229MB processed, job: job_wB5DLHo6B1LFJbAGDkvBpx3KKJDO)<br />\n",
       "    <script>\n",
       "\n",
       "      require.config({\n",
       "        paths: {\n",
       "          d3: '//cdnjs.cloudflare.com/ajax/libs/d3/3.4.13/d3',\n",
       "          plotly: 'https://cdn.plot.ly/plotly-1.5.1.min.js?noext',\n",
       "          jquery: '//ajax.googleapis.com/ajax/libs/jquery/2.0.0/jquery.min'\n",
       "        },\n",
       "        map: {\n",
       "          '*': {\n",
       "            datalab: 'nbextensions/gcpdatalab'\n",
       "          }\n",
       "        },\n",
       "        shim: {\n",
       "          plotly: {\n",
       "            deps: ['d3', 'jquery'],\n",
       "            exports: 'plotly'\n",
       "          }\n",
       "        }\n",
       "      });\n",
       "\n",
       "      require(['datalab/charting', 'datalab/element!1_151508482315', 'base/js/events',\n",
       "          'datalab/style!/nbextensions/gcpdatalab/charting.css'],\n",
       "        function(charts, dom, events) {\n",
       "          charts.render('gcharts', dom, events, 'table', [], {\"rows\": [{\"c\": [{\"v\": \"\"}, {\"v\": \"Ask HN: What's your speciality, and what's your \\\"FizzBuzz\\\" equivalent?\"}, {\"v\": 260}]}, {\"c\": [{\"v\": \"\"}, {\"v\": \"Ask HN: Books with a high signal to noise ratio?\"}, {\"v\": 260}]}, {\"c\": [{\"v\": \"\"}, {\"v\": \"Ask HN: Why can't I make as much as I make?\"}, {\"v\": 262}]}, {\"c\": [{\"v\": \"\"}, {\"v\": \"Ask HN: Is it OK to submit job postings on HN?\"}, {\"v\": 11}]}, {\"c\": [{\"v\": \"\"}, {\"v\": \"Ask HN: Can I help you be more awesome today? (No strings. Inquire within.)\"}, {\"v\": 11}]}, {\"c\": [{\"v\": \"\"}, {\"v\": \"Show HN: Free and Open Source book to teach Firefox OS app development\"}, {\"v\": 11}]}, {\"c\": [{\"v\": \"http://empowerunited.com/\"}, {\"v\": \"Could this be the solution for the 99%?\"}, {\"v\": 11}]}, {\"c\": [{\"v\": \"https://github.com/Groundworkstech/Submicron\"}, {\"v\": \"Deep-Submicron Backdoors\"}, {\"v\": 11}]}, {\"c\": [{\"v\": \"http://vancouver.en.craigslist.ca/van/roo/2035580033.html\"}, {\"v\": \"Best Roommate Ad Ever\"}, {\"v\": 11}]}, {\"c\": [{\"v\": \"https://www.kickstarter.com/projects/carlosxcl/code-cards\"}, {\"v\": \"Show HN: Code Cards, Like Texas hold 'em for people who want to code\"}, {\"v\": 11}]}], \"cols\": [{\"type\": \"string\", \"id\": \"url\", \"label\": \"url\"}, {\"type\": \"string\", \"id\": \"title\", \"label\": \"title\"}, {\"type\": \"number\", \"id\": \"score\", \"label\": \"score\"}]},\n",
       "            {\n",
       "              pageSize: 25,\n",
       "              cssClassNames:  {\n",
       "                tableRow: 'gchart-table-row',\n",
       "                headerRow: 'gchart-table-headerrow',\n",
       "                oddTableRow: 'gchart-table-oddrow',\n",
       "                selectedTableRow: 'gchart-table-selectedrow',\n",
       "                hoverTableRow: 'gchart-table-hoverrow',\n",
       "                tableCell: 'gchart-table-cell',\n",
       "                headerCell: 'gchart-table-headercell',\n",
       "                rowNumberCell: 'gchart-table-rownumcell'\n",
       "              }\n",
       "            },\n",
       "            {source_index: 0, fields: 'url,title,score'},\n",
       "            0,\n",
       "            10);\n",
       "        }\n",
       "      );\n",
       "    </script>\n",
       "  "
      ],
      "text/plain": [
       "QueryResultsTable job_wB5DLHo6B1LFJbAGDkvBpx3KKJDO"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%bq query\n",
    "SELECT\n",
    "  url, title, score\n",
    "FROM\n",
    "  `bigquery-public-data.hacker_news.stories`\n",
    "WHERE\n",
    "  LENGTH(title) > 10\n",
    "  AND score > 10\n",
    "LIMIT 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do some regular expression parsing in BigQuery to get the source of the newspaper article from the URL. For example, if the url is http://mobile.nytimes.com/...., I want to be left with <i>nytimes</i>. To ensure that the parsing works for all URLs of interest, I'll group by the source to make sure there are no weird names left. This was an iterative process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "query=\"\"\"\n",
    "SELECT\n",
    "  ARRAY_REVERSE(SPLIT(REGEXP_EXTRACT(url, '.*://(.[^/]+)/'), '.'))[OFFSET(1)] AS source,\n",
    "  COUNT(title) AS num_articles\n",
    "FROM\n",
    "  `bigquery-public-data.hacker_news.stories`\n",
    "WHERE\n",
    "  REGEXP_CONTAINS(REGEXP_EXTRACT(url, '.*://(.[^/]+)/'), '.com$')\n",
    "  AND LENGTH(title) > 10\n",
    "GROUP BY\n",
    "  source\n",
    "ORDER BY num_articles DESC\n",
    "LIMIT 10\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source</th>\n",
       "      <th>num_articles</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>blogspot</td>\n",
       "      <td>41386</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>github</td>\n",
       "      <td>36525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>techcrunch</td>\n",
       "      <td>30891</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>youtube</td>\n",
       "      <td>30848</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>nytimes</td>\n",
       "      <td>28787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>medium</td>\n",
       "      <td>18422</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>google</td>\n",
       "      <td>18235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>wordpress</td>\n",
       "      <td>17667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>arstechnica</td>\n",
       "      <td>13749</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>wired</td>\n",
       "      <td>12841</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        source  num_articles\n",
       "0     blogspot         41386\n",
       "1       github         36525\n",
       "2   techcrunch         30891\n",
       "3      youtube         30848\n",
       "4      nytimes         28787\n",
       "5       medium         18422\n",
       "6       google         18235\n",
       "7    wordpress         17667\n",
       "8  arstechnica         13749\n",
       "9        wired         12841"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import google.datalab.bigquery as bq\n",
    "df = bq.Query(query).execute().result().to_dataframe()\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have good parsing of the URL to get the source, let's put together a dataset of source and titles. This will be our labeled dataset for machine learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>github</td>\n",
       "      <td>php bdd is now  nice</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>github</td>\n",
       "      <td>mpv video player 0.2 release</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>github</td>\n",
       "      <td>show hn  re-thinking the business card with dr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>github</td>\n",
       "      <td>update css js from chrome developer tool</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>github</td>\n",
       "      <td>simple way to start development with flask usi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   source                                              title\n",
       "0  github                              php bdd is now  nice \n",
       "1  github                       mpv video player 0.2 release\n",
       "2  github  show hn  re-thinking the business card with dr...\n",
       "3  github           update css js from chrome developer tool\n",
       "4  github  simple way to start development with flask usi..."
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query=\"\"\"\n",
    "SELECT source, LOWER(REGEXP_REPLACE(title, '[^a-zA-Z0-9 $.-]', ' ')) AS title FROM\n",
    "(SELECT\n",
    "  ARRAY_REVERSE(SPLIT(REGEXP_EXTRACT(url, '.*://(.[^/]+)/'), '.'))[OFFSET(1)] AS source,\n",
    "  title\n",
    "FROM\n",
    "  `bigquery-public-data.hacker_news.stories`\n",
    "WHERE\n",
    "  REGEXP_CONTAINS(REGEXP_EXTRACT(url, '.*://(.[^/]+)/'), '.com$')\n",
    "  AND LENGTH(title) > 10\n",
    ")\n",
    "WHERE (source = 'github' OR source = 'nytimes' OR source = 'techcrunch')\n",
    "\"\"\"\n",
    "df = bq.Query(query + \" LIMIT 10\").execute().result().to_dataframe()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For ML training, we will need to split our dataset into training and evaluation datasets (and perhaps an independent test dataset if we are going to do model or feature selection based on the evaluation dataset).  A simple way to do this is to use the hash of a well-distributed column in our data (See https://www.oreilly.com/learning/repeatable-sampling-of-data-sets-in-bigquery-for-machine-learning).\n",
    "<p>\n",
    "So, let's do that and save the results as CSV files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>github</td>\n",
       "      <td>this guy just found out how to bypass adblocker</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>github</td>\n",
       "      <td>show hn  dodo   command line task management f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>github</td>\n",
       "      <td>show hn  webservicemock   mock out external ca...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>github</td>\n",
       "      <td>magento category attributes dependency</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>github</td>\n",
       "      <td>write actionscript in swift whaa</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   source                                              title\n",
       "0  github    this guy just found out how to bypass adblocker\n",
       "1  github  show hn  dodo   command line task management f...\n",
       "2  github  show hn  webservicemock   mock out external ca...\n",
       "3  github             magento category attributes dependency\n",
       "4  github                  write actionscript in swift whaa "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "traindf = bq.Query(query + \" AND MOD(ABS(FARM_FINGERPRINT(title)),4) > 0\").execute().result().to_dataframe()\n",
    "evaldf  = bq.Query(query + \" AND MOD(ABS(FARM_FINGERPRINT(title)),4) = 0\").execute().result().to_dataframe()\n",
    "traindf.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "github        27445\n",
       "techcrunch    23131\n",
       "nytimes       21586\n",
       "Name: source, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "traindf['source'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "github        9080\n",
       "techcrunch    7760\n",
       "nytimes       7201\n",
       "Name: source, dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaldf['source'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import os, shutil\n",
    "DATADIR='data/txtcls'\n",
    "shutil.rmtree(DATADIR, ignore_errors=True)\n",
    "os.makedirs(DATADIR)\n",
    "traindf.to_csv( os.path.join(DATADIR,'train.csv'), header=False, index=False, encoding='utf-8', sep='\\t')\n",
    "evaldf.to_csv( os.path.join(DATADIR,'eval.csv'), header=False, index=False, encoding='utf-8', sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "github\tthis guy just found out how to bypass adblocker\r\n",
      "github\tshow hn  dodo   command line task management for developers\r\n",
      "github\tshow hn  webservicemock   mock out external calls for local development\r\n"
     ]
    }
   ],
   "source": [
    "!head -3 data/txtcls/train.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  24041 data/txtcls/eval.csv\r\n",
      "  72162 data/txtcls/train.csv\r\n",
      "  96203 total\r\n"
     ]
    }
   ],
   "source": [
    "!wc -l data/txtcls/*.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Copying file://data/txtcls/eval.csv [Content-Type=text/csv]...\n",
      "/ [0 files][    0.0 B/  1.4 MiB]                                                \r",
      "/ [1 files][  1.4 MiB/  1.4 MiB]                                                \r",
      "-\r",
      "Copying file://data/txtcls/train.csv [Content-Type=text/csv]...\n",
      "- [1 files][  1.4 MiB/  5.4 MiB]                                                \r",
      "\\\r",
      "\\ [2 files][  5.4 MiB/  5.4 MiB]                                                \r",
      "|\r\n",
      "Operation completed over 2 objects/5.4 MiB.                                      \n"
     ]
    }
   ],
   "source": [
    "%bash\n",
    "gsutil cp data/txtcls/*.csv gs://${BUCKET}/txtcls/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> TensorFlow code </h2>\n",
    "\n",
    "Please explore the code in this <a href=\"txtcls1/trainer\">directory</a> -- <a href=\"txtcls1/trainer/model.py\">model.py</a> contains the key TensorFlow model and <a href=\"txtcls1/trainer/task.py\">task.py</a> has a main() that launches off the training job.\n",
    "\n",
    "However, the following cells should give you an idea of what the model code does:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.2.1\n",
      "12 words into vocab.tsv\n",
      "some title --> [8 4]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.contrib import lookup\n",
    "from tensorflow.python.platform import gfile\n",
    "\n",
    "print tf.__version__\n",
    "MAX_DOCUMENT_LENGTH = 5  \n",
    "PADWORD = 'ZYXW'\n",
    "\n",
    "# vocabulary\n",
    "lines = ['Some title', 'A longer title', 'An even longer title', 'This is longer than doc length']\n",
    "lines = [line.lower() for line in lines]\n",
    "\n",
    "# create vocabulary\n",
    "vocab_processor = tf.contrib.learn.preprocessing.VocabularyProcessor(MAX_DOCUMENT_LENGTH)\n",
    "vocab_processor.fit(lines)\n",
    "with gfile.Open('vocab.tsv', 'wb') as f:\n",
    "    f.write(\"{}\\n\".format(PADWORD))\n",
    "    for word, index in vocab_processor.vocabulary_._mapping.iteritems():\n",
    "      f.write(\"{}\\n\".format(word))\n",
    "N_WORDS = len(vocab_processor.vocabulary_)\n",
    "print '{} words into vocab.tsv'.format(N_WORDS)\n",
    "\n",
    "# can use the vocabulary to convert words to numbers\n",
    "table = lookup.index_table_from_file(\n",
    "  vocabulary_file='vocab.tsv', num_oov_buckets=1, vocab_size=None, default_value=-1)\n",
    "numbers = table.lookup(tf.constant(lines[0].split()))\n",
    "with tf.Session() as sess:\n",
    "  tf.tables_initializer().run()\n",
    "  print \"{} --> {}\".format(lines[0], numbers.eval())   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ZYXW\r\n",
      "a\r\n",
      "even\r\n",
      "longer\r\n",
      "title\r\n",
      "this\r\n",
      "doc\r\n",
      "is\r\n",
      "some\r\n",
      "an\r\n",
      "length\r\n",
      "than\r\n",
      "<UNK>\r\n"
     ]
    }
   ],
   "source": [
    "!cat vocab.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "titles= ['some title' 'a longer title' 'an even longer title'\n",
      " 'this is longer than doc length'] (4,)\n",
      "words= SparseTensorValue(indices=array([[0, 0],\n",
      "       [0, 1],\n",
      "       [1, 0],\n",
      "       [1, 1],\n",
      "       [1, 2],\n",
      "       [2, 0],\n",
      "       [2, 1],\n",
      "       [2, 2],\n",
      "       [2, 3],\n",
      "       [3, 0],\n",
      "       [3, 1],\n",
      "       [3, 2],\n",
      "       [3, 3],\n",
      "       [3, 4],\n",
      "       [3, 5]]), values=array(['some', 'title', 'a', 'longer', 'title', 'an', 'even', 'longer',\n",
      "       'title', 'this', 'is', 'longer', 'than', 'doc', 'length'], dtype=object), dense_shape=array([4, 6]))\n",
      "dense= [['some' 'title' 'ZYXW' 'ZYXW' 'ZYXW' 'ZYXW']\n",
      " ['a' 'longer' 'title' 'ZYXW' 'ZYXW' 'ZYXW']\n",
      " ['an' 'even' 'longer' 'title' 'ZYXW' 'ZYXW']\n",
      " ['this' 'is' 'longer' 'than' 'doc' 'length']] (?, ?)\n",
      "numbers= [[ 8  4  0  0  0  0]\n",
      " [ 1  3  4  0  0  0]\n",
      " [ 9  2  3  4  0  0]\n",
      " [ 5  7  3 11  6 10]] (?, ?)\n",
      "padding= [[0 0]\n",
      " [0 5]] (2, 2)\n",
      "padded= [[ 8  4  0  0  0  0  0  0  0  0  0]\n",
      " [ 1  3  4  0  0  0  0  0  0  0  0]\n",
      " [ 9  2  3  4  0  0  0  0  0  0  0]\n",
      " [ 5  7  3 11  6 10  0  0  0  0  0]] (?, ?)\n",
      "sliced= [[ 8  4  0  0  0]\n",
      " [ 1  3  4  0  0]\n",
      " [ 9  2  3  4  0]\n",
      " [ 5  7  3 11  6]] (?, 5)\n"
     ]
    }
   ],
   "source": [
    "# string operations\n",
    "titles = tf.constant(lines)\n",
    "words = tf.string_split(titles)\n",
    "densewords = tf.sparse_tensor_to_dense(words, default_value=PADWORD)\n",
    "numbers = table.lookup(densewords)\n",
    "\n",
    "# now pad out with zeros and then slice to constant length\n",
    "padding = tf.constant([[0,0],[0,MAX_DOCUMENT_LENGTH]])\n",
    "padded = tf.pad(numbers, padding)\n",
    "sliced = tf.slice(padded, [0,0], [-1, MAX_DOCUMENT_LENGTH])\n",
    "\n",
    "with tf.Session() as sess:\n",
    "  tf.tables_initializer().run()\n",
    "  print \"titles=\", titles.eval(), titles.shape\n",
    "  print \"words=\", words.eval()\n",
    "  print \"dense=\", densewords.eval(), densewords.shape\n",
    "  print \"numbers=\", numbers.eval(), numbers.shape\n",
    "  print \"padding=\", padding.eval(), padding.shape\n",
    "  print \"padded=\", padded.eval(), padded.shape\n",
    "  print \"sliced=\", sliced.eval(), sliced.shape \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "def init(hparams):\n",
      "def save_vocab(trainfile, txtcolname, outfilename):\n",
      "def read_dataset(hparams, prefix, batch_size=BATCH_SIZE):\n",
      "def get_embedding(hparams, titles, embed_size):\n",
      "def cnn_model(features, labels, mode, params):\n",
      "def serving_input_fn():\n",
      "def get_train(hparams):\n",
      "def get_valid(hparams, batch_size):\n",
      "def make_experiment_fn(output_dir, hparams):\n"
     ]
    }
   ],
   "source": [
    "%bash\n",
    "grep \"^def\" txtcls1/trainer/model.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's make sure the code works locally on a small dataset for a few steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%bash\n",
    "echo \"bucket=${BUCKET}\"\n",
    "rm -rf outputdir\n",
    "export PYTHONPATH=${PYTHONPATH}:${PWD}/txtcls1\n",
    "python -m trainer.task \\\n",
    "   --bucket=${BUCKET} \\\n",
    "   --output_dir=outputdir \\\n",
    "   --job-dir=./tmp --train_steps=200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When I ran it, I got a 55% accuracy after a few steps. Because batchsize=32, 200 steps is essentially 6400 examples -- the full dataset is 72,000 examples, so this is not even the full dataset. And already, we are doing much better than random chance.\n",
    "<p>\n",
    "Once the code works in standalone mode, you can run it on Cloud ML Engine. You can monitor the job from the GCP console in the Cloud Machine Learning Engine section.  Since we have 72,000 examples and batchsize=32, train_steps=36,000 essentially means 16 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%bash\n",
    "OUTDIR=gs://${BUCKET}/txtcls1/trained_model\n",
    "JOBNAME=txtcls_$(date -u +%y%m%d_%H%M%S)\n",
    "echo $OUTDIR $REGION $JOBNAME\n",
    "gsutil -m rm -rf $OUTDIR\n",
    "gsutil cp txtcls1/trainer/*.py $OUTDIR\n",
    "gcloud ml-engine jobs submit training $JOBNAME \\\n",
    "   --region=$REGION \\\n",
    "   --module-name=trainer.task \\\n",
    "   --package-path=$(pwd)/txtcls1/trainer \\\n",
    "   --job-dir=$OUTDIR \\\n",
    "   --staging-bucket=gs://$BUCKET \\\n",
    "   --scale-tier=BASIC --runtime-version=1.2 \\\n",
    "   -- \\\n",
    "   --bucket=${BUCKET} \\\n",
    "   --output_dir=${OUTDIR} \\\n",
    "   --train_steps=36000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training finished with an accuracy of 73%.  Obviously, this was trained on a really small dataset and with more data will hopefully come even greater accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> Deploy trained model </h2>\n",
    "<p>\n",
    "Deploying the trained model to act as a REST web service is a simple gcloud call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://cloud-training-demos-ml/txtcls1/trained_model/export/Servo/\n",
      "gs://cloud-training-demos-ml/txtcls1/trained_model/export/Servo/1515090618/\n"
     ]
    }
   ],
   "source": [
    "%bash\n",
    "gsutil ls gs://${BUCKET}/txtcls1/trained_model/export/Servo/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%bash\n",
    "MODEL_NAME=\"txtcls\"\n",
    "MODEL_VERSION=\"v1\"\n",
    "MODEL_LOCATION=$(gsutil ls gs://${BUCKET}/txtcls1/trained_model/export/Servo/ | tail -1)\n",
    "echo \"Deleting and deploying $MODEL_NAME $MODEL_VERSION from $MODEL_LOCATION ... this will take a few minutes\"\n",
    "#gcloud ml-engine versions delete ${MODEL_VERSION} --model ${MODEL_NAME}\n",
    "#gcloud ml-engine models delete ${MODEL_NAME}\n",
    "#gcloud ml-engine models create ${MODEL_NAME} --regions $REGION\n",
    "gcloud ml-engine versions create ${MODEL_VERSION} --model ${MODEL_NAME} --origin ${MODEL_LOCATION}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> Use model to predict </h2>\n",
    "<p>\n",
    "Send a JSON request to the endpoint of the service to make it predict which publication the article is more likely to run in. These are actual titles of articles in the New York Times, github, and TechCrunch on June 19.   These titles were not part of the training or evaluation datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "          <script src=\"/static/components/requirejs/require.js\"></script>\n",
       "          <script>\n",
       "            requirejs.config({\n",
       "              paths: {\n",
       "                base: '/static/base',\n",
       "              },\n",
       "            });\n",
       "          </script>\n",
       "          "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "response={u'predictions': [{u'source': u'nytimes', u'prob': [0.8374465107917786, 0.04888102412223816, 0.11367242783308029], u'class': 0}, {u'source': u'github', u'prob': [0.012753208167850971, 0.987160325050354, 8.644349873065948e-05], u'class': 1}, {u'source': u'techcrunch', u'prob': [0.006752816028892994, 0.0005263009225018322, 0.992720901966095], u'class': 2}]}\n"
     ]
    }
   ],
   "source": [
    "from googleapiclient import discovery\n",
    "from oauth2client.client import GoogleCredentials\n",
    "import json\n",
    "\n",
    "credentials = GoogleCredentials.get_application_default()\n",
    "api = discovery.build('ml', 'v1', credentials=credentials,\n",
    "            discoveryServiceUrl='https://storage.googleapis.com/cloud-ml/discovery/ml_v1_discovery.json')\n",
    "\n",
    "request_data = {'instances':\n",
    "  [\n",
    "      {\n",
    "        'title': 'Supreme Court to Hear Major Case on Partisan Districts'.lower()\n",
    "      },\n",
    "      {\n",
    "        'title': 'Furan -- build and push Docker images from GitHub to target'.lower()\n",
    "      },\n",
    "      {\n",
    "        'title': 'Time Warner will spend $100M on Snapchat original shows and ads'.lower()\n",
    "      },\n",
    "  ]\n",
    "}\n",
    "\n",
    "parent = 'projects/%s/models/%s/versions/%s' % (PROJECT, 'txtcls', 'v1')\n",
    "response = api.projects().predict(body=request_data, name=parent).execute()\n",
    "print \"response={0}\".format(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the trained model predicts that the Supreme Court article is 84% likely to come from New York Times. The Docker article is 99% likely to be from GitHub according to the service and the Time Warner one is 99% likely to be from TechCrunch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2017 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}