xgboost/notebooks/Online Prediction with XGBoost.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Online Prediction with XGBoost on Cloud Machine Learning Engine\n",
    "This notebook uses the [Census Income Data Set](https://archive.ics.uci.edu/ml/datasets/Census+Income) to create a simple XGBoost model, upload the model to Cloud Machine Learning Engine (ML Engine), and query it for predictions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to bring your model to ML Engine\n",
    "Getting your model ready for predictions can be done in 5 steps:\n",
    "1. Save your model to a file\n",
    "2. Upload the saved model to [Google Cloud Storage](https://cloud.google.com/storage)\n",
    "3. Create a model resource on ML Engine\n",
    "4. Create a model version (linking your XGBoost model)\n",
    "5. Make an online prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prerequisites\n",
    "Before we begin, let’s cover some of the different tools you’ll use to get online prediction up and running on ML Engine. \n",
    "\n",
    "[Google Cloud Platform](https://cloud.google.com/) (GCP) lets you build and host applications and websites, store data, and analyze data on Google's scalable infrastructure.\n",
    "\n",
    "[Cloud ML Engine](https://cloud.google.com/ml-engine/) is a managed service that enables you to easily build machine learning models that work on any type of data, of any size.\n",
    "\n",
    "[Google Cloud Storage](https://cloud.google.com/storage/) (GCS) is a unified object storage for developers and enterprises, from live data serving to data analytics/ML to data archiving.\n",
    "\n",
    "[Cloud SDK](https://cloud.google.com/sdk/) is a command line tool which allows you to interact with Google Cloud products. In order to run this notebook, make sure that Cloud SDK is [installed](https://cloud.google.com/sdk/downloads) in the same environment as your Jupyter kernel.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 0: Setup\n",
    "* [Create a project on GCP](https://cloud.google.com/resource-manager/docs/creating-managing-projects)\n",
    "* [Create a GCS Bucket](https://cloud.google.com/storage/docs/quickstart-console)\n",
    "* [Enable Cloud Machine Learning Engine and Compute Engine APIs](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com,compute_component&_ga=2.217405014.1312742076.1516128282-1417583630.1516128282)\n",
    "* [Install Cloud SDK](https://cloud.google.com/sdk/downloads)\n",
    "* [Install XGBoost](http://xgboost.readthedocs.io/en/latest/build.html)\n",
    "* [Install scikit-learn](http://scikit-learn.org/stable/install.html)\n",
    "* [Install NumPy](https://docs.scipy.org/doc/numpy/user/install.html)\n",
    "* [Install pandas](https://pandas.pydata.org/pandas-docs/stable/install.html)\n",
    "* [Install Google API Python Client](https://github.com/google/google-api-python-client)\n",
    "\n",
    "\n",
    "These variables will be needed for the following steps.\n",
    "\n",
    "In the cell below, **replace** the following highlighted elements:\n",
    "* `<YOUR_PROJECT_ID>` - with your project's id. Use the PROJECT_ID that matches your Google Cloud Platform project.\n",
    "* `<YOUR_BUCKET_ID>` - with the bucket id you created above.\n",
    "* `<YOUR_MODEL_NAME>` - with your model name, such as \"census\"\n",
    "* `<YOUR_VERSION>` - with your version name, such as \"v1\"\n",
    "* `<ML_ENGINE_REGION>` - select a region from [here](https://cloud.google.com/ml-engine/docs/regions) or use the default 'us-central1'. The region is where the model will be deployed.\n",
    "* `data.json` - a JSON file that contains the data used as input to your model’s predict method. (You'll create this in the code below)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: PROJECT_ID=<YOUR_PROJECT_ID>\n",
      "env: BUCKET_ID=<YOUR_BUCKET_ID>\n",
      "env: MODEL_NAME=<YOUR_MODEL_NAME>\n",
      "env: VERSION_NAME=<YOUR_VERSION>\n",
      "env: REGION=<ML_ENGINE_REGION>\n",
      "env: INPUT_DATA_FILE=data.json\n"
     ]
    }
   ],
   "source": [
    "%env PROJECT_ID <YOUR_PROJECT_ID>\n",
    "%env BUCKET_ID <YOUR_BUCKET_ID>\n",
    "%env MODEL_NAME <YOUR_MODEL_NAME>\n",
    "%env VERSION_NAME <YOUR_VERSION>\n",
    "%env REGION <ML_ENGINE_REGION>\n",
    "%env INPUT_DATA_FILE data.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download the data\n",
    "The [Census Income Data Set](https://archive.ics.uci.edu/ml/datasets/Census+Income) that this sample\n",
    "uses for training is hosted by the [UC Irvine Machine Learning\n",
    "Repository](https://archive.ics.uci.edu/ml/datasets/):\n",
    "\n",
    " * Training file is `adult.data`\n",
    " * Evaluation file is `adult.test`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Disclaimer\n",
    "This dataset is provided by a third party. Google provides no representation,\n",
    "warranty, or other guarantees about the validity or any other aspects of this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: cannot create directory ‘census_data’: File exists\n",
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 3881k  100 3881k    0     0  4617k      0 --:--:-- --:--:-- --:--:-- 4614k\n",
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 1956k  100 1956k    0     0  1885k      0  0:00:01  0:00:01 --:--:-- 1886k\n"
     ]
    }
   ],
   "source": [
    "# Create a directory to hold the data\n",
    "! mkdir census_data\n",
    "\n",
    "# Download the data\n",
    "! curl https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data --output census_data/adult.data\n",
    "! curl https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test --output census_data/adult.test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Train/Save the model \n",
    "First, the data is loaded into a pandas DataFrame. Then a simple model is created with the training set. Lastly, the model is saved to a .bst file that can then be uploaded to ML Engine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data loading complete\n"
     ]
    }
   ],
   "source": [
    "import xgboost as xgb\n",
    "import pandas as pd\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "# these are the column labels from the census data files\n",
    "COLUMNS = (\n",
    "    'age',\n",
    "    'workclass',\n",
    "    'fnlwgt',\n",
    "    'education',\n",
    "    'education-num',\n",
    "    'marital-status',\n",
    "    'occupation',\n",
    "    'relationship',\n",
    "    'race',\n",
    "    'sex',\n",
    "    'capital-gain',\n",
    "    'capital-loss',\n",
    "    'hours-per-week',\n",
    "    'native-country',\n",
    "    'income-level'\n",
    ")\n",
    "\n",
    "# categorical columns contain data that need to be turned into numerical values before being used by XGBoost\n",
    "CATEGORICAL_COLUMNS = (\n",
    "    'workclass',\n",
    "    'education',\n",
    "    'marital-status',\n",
    "    'occupation',\n",
    "    'relationship',\n",
    "    'race',\n",
    "    'sex',\n",
    "    'native-country'\n",
    ")\n",
    "\n",
    "# load training set\n",
    "with open('./census_data/adult.data', 'r') as train_data:\n",
    "    raw_training_data = pd.read_csv(train_data, header=None, names=COLUMNS)\n",
    "# remove column we are trying to predict ('income-level') from features list\n",
    "train_features = raw_training_data.drop('income-level', axis=1)\n",
    "# create training labels list\n",
    "train_labels = (raw_training_data['income-level'] == ' >50K')\n",
    "\n",
    "\n",
    "# load test set\n",
    "with open('./census_data/adult.test', 'r') as test_data:\n",
    "    raw_testing_data = pd.read_csv(test_data, names=COLUMNS, skiprows=1)\n",
    "# remove column we are trying to predict ('income-level') from features list\n",
    "test_features = raw_testing_data.drop('income-level', axis=1)\n",
    "# create training labels list\n",
    "test_labels = (raw_testing_data['income-level'] == ' >50K.')\n",
    "\n",
    "# convert data in categorical columns to numerical values\n",
    "encoders = {col:LabelEncoder() for col in CATEGORICAL_COLUMNS}\n",
    "for col in CATEGORICAL_COLUMNS:\n",
    "    train_features[col] = encoders[col].fit_transform(train_features[col])\n",
    "for col in CATEGORICAL_COLUMNS:\n",
    "    test_features[col] = encoders[col].fit_transform(test_features[col])\n",
    "\n",
    "# load data into DMatrix object\n",
    "dtrain = xgb.DMatrix(train_features, train_labels)\n",
    "dtest = xgb.DMatrix(test_features)\n",
    "\n",
    "print 'data loading complete'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model trained and saved\n"
     ]
    }
   ],
   "source": [
    "# train XGBoost model\n",
    "bst = xgb.train({}, dtrain, 20)\n",
    "bst.save_model('./model.bst')\n",
    "\n",
    "print 'model trained and saved'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2: Upload the model\n",
    "To use your model with ML Engine, it needs to be uploaded to Google Cloud Storage (GCS). So next, we'll take your local ‘model.bst’ file and upload it to GCS via the Cloud SDK using gsutil.\n",
    "\n",
    "Before continuing, make sure you're [properly authenticated](https://cloud.google.com/sdk/gcloud/reference/auth/) and have [access to the bucket](https://cloud.google.com/storage/docs/access-control/). This next command sets your project to the one specified above.\n",
    "\n",
    "Note: If you get an error below, make sure the Cloud SDK is installed in the kernel's environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\r\n"
     ]
    }
   ],
   "source": [
    "! gcloud config set project $PROJECT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: The exact file name of of the exported model you upload to GCS is important! Your model must be named “model.joblib”, “model.pkl”, or “model.bst” with respect to the library you used to export it. This restriction ensures that the model will be safely reconstructed later by using the same technique for import as was used during export."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying file://./model.bst [Content-Type=application/octet-stream]...\n",
      "/ [1 files][ 77.8 KiB/ 77.8 KiB]                                                \n",
      "Operation completed over 1 objects/77.8 KiB.                                     \n"
     ]
    }
   ],
   "source": [
    "! gsutil cp ./model.bst gs://$BUCKET_ID/model.bst"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3: Create a model resource\n",
    "Cloud ML Engine organizes your trained models using model and version resources. A Cloud ML Engine model is a container for the versions of your machine learning model. For more information on model resources and model versions look [here](https://cloud.google.com/ml-engine/docs/deploying-models#creating_a_model_version). \n",
    "\n",
    "At this step, you create a container that you can use to hold several different versions of your actual model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created ml engine model [projects/demos-191212/models/census].\r\n"
     ]
    }
   ],
   "source": [
    "! gcloud ml-engine models create $MODEL_NAME --regions $REGION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4: Create a model version\n",
    "\n",
    "Now it’s time to get your model online and ready for predictions. The model version requires a few components as specified [here](https://cloud.google.com/ml-engine/reference/rest/v1/projects.models.versions#Version).\n",
    "\n",
    "* __name__ - The name specified for the version when it was created. This will be the `VERSION_NAME` variable you declared at the beginning.\n",
    "* __deployment Uri__ (curl) - The Google Cloud Storage location of the trained model used to create the version. This is the bucket that you uploaded the model to with your `BUCKET_ID`\n",
    "* __runtime__ version - The Google Cloud ML runtime version to use for this deployment. This is set to 1.4\n",
    "* __framework__ - The framework specifies if you are using: `TENSORFLOW`, `SCIKIT_LEARN`, `XGBOOST`. This is set to `XGBOOST`\n",
    "\n",
    "Note: Runtime version 1.5 uses XGBoost 0.7. Runtime version 1.4 uses XGBoost 0.6. Please refer to the [runtime version dependency list](https://cloud.google.com/ml-engine/docs/runtime-version-list).\n",
    "\n",
    "Note: It can take several minutes for you model to be available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "  \"name\": \"projects/demos-191212/operations/create_census_v1-1519973406339\",\r\n",
      "  \"metadata\": {\r\n",
      "    \"@type\": \"type.googleapis.com/google.cloud.ml.v1.OperationMetadata\",\r\n",
      "    \"createTime\": \"2018-03-02T06:50:06Z\",\r\n",
      "    \"operationType\": \"CREATE_VERSION\",\r\n",
      "    \"modelName\": \"projects/demos-191212/models/census\",\r\n",
      "    \"version\": {\r\n",
      "      \"name\": \"projects/demos-191212/models/census/versions/v1\",\r\n",
      "      \"deploymentUri\": \"gs://xgboost-tutorial-notebook-test/\",\r\n",
      "      \"createTime\": \"2018-03-02T06:50:06Z\",\r\n",
      "      \"runtimeVersion\": \"1.4\",\r\n",
      "      \"framework\": \"XGBOOST\"\r\n",
      "    }\r\n",
      "  }\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "! curl -X POST -H \"Content-Type: application/json\" \\\n",
    "   -d '{\"name\": \"'$VERSION_NAME'\", \"deploymentUri\": \"gs://'$BUCKET_ID'/\", \"runtimeVersion\": \"1.4\", \"framework\": \"XGBOOST\"}' \\\n",
    "   -H \"Authorization: Bearer `gcloud auth print-access-token`\" \\\n",
    "    https://ml.googleapis.com/v1/projects/$PROJECT_ID/models/$MODEL_NAME/versions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 5: Make an online prediction\n",
    "It’s time to make an online prediction with your newly deployed model. Before you begin, you'll need to take some of the test data and prepare it, so that the test data can be used by the deployed model.\n",
    "\n",
    "To get online predictions, the data needs to be converted from a numpy array to a json array. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Show a person that makes <=50K:\n",
      "\tFeatures: [25, 4, 226802, 1, 7, 4, 7, 3, 2, 1, 0, 0, 40, 38] --> Label: False\n",
      "\n",
      "Show a person that makes >50K:\n",
      "\tFeatures: [28, 2, 336951, 7, 12, 2, 11, 0, 4, 1, 0, 0, 40, 38] --> Label: True\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import numpy as np\n",
    "\n",
    "data = []\n",
    "for i in range(len(test_features)):\n",
    "  data.append([])\n",
    "  for col in COLUMNS[:-1]: # ignore 'income-level' column as it isn't in feature set.\n",
    "    # convert from numpy integers to standard integers\n",
    "    data[i].append(int(np.uint64(test_features[col][i]).item()))\n",
    "\n",
    "# write the test data to a json file\n",
    "with open('data.json', 'w') as outfile:\n",
    "  json.dump(data, outfile)\n",
    "\n",
    "# get one person that makes <=50K and one that makes >50K to test our model.\n",
    "print('Show a person that makes <=50K:')\n",
    "print('\\tFeatures: {0} --> Label: {1}\\n'.format(data[0], test_labels[0]))\n",
    "\n",
    "with open('less_than_50K.json', 'w') as outfile:\n",
    "  json.dump(data[0], outfile)\n",
    "\n",
    "  \n",
    "print('Show a person that makes >50K:')\n",
    "print('\\tFeatures: {0} --> Label: {1}'.format(data[2], test_labels[2]))\n",
    "\n",
    "with open('more_than_50K.json', 'w') as outfile:\n",
    "  json.dump(data[2], outfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use Gcloud to make online predictions\n",
    "Use the two people (as seen in the table) gathered in the previous step for the gcloud predictions.\n",
    "\n",
    "| **Person** | age | workclass | fnlwgt | education | education-num | marital-status | occupation |\n",
    "|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\n",
    "| **1** | 25| 4 | 226802 | 1 | 7 | 4 | 7 |\n",
    "| **2** | 28| 2 | 336951 | 7 | 12 | 2 | 11 |\n",
    "\n",
    "| **Person** | relationship | race | sex | capital-gain | capital-loss | hours-per-week | native-country || (Label) income-level|\n",
    "|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:||:-:\n",
    "| **1** | 3 | 2 | 1 | 0 | 0 | 40 | 38 || False (<=50K) |\n",
    "| **2** | 0 | 4 | 1 | 0 | 0 | 40 | 38 || True (>50K) |\n",
    "\n",
    "\n",
    "Creating a model version can take several minutes, check the status of your model version to see if it is available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME  DEPLOYMENT_URI                        STATE\r\n",
      "v1    gs://xgboost-tutorial-notebook-test/  CREATING\r\n"
     ]
    }
   ],
   "source": [
    "! gcloud ml-engine versions list --model $MODEL_NAME"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME  DEPLOYMENT_URI                        STATE\r\n",
      "v1    gs://xgboost-tutorial-notebook-test/  READY\r\n"
     ]
    }
   ],
   "source": [
    "! gcloud ml-engine versions list --model $MODEL_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test the model with an online prediction using the data of a person who makes <=50K.\n",
    "\n",
    "Note: If you see an error, the model from Part 4 may not be created yet as it takes several minutes for a new model version to be created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.09139516949653625]\r\n"
     ]
    }
   ],
   "source": [
    "! gcloud ml-engine predict --model $MODEL_NAME --version $VERSION_NAME --json-instances less_than_50K.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test the model with an online prediction using the data of a person who makes >50K."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.4006653428077698]\r\n"
     ]
    }
   ],
   "source": [
    "! gcloud ml-engine predict --model $MODEL_NAME --version $VERSION_NAME --json-instances more_than_50K.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Realise how the cells above return floats instead of booleans. Let's deal with that below so the output type of the predictions match those of the test set labels. We'll set the prediction to True if it's greater than 0.5 and to False otherwise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use Python to make online predictions\n",
    "We'll test the model with the entire test set and print out some of the results.\n",
    "\n",
    "Note: If running notebook server on Compute Engine, make sure to [\"allow full access to all Cloud APIs\"](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#changeserviceaccountandscopes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction: False\tLabel: False\n",
      "Prediction: False\tLabel: False\n",
      "Prediction: False\tLabel: True\n",
      "Prediction: True\tLabel: True\n",
      "Prediction: False\tLabel: False\n"
     ]
    }
   ],
   "source": [
    "import googleapiclient.discovery\n",
    "import os\n",
    "\n",
    "PROJECT_ID = os.environ['PROJECT_ID']\n",
    "VERSION_NAME = os.environ['VERSION_NAME']\n",
    "MODEL_NAME = os.environ['MODEL_NAME']\n",
    "\n",
    "service = googleapiclient.discovery.build('ml', 'v1')\n",
    "name = 'projects/{}/models/{}'.format(PROJECT_ID, MODEL_NAME)\n",
    "name += '/versions/{}'.format(VERSION_NAME)\n",
    "\n",
    "response = service.projects().predict(\n",
    "    name=name,\n",
    "    body={'instances': data}\n",
    ").execute()\n",
    "\n",
    "if 'error' in response:\n",
    "  print (response['error'])\n",
    "else:\n",
    "  online_results = response['predictions']\n",
    "  # convert floats to booleans\n",
    "  converted_responses = [x > 0.5 for x in online_results]\n",
    "  # Print the first 10 responses\n",
    "  for i, response in enumerate(converted_responses[:5]):\n",
    "    print('Prediction: {}\\tLabel: {}'.format(response, test_labels[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Optional] Part 6: Verify Results\n",
    "Let's visualise our predictions with a confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>online</th>\n",
       "      <th>False</th>\n",
       "      <th>True</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>actual</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>False</th>\n",
       "      <td>11842</td>\n",
       "      <td>593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>True</th>\n",
       "      <td>1520</td>\n",
       "      <td>2326</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "online  False  True \n",
       "actual              \n",
       "False   11842    593\n",
       "True     1520   2326"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "actual = pd.Series(test_labels, name='actual')\n",
    "online = pd.Series(converted_responses, name='online')\n",
    "\n",
    "pd.crosstab(actual,online)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare this with the confusion matrix of our local model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>local</th>\n",
       "      <th>False</th>\n",
       "      <th>True</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>actual</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>False</th>\n",
       "      <td>11842</td>\n",
       "      <td>593</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>True</th>\n",
       "      <td>1520</td>\n",
       "      <td>2326</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "local   False  True \n",
       "actual              \n",
       "False   11842    593\n",
       "True     1520   2326"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "local_results = bst.predict(dtest)\n",
    "converted_local = [x > 0.5 for x in local_results]\n",
    "local = pd.Series(converted_local, name='local')\n",
    "\n",
    "pd.crosstab(actual,local)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Better, let's compare the raw results (pre-boolean-conversion) of our local and online models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "identical: 16281, different: 0\n"
     ]
    }
   ],
   "source": [
    "identical = 0\n",
    "different = 0\n",
    "\n",
    "for i in xrange(len(online_results)):\n",
    "    if online_results[i] == local_results[i]:\n",
    "        identical += 1\n",
    "    else:\n",
    "        different += 1\n",
    "        \n",
    "print 'identical: {}, different: {}'.format(identical,different)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If all results are identical, it means we've successfully uploaded our local model to ML Engine and performed online predictions correctly."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}