autopilot_customer_churn_high_level_with_evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of Autopilot candidates to evaluate and run batch transform jobs.\n",
    "# Make sure that you do not put a larger TOP_N_CANDIDATES than the Batch Transform limit for ml.m5.xlarge instances in your account.\n",
    "TOP_N_CANDIDATES = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Top Candidates Customer Churn Prediction with Amazon SageMaker Autopilot and Batch Transform (Python SDK)\n",
    "_**Using AutoPilot to Predict Mobile Customer Departure and Custom Evaluations with the Python SDK**_\n",
    "\n",
    "---\n",
    "\n",
    "---\n",
    "\n",
    "This notebook works with sagemaker python sdk 2.x\n",
    "\n",
    "## Contents\n",
    "\n",
    "1. [Introduction](#Introduction)\n",
    "1. [Setup](#Setup)\n",
    "1. [Data](#Data)\n",
    "1. [Train](#Settingup)\n",
    "1. [Autopilot Results](#Results)\n",
    "1. [Evaluate Top Candidates](#Evaluation)\n",
    "1. [Cleanup](#Cleanup)\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "## Introduction\n",
    "\n",
    "Amazon SageMaker Autopilot is an automated machine learning (commonly referred to as AutoML) solution for tabular datasets. You can use SageMaker Autopilot in different ways: on autopilot (hence the name) or with human guidance, without code through SageMaker Studio, or using the AWS SDKs. This notebook, as a first glimpse, will use the AWS SDKs to simply create and deploy a machine learning model.\n",
    "\n",
    "With the new feature released on 6/19/2020 (sagemaker python sdk >= 1.65.1), Autopilot generated models can be configured to return probabilities of each inference. This will enable data scientists to further analyze the performance of the model with additional metrics and to choose an appropriate threshold as operating point. This notebook demonstrate how to configure the model to obtain the inference probability (see [Customize the Inference Response](#Customize-the-Inference-Response)), select top N models, and make Batch Transform on a hold-out test set for evaluation. Note that this feature only works with Autopilot models created after the release of the new feature, i.e. after 6/19/2020, you will have to rerun Autopilot jobs to recreate the models in order to obtain the probability score during inference.\n",
    "\n",
    "Losing customers is costly for any business.  Identifying unhappy customers early on gives you a chance to offer them incentives to stay.  This notebook describes using machine learning (ML) for the automated identification of unhappy customers, also known as customer churn prediction. ML models rarely give perfect predictions though, so this notebook is also about how to incorporate the relative costs of prediction mistakes when determining the financial outcome of using ML.\n",
    "\n",
    "We use an example of churn that is familiar to all of us–leaving a mobile phone operator.  Seems like I can always find fault with my provider du jour! And if my provider knows that I’m thinking of leaving, it can offer timely incentives–I can always use a phone upgrade or perhaps have a new feature activated–and I might just stick around. Incentives are often much more cost effective than losing and reacquiring a customer.\n",
    "\n",
    "In this notebook, we use the high-level [SageMaker Python SDK](https://github.com/aws/sagemaker-python-sdk). There's also [another example notebook](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/autopilot/autopilot_customer_churn.ipynb) with the same Customer Churn dataset that uses the [low-level Boto3 SDK](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html)\n",
    "\n",
    "---\n",
    "## Setup\n",
    "\n",
    "_This notebook was created and tested on an ml.t3.medium notebook instance._\n",
    "\n",
    "Let's start by specifying:\n",
    "\n",
    "- The S3 bucket and prefix that you want to use for training and model data.  This should be within the same region as the Notebook Instance, training, and hosting.\n",
    "- The IAM role arn used to give training and hosting access to your data. See the documentation for how to create these.  Note, if more than one role is required for notebook instances, training, and/or hosting, please replace the boto regexp with a the appropriate full IAM role arn string(s).\n",
    "- The number of top candidates (`TOP_N_CANDIDATES`) to explore and compare with different metrics  (AUC, Precision-recall and ROC) on the heldout data split. We demonstrate how to compute these metrics for each candidate by running transform jobs. (Note that `TOP_N_CANDIDATES` was defined in the first cell of this notebook). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to get the latest sagemaker python sdk\n",
    "!pip install -U sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "isConfigCell": true,
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "print(sagemaker.__version__)\n",
    "\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "session = sagemaker.Session()\n",
    "\n",
    "# You can modify the following to use a bucket of your choosing\n",
    "bucket = session.default_bucket()\n",
    "prefix = 'sagemaker/DEMO-autopilot-churn'\n",
    "\n",
    "role = get_execution_role()\n",
    "\n",
    "sm = boto3.Session().client(service_name='sagemaker', region_name=region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll import the Python libraries we'll need for the remainder of the exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sagemaker import AutoML\n",
    "from time import gmtime, strftime, sleep\n",
    "from pprint import pprint\n",
    "import json\n",
    "import io\n",
    "from urllib.parse import urlparse\n",
    "from sklearn.metrics import roc_auc_score, roc_curve, precision_recall_curve, classification_report, average_precision_score, precision_score, recall_score\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Data\n",
    "\n",
    "Mobile operators have historical records on which customers ultimately ended up churning and which continued using the service. We can use this historical information to construct an ML model of one mobile operator’s churn using a process called training. After training the model, we can pass the profile information of an arbitrary customer (the same profile information that we used to train the model) to the model, and have the model predict whether this customer is going to churn. Of course, we expect the model to make mistakes–after all, predicting the future is tricky business! But I’ll also show how to deal with prediction errors.\n",
    "\n",
    "The dataset we will use is synthetically generated, but indictive of the types of features you'd see in this use case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 cp s3://sagemaker-sample-files/datasets/tabular/synthetic/churn.txt ./"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspect your Dataset\n",
    "\n",
    "Before you run Autopilot on the dataset, first perform a check of the dataset to make sure that it has no obvious errors. The Autopilot process can take long time, and it's generally a good practice to inspect the dataset before you start a job. This particular dataset is small, so you can inspect it in the notebook instance itself. If you have a larger dataset that will not fit in a notebook instance memory, inspect the dataset offline using a big data analytics tool like Apache Spark. [Deequ](https://github.com/awslabs/deequ) is a library built on top of Apache Spark that can be helpful for performing checks on large datasets. Autopilot is capable of handling datasets up to 5 GB.\n",
    "\n",
    "Read the data into a Pandas data frame and take a look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "churn = pd.read_csv('./churn.txt')\n",
    "pd.set_option('display.max_columns', 500)\n",
    "churn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By modern standards, it’s a relatively small dataset, with only 5,000 records, where each record uses 21 attributes to describe the profile of a customer of an unknown US mobile operator. The attributes are:\n",
    "\n",
    "- `State`: the US state in which the customer resides, indicated by a two-letter abbreviation; for example, OH or NJ\n",
    "- `Account Length`: the number of days that this account has been active\n",
    "- `Area Code`: the three-digit area code of the corresponding customer’s phone number\n",
    "- `Phone`: the remaining seven-digit phone number\n",
    "- `Int’l Plan`: whether the customer has an international calling plan: yes/no\n",
    "- `VMail Plan`: whether the customer has a voice mail feature: yes/no\n",
    "- `VMail Message`: presumably the average number of voice mail messages per month\n",
    "- `Day Mins`: the total number of calling minutes used during the day\n",
    "- `Day Calls`: the total number of calls placed during the day\n",
    "- `Day Charge`: the billed cost of daytime calls\n",
    "- `Eve Mins, Eve Calls, Eve Charge`: the billed cost for calls placed during the evening\n",
    "- `Night Mins`, `Night Calls`, `Night Charge`: the billed cost for calls placed during nighttime\n",
    "- `Intl Mins`, `Intl Calls`, `Intl Charge`: the billed cost for international calls\n",
    "- `CustServ Calls`: the number of calls placed to Customer Service\n",
    "- `Churn?`: whether the customer left the service: true/false\n",
    "\n",
    "The last attribute, `Churn?`, is known as the target attribute–the attribute that we want the ML model to predict."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reserve some data for calling inference on the model\n",
    "\n",
    "Divide the data into training and testing splits. The training split is used by SageMaker Autopilot. The testing split is reserved to perform inference using the suggested model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = churn.sample(frac=0.8,random_state=200)\n",
    "\n",
    "test_data = churn.drop(train_data.index)\n",
    "\n",
    "test_data_no_target = test_data.drop(columns=['Churn?'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's save these train and test data splits to csv files locally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_file = 'train_data.csv'\n",
    "train_data.to_csv(train_file, index=False, header=True)\n",
    "\n",
    "test_file = 'test_data.csv'\n",
    "test_data_no_target.to_csv(test_file, index=False, header=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up the SageMaker Autopilot Job<a name=\"Settingup\"></a>\n",
    "\n",
    "We will use the `AutoML` estimator from SageMaker Python SDK to invoke Autopilot to find the best ML pipeline to train a model on this dataset. \n",
    "\n",
    "The required inputs for invoking a Autopilot job are:\n",
    "* local or s3 location for input dataset (if local, the dataset will be uploaded to s3)\n",
    "* Name of the column of the dataset you want to predict (`Churn?` in this case) \n",
    "* An IAM role\n",
    "\n",
    "Currently Autopilot supports only tabular datasets in CSV format. Either all files should have a header row, or the first file of the dataset, when sorted in alphabetical/lexical order by name, is expected to have a header row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import AutoML\n",
    "from time import gmtime, strftime, sleep\n",
    "\n",
    "timestamp_suffix = strftime('%d-%H-%M-%S', gmtime())\n",
    "base_job_name = 'automl-churn-sdk-' + timestamp_suffix\n",
    "\n",
    "target_attribute_name = 'Churn?'\n",
    "target_attribute_values = np.unique(train_data[target_attribute_name])\n",
    "target_attribute_true_value = target_attribute_values[1] # 'True.'\n",
    "\n",
    "automl = AutoML(role=role,\n",
    "                target_attribute_name=target_attribute_name,\n",
    "                base_job_name=base_job_name,\n",
    "                sagemaker_session=session,\n",
    "                max_candidates=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also specify the type of problem you want to solve with your dataset (`Regression, MulticlassClassification, BinaryClassification`) with the `problem_type` keywork argument. In case you are not sure, SageMaker Autopilot will infer the problem type based on statistics of the target column (the column you want to predict). \n",
    "\n",
    "Because the target attribute, ```Churn?```, is binary, our model will be performing binary prediction, also known as binary classification. In this example we will let AutoPilot infer the type of problem for us.\n",
    "\n",
    "You have the option to limit the running time of a SageMaker Autopilot job by providing either the maximum number of pipeline evaluations or candidates (one pipeline evaluation is called a `Candidate` because it generates a candidate model) or providing the total time allocated for the overall Autopilot job. Under default settings, this job takes about four hours to run. This varies between runs because of the nature of the exploratory process Autopilot uses to find optimal training parameters.\n",
    "\n",
    "We limit the number of candidates to 20 so that the job finishes in a few minutes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Launching the SageMaker Autopilot Job<a name=\"Launching\"></a>\n",
    "\n",
    "You can now launch the Autopilot job by calling the `fit` method of the `AutoML` estimator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl.fit(train_file, job_name=base_job_name, wait=False, logs=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tracking SageMaker Autopilot Job Progress<a name=\"Tracking\"></a>\n",
    "SageMaker Autopilot job consists of the following high-level steps : \n",
    "* Analyzing Data, where the dataset is analyzed and Autopilot comes up with a list of ML pipelines that should be tried out on the dataset. The dataset is also split into train and validation sets.\n",
    "* Feature Engineering, where Autopilot performs feature transformation on individual features of the dataset as well as at an aggregate level.\n",
    "* Model Tuning, where the top performing pipeline is selected along with the optimal hyperparameters for the training algorithm (the last stage of the pipeline). \n",
    "\n",
    "We can use the `describe_auto_ml_job` method to check the status of our SageMaker Autopilot job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print ('JobStatus - Secondary Status')\n",
    "print('------------------------------')\n",
    "\n",
    "\n",
    "describe_response = automl.describe_auto_ml_job()\n",
    "print (describe_response['AutoMLJobStatus'] + \" - \" + describe_response['AutoMLJobSecondaryStatus'])\n",
    "job_run_status = describe_response['AutoMLJobStatus']\n",
    "    \n",
    "while job_run_status not in ('Failed', 'Completed', 'Stopped'):\n",
    "    describe_response = automl.describe_auto_ml_job()\n",
    "    job_run_status = describe_response['AutoMLJobStatus']\n",
    "    \n",
    "    print(describe_response['AutoMLJobStatus'] + \" - \" + describe_response['AutoMLJobSecondaryStatus'])\n",
    "    sleep(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true
   },
   "source": [
    "---\n",
    "## Describing the SageMaker Autopilot Job Results <a name=\"Results\"></a>\n",
    "\n",
    "We can use the `describe_auto_ml_job` method to look up the best candidate generated by the SageMaker Autopilot job. This notebook demonstrate end-to-end Autopilot so that we have a already initialized `automl` object. \n",
    "\n",
    "**Note: Using Another Autopilot Job**\n",
    "\n",
    "If you want to retrieve a previous Autopilot job or an Autopilot job launched outside of this notebook, such as from the SageMaker Studio UI, from the CLI, etc, you can use the following lines to prior to the next cell. If you are using a different dataset, you must also override the following variables defined in the [Data](#Data) section in order to run the batch jobs and perform the analysis: `test_data`, `test_data_no_target`, `test_file`, `target_attribute_name`, `target_attribute_values`, and `target_attribute_true_value`.\n",
    "\n",
    "```python\n",
    "from sagemaker import AutoML\n",
    "automl = AutoML.attach(auto_ml_job_name='<autopilot-job-name>')\n",
    "\n",
    "test_data = ... # test_data to be used (with target column)\n",
    "test_data_no_target = ... # test_data to be used (without target column)\n",
    "test_file = ... # path of data to upload to S3 and perform batch inference (csv file of test_data_no_target)\n",
    "target_attribute_name = ... # name of target column (values to predict)\n",
    "target_attribute_values = ... # list of unique values in target column (sorted)\n",
    "target_attribute_true_value = ... # second value in target column (binary classification \"True\" class)\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_candidate = automl.describe_auto_ml_job()['BestCandidate']\n",
    "best_candidate_name = best_candidate['CandidateName']\n",
    "pprint(best_candidate)\n",
    "print('\\n')\n",
    "print(\"CandidateName: \" + best_candidate_name)\n",
    "print(\"FinalAutoMLJobObjectiveMetricName: \" + best_candidate['FinalAutoMLJobObjectiveMetric']['MetricName'])\n",
    "print(\"FinalAutoMLJobObjectiveMetricValue: \" + str(best_candidate['FinalAutoMLJobObjectiveMetric']['Value']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Due to some randomness in the algorithms involved, different runs will provide slightly different results, but accuracy will be around or above $93\\%$, which is a good result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check Top Candidates\n",
    "\n",
    "In addition to the `best_candidate`, we can also explore the other top candidates generated by SageMaker Autopilot. \n",
    "\n",
    "We use the `list_candidates` method to see our other top candidates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TOP_N_CANDIDATES # number of top candidates to check, defined at the beginning of the notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "candidates = automl.list_candidates(sort_by='FinalObjectiveMetricValue',\n",
    "                                    sort_order='Descending',\n",
    "                                    max_results=TOP_N_CANDIDATES)\n",
    "\n",
    "for candidate in candidates:\n",
    "    print(\"Candidate name: \", candidate['CandidateName'])\n",
    "    print(\"Objective metric name: \", candidate['FinalAutoMLJobObjectiveMetric']['MetricName'])\n",
    "    print(\"Objective metric value: \", candidate['FinalAutoMLJobObjectiveMetric']['Value'])\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Evaluate Top Candidates <a name=\"Evaluation\"></a>\n",
    "\n",
    "Once our SageMaker Autopilot job has finished, we can start running inference on the top candidates. In SageMaker, you can perform inference in two ways: online endpoint inference or batch transform inference. Lets focus on batch transform inference.\n",
    "\n",
    "We'll perform batch transform on our top candidates and analyze some custom metrics from our top candidates' prediction results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload Data for Transform Jobs\n",
    "\n",
    "We'll use the `test_data` which we defined when we split out data in train and test splits. We need to upload this data to S3. As a refresher, here's `test_data`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_data_transform = session.upload_data(path=test_file, bucket=bucket, key_prefix=prefix)\n",
    "print('Uploaded transform data to {}'.format(input_data_transform))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Customize the Inference Response\n",
    "\n",
    "For classification problem types, the inference containers generated by SageMaker Autopilot allow you to select the response content for predictions. Valid inference response content are defined below for binary classification and multiclass classification problem types.\n",
    "\n",
    "- `'predicted_label'` - predicted class\n",
    "- `'probability'` - In binary classification, the probability that the result is predicted as the second or `True` class in the target column. In multiclass classification, the probability of the winning class.\n",
    "- `'labels'` - list of all possible classes\n",
    "- `'probabilities'` - list of all probabilities for all classes (order corresponds with `'labels'`)\n",
    "\n",
    "By default the inference contianers are configured to generate the `'predicted_label'`.\n",
    "\n",
    "In this example we use `‘predicted_label’` and `‘probability’` to demonstrate how to evaluate the models with custom metrics. For the Churn dataset, the second or `True` class is the string`'True.'`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_response_keys = ['predicted_label', 'probability']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Models and Tranform Estimators\n",
    "\n",
    "Let's create our Models and Batch Transform Estimators using the `create_model` method. We can specify our inference response using the `inference_response_keys` keyword argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_transform_output_path = 's3://{}/{}/inference-results/'.format(bucket, prefix);\n",
    "\n",
    "transformers = []\n",
    "\n",
    "for candidate in candidates:\n",
    "    model = automl.create_model(name=candidate['CandidateName'],\n",
    "                                candidate=candidate,\n",
    "                                inference_response_keys=inference_response_keys)\n",
    "    \n",
    "    output_path = s3_transform_output_path + candidate['CandidateName'] +'/'\n",
    "    \n",
    "    transformers.append(\n",
    "        model.transformer(instance_count=1, \n",
    "                          instance_type='ml.m5.xlarge',\n",
    "                          assemble_with='Line',\n",
    "                          output_path=output_path))\n",
    "\n",
    "print(\"Setting up {} Batch Transform Jobs in `transformers`\".format(len(transformers)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start the Transform Jobs\n",
    "\n",
    "Let's start all the transform jobs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for transformer in transformers:\n",
    "    transformer.transform(data=input_data_transform, split_type='Line', content_type='text/csv', wait=False)\n",
    "    print(\"Starting transform job {}\".format(transformer._current_job_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we wait for our transform jobs to finish."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pending_complete = True\n",
    "\n",
    "while pending_complete:\n",
    "    pending_complete = False\n",
    "    num_transform_jobs = len(transformers)\n",
    "    for transformer in transformers:\n",
    "        desc = sm.describe_transform_job(TransformJobName=transformer._current_job_name)\n",
    "        if desc['TransformJobStatus'] not in ['Failed', 'Completed']:\n",
    "            pending_complete = True\n",
    "        else:\n",
    "            num_transform_jobs -= 1\n",
    "    print(\"{} out of {} transform jobs are running.\".format(num_transform_jobs, len(transformers)))\n",
    "    sleep(30)\n",
    "    \n",
    "for transformer in transformers:\n",
    "    desc = sm.describe_transform_job(TransformJobName=transformer._current_job_name)\n",
    "    print(\"Transform job '{}' finished with status {}\".format(transformer._current_job_name, desc['TransformJobStatus']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate the Inference Results\n",
    "\n",
    "Now we analyze our inference results. The batch transform results are stored in S3. So we define a helper method to get the results from S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import io\n",
    "from urllib.parse import urlparse\n",
    "\n",
    "def get_csv_from_s3(s3uri, file_name):\n",
    "    parsed_url = urlparse(s3uri)\n",
    "    bucket_name = parsed_url.netloc\n",
    "    prefix = parsed_url.path[1:].strip('/')\n",
    "    s3 = boto3.resource('s3')\n",
    "    obj = s3.Object(bucket_name, '{}/{}'.format(prefix, file_name))\n",
    "    return obj.get()[\"Body\"].read().decode('utf-8')    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = []\n",
    "\n",
    "for transformer in transformers:\n",
    "    print(transformer.output_path)\n",
    "    pred_csv = get_csv_from_s3(transformer.output_path, '{}.out'.format(test_file))\n",
    "    predictions.append(pd.read_csv(io.StringIO(pred_csv), header=None))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the `sklearn.metrics` module to analyze our prediction results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score, roc_curve, precision_recall_curve, classification_report, average_precision_score\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "labels = test_data[target_attribute_name].apply(lambda row: True if row==target_attribute_true_value else False)\n",
    "\n",
    "# calculate auc score\n",
    "for prediction, candidate in zip(predictions, candidates):\n",
    "    roc_auc = roc_auc_score(labels, prediction.loc[:,1])\n",
    "    ap = average_precision_score(labels, prediction.loc[:,1])\n",
    "    print('%s\\'s ROC AUC = %.2f, Average Precision = %.2f' % (candidate['CandidateName'], roc_auc, ap))\n",
    "    print(classification_report(test_data[target_attribute_name], prediction.loc[:,0]))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the ROC curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fpr_tpr = []\n",
    "for prediction in predictions:\n",
    "    fpr, tpr, _ = roc_curve(labels, prediction.loc[:,1])\n",
    "    fpr_tpr.append(fpr)\n",
    "    fpr_tpr.append(tpr)\n",
    "\n",
    "plt.figure(num=None, figsize=(16, 9), dpi=160, facecolor='w', edgecolor='k')\n",
    "plt.plot(*fpr_tpr)\n",
    "plt.legend([candidate['CandidateName'] for candidate in candidates], loc=\"lower right\")\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('ROC Curve')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the precision-recall curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "precision_recall = []\n",
    "for prediction in predictions:\n",
    "    precision, recall, _ = precision_recall_curve(labels, prediction.loc[:,1])\n",
    "    precision_recall.append(recall)\n",
    "    precision_recall.append(precision)\n",
    "\n",
    "plt.figure(num=None, figsize=(16, 9), dpi=160, facecolor='w', edgecolor='k')\n",
    "plt.plot(*precision_recall)\n",
    "plt.legend([candidate['CandidateName'] for candidate in candidates], loc=\"lower left\")\n",
    "plt.xlabel('Recall')\n",
    "plt.ylabel('Precision')\n",
    "plt.title('Precision-Recall Curve')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given the target minimal precision, we will find the model that provides the best recall and the operation point for that model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_min_precision = 0.75\n",
    "\n",
    "best_recall = 0\n",
    "best_candidate_idx = -1\n",
    "best_candidate_threshold = -1\n",
    "candidate_idx = 0\n",
    "for prediction in predictions:\n",
    "    precision, recall, thresholds = precision_recall_curve(labels, prediction.loc[:,1])\n",
    "    threshold_idx = np.argmax(precision>=target_min_precision)\n",
    "    if recall[threshold_idx] > best_recall:\n",
    "        best_recall = recall[threshold_idx]\n",
    "        best_candidate_threshold = thresholds[threshold_idx]\n",
    "        best_candidate_idx = candidate_idx\n",
    "    candidate_idx += 1\n",
    "\n",
    "print(\"Best Candidate Name: {}\".format(candidates[best_candidate_idx]['CandidateName']))\n",
    "print(\"Best Candidate Threshold (Operation Point): {}\".format(best_candidate_threshold))\n",
    "print(\"Best Candidate Recall: {}\".format(best_recall))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get predictions of the best model based on the selected operating point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction_default = predictions[best_candidate_idx].loc[:,0] == target_attribute_true_value\n",
    "prediction_updated = predictions[best_candidate_idx].loc[:,1] >= best_candidate_threshold\n",
    "\n",
    "# compare the updated predictions to Autopilot's default\n",
    "from sklearn.metrics import precision_score, recall_score\n",
    "print(\"Default Operating Point: recall={}, precision={}\".format(recall_score(labels, prediction_default), precision_score(labels, prediction_default)))\n",
    "print(\"Updated Operating Point: recall={}, precision={}\".format(recall_score(labels, prediction_updated), precision_score(labels, prediction_updated)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy the Selected Candidate\n",
    "\n",
    "After performing the analysis above, we can deploy the candidate that provides the best recall. We will use the `deploy` method to create the online inference endpoint. We'll use the same `inference_response_keys` from out batch transform jobs, but you can customize this as you wish. If `inference_response_keys` is not specified, only the `'predicted_label'` will be returned.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_response_keys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.predictor import Predictor\n",
    "from sagemaker.serializers import CSVSerializer\n",
    "from sagemaker.deserializers import CSVDeserializer\n",
    "\n",
    "predictor = automl.deploy(initial_instance_count=1,\n",
    "                          instance_type='ml.m5.2xlarge',\n",
    "                          candidate=candidates[best_candidate_idx],\n",
    "                          inference_response_keys=inference_response_keys,\n",
    "                          predictor_cls=Predictor,\n",
    "                          serializer=CSVSerializer(),\n",
    "                          deserializer=CSVDeserializer())\n",
    "\n",
    "print(\"Created endpoint: {}\".format(predictor.endpoint_name))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have created our endpoint, we can send real-time predictions to the endpoint. The inference output will contain the model's`predicted_label` and `probability`. We use the custom threshold calculated above to determine our own `custom_predicted_label` based on the probability score in the inference response. If a `probability` is less than the `best_candidate_threshold`, the `custom_predicted_label` is the `False.` class. If a `probability` is greater than of equal to the `best_candidate_threshold`, the `custom_predicted_label` is the `True.` class. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_candidate_threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction = predictor.predict(test_data_no_target.to_csv(sep=',', header=False, index=False))\n",
    "prediction_df = pd.DataFrame(prediction, columns=inference_response_keys)\n",
    "custom_predicted_labels = prediction_df.iloc[:,1].astype(float).values >= best_candidate_threshold\n",
    "prediction_df['custom_predicted_label'] = custom_predicted_labels\n",
    "prediction_df['custom_predicted_label'] = prediction_df['custom_predicted_label'].map({False: target_attribute_values[0], True: target_attribute_values[1]})\n",
    "prediction_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Cleanup <a name=\"Cleanup\"></a>\n",
    "\n",
    "The Autopilot job creates many underlying artifacts such as dataset splits, preprocessing scripts, or preprocessed data, etc. This code, when un-commented, deletes them. This operation deletes all the generated models and the auto-generated notebooks as well. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# s3 = boto3.resource('s3')\n",
    "# s3_bucket = s3.Bucket(bucket)\n",
    "\n",
    "# s3_bucket.objects.filter(Prefix=prefix).delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we delete the models and the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for transformer in transformers:\n",
    "#     transformer.delete_model()\n",
    "\n",
    "# predictor.delete_endpoint()\n",
    "# predictor.delete_model()"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-west-2:236514542706:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 4
}