Use Spark and Scala to predict Equipment Purchase.ipynb

{
    "nbformat_minor": 1, 
    "cells": [
        {
            "source": "<div style=\"background:#F5F7FA; height:100px; padding: 2em; font-size:14px;\">\n<span style=\"font-size:18px;color:#152935;\">Want to do more?</span><span style=\"border: 1px solid #3d70b2;padding: 15px;float:right;margin-right:40px; color:#3d70b2; \"><a href=\"https://ibm.co/wsnotebooks\" target=\"_blank\" style=\"color: #3d70b2;text-decoration: none;\">Sign Up</a></span><br>\n<span style=\"color:#5A6872;\"> Try out this notebook with your free trial of IBM Watson Studio.</span>\n</div>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<table style=\"border: none\" align=\"left\">\n   <tr style=\"border: none\">\n      <th style=\"border: none\"><font face=\"verdana\" size=\"5\" color=\"black\"><b>Use Spark and Scala to predict Equipment Purchase</b></th>\n      <th style=\"border: none\"><img src=\"https://github.com/pmservice/customer-satisfaction-prediction/blob/master/app/static/images/ml_icon_gray.png?raw=true\" alt=\"Watson Machine Learning icon\" height=\"40\" width=\"40\"></th>\n   </tr>\n   <tr style=\"border: none\">\n       <th style=\"border: none\"><img src=\"https://github.com/pmservice/wml-sample-models/blob/master/spark/product-line-prediction/images/products_graphics.png?raw=true\" alt=\"Icon\" width=\"800\"> </th>\n   </tr>\n</table>", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "source": "This notebook contains steps and code to get data from the IBM Data Science Experience Community, create a predictive model, and start scoring new data. This notebook introduces commands for getting data and for basic data cleaning and exploration, pipeline creation, model training, model persistance to Watson Machine Learning repository, model deployment, and scoring.\n\nSome familiarity with Scala is helpful. This notebook uses Scala 2.11 and Apache\u00ae Spark 2.1.\n\nYou will use a publicly available data set, **GoSales Transactions for Naive Bayes Model**, which details anonymous outdoor equipment purchases. Use the details of this data set to predict clients' interests in terms of product line, such as golf accessories, camping equipment, and others.\n \n## Learning goals\n\nThe learning goals of this notebook are:\n\n-  Load a CSV file into an Apache\u00ae Spark DataFrame.\n-  Explore data.\n-  Prepare data for training and evaluation.\n-  Create an Apache\u00ae Spark machine learning pipeline.\n-  Train and evaluate a model.\n-  Persist a pipeline and model in Watson Machine Learning repository.\n-  Deploy a model for online scoring using Wastson Machine Learning API.\n-  Score sample scoring data using the Watson Machine Learning API.\n\n\n## Contents\n\nThis notebook contains the following parts:\n\n1.\t[Set up](#setup)\n2.\t[Load and explore data](#load)\n3.\t[Create spark ml model](#model)\n4.\t[Persist model](#persistence)\n5.\t[Deploy and score in a Cloud](#scoring)\n6.\t[Summary and next steps](#summary)", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"setup\"></a>\n## 1. Set up\n\nBefore you use the sample code in this notebook, you must perform the following setup tasks:\n\n-  Create a [Watson Machine Learning (WML) Service](https://console.ng.bluemix.net/catalog/services/ibm-watson-machine-learning/) instance (a free plan is offered and information about how to create the instance is [here](https://dataplatform.ibm.com/docs/content/analyze-data/wml-setup.html))\n-  Make sure that you are using a Spark 2.1 kernel.\n-  Download **GoSales Transactions** from Watson Studio Community (code provided below).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"load\"></a>\n## 2. Load and explore data\n\nIn this section you will load the data as an Apache\u00ae Spark DataFrame and perform a basic exploration.\n\nLoad the data to the Spark DataFrame by using *wget* to upload the data to gpfs and then use spark *read* method to read data. ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 1, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "--2018-04-03 07:20:15--  https://apsportal.ibm.com/exchange-api/v1/entries/8044492073eb964f46597b4be06ff5ea/data?accessKey=9561295fa407698694b1e254d0099600\nResolving apsportal.ibm.com (apsportal.ibm.com)... 158.85.156.20\nConnecting to apsportal.ibm.com (apsportal.ibm.com)|158.85.156.20|:443... connected.\nHTTP request sent, awaiting response... 301 Moved Permanently\nLocation: https://dataplatform.ibm.com/exchange-api/v1/entries/8044492073eb964f46597b4be06ff5ea/data?accessKey=9561295fa407698694b1e254d0099600 [following]\n--2018-04-03 07:20:15--  https://dataplatform.ibm.com/exchange-api/v1/entries/8044492073eb964f46597b4be06ff5ea/data?accessKey=9561295fa407698694b1e254d0099600\nResolving dataplatform.ibm.com (dataplatform.ibm.com)... 75.126.81.68\nConnecting to dataplatform.ibm.com (dataplatform.ibm.com)|75.126.81.68|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: unspecified [application/octet-stream]\nSaving to: \u2018GoSalesNaiveBayes.csv\u2019\n\n     0K .......... .......... .......... .......... .......... 1.42M\n    50K .......... .......... .......... .......... .......... 7.56M\n   100K .......... .......... .......... .......... .......... 7.31M\n   150K .......... .......... .......... .......... .......... 11.7M\n   200K .......... .......... .......... .......... .......... 15.9M\n   250K .......... .......... .......... .......... .......... 4.48M\n   300K .......... .......... .......... .......... .......... 17.0M\n   350K .......... .......... .......... .......... .......... 11.0M\n   400K .......... .......... .......... .......... .......... 18.1M\n   450K .......... .......... .......... .......... .......... 12.1M\n   500K .......... .......... .......... .......... .......... 18.8M\n   550K .......... .......... .......... .......... .......... 19.5M\n   600K .......... .......... .......... .......... .......... 18.9M\n   650K .......... .......... .......... .......... .......... 12.6M\n   700K .......... .......... .......... .......... .......... 20.1M\n   750K .......... .......... .......... .......... .......... 19.7M\n   800K .......... .......... .......... .......... .......... 21.0M\n   850K .......... .......... .......... .......... .......... 13.2M\n   900K .......... .......... .......... .......... .......... 19.9M\n   950K .......... .......... .......... .......... .......... 18.5M\n  1000K .......... .......... .......... .......... .......... 17.0M\n  1050K .......... .......... .......... .......... .......... 20.8M\n  1100K .......... .......... .......... .......... .......... 21.4M\n  1150K .......... .......... .......... .......... .......... 11.4M\n  1200K .......... .......... .......... .......... .......... 19.6M\n  1250K .......... .......... .......... .......... .......... 11.2M\n  1300K .......... .......... .......... .......... .......... 20.0M\n  1350K .......... .......... .......... .......... .......... 19.9M\n  1400K .......... .......... .......... .......... .......... 20.5M\n  1450K .......... .......... .......... .......... .......... 12.0M\n  1500K .......... .......... .......... .......... .......... 19.9M\n  1550K .......... .......... .......... .......... .......... 20.9M\n  1600K .......... .......... .......... .......... .......... 20.1M\n  1650K .......... .......... .......... .......... .......... 19.0M\n  1700K .......... .......... .......... .......... .......... 19.0M\n  1750K .......... .......... .......... .......... .......... 10.7M\n  1800K .......... .......... .......... .......... .......... 67.2M\n  1850K .......... .......... .......... .......... .......... 12.7M\n  1900K .......... .......... .......... .......... .......... 20.3M\n  1950K .......... .......... .......... .......... .......... 20.2M\n  2000K .......... .......... .......... .......... .......... 62.8M\n  2050K .......... .......... .......... .......... .......... 12.3M\n  2100K .......... .......... .......... .......... .......... 20.3M\n  2150K .......... .......... .......... .......... .......... 13.8M\n  2200K .......... .......... .......... .......... .......... 20.0M\n  2250K .......... .......... .......... .......... .......... 15.3M\n  2300K .......... .......... .......... .......... .......... 21.3M\n  2350K .......... .......... .......... .......... .......... 19.4M\n  2400K .......... .......... .......... .......... .......... 20.9M\n  2450K .                                                      32.0M=0.2s\n\n2018-04-03 07:20:16 (12.8 MB/s) - \u2018GoSalesNaiveBayes.csv\u2019 saved [2510539]\n\n"
                }, 
                {
                    "execution_count": 1, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "0"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "import scala.sys.process._\n\nval filename = \"GoSalesNaiveBayes.csv\"\n\ns\"wget -O $filename https://apsportal.ibm.com/exchange-api/v1/entries/8044492073eb964f46597b4be06ff5ea/data?accessKey=9561295fa407698694b1e254d0099600\".!"
        }, 
        {
            "execution_count": 2, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "display_data", 
                    "data": {
                        "text/plain": "Waiting for a Spark session to start..."
                    }, 
                    "metadata": {}
                }, 
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "[Stage 2:=============================>                             (1 + 1) / 2]"
                }
            ], 
            "source": "val df_data = spark.\n    read.format(\"org.apache.spark.sql.execution.datasources.csv.CSVFileFormat\").\n    option(\"header\", \"true\").\n    option(\"inferSchema\", \"true\").\n    load(filename)"
        }, 
        {
            "source": "Explore the loaded data by using the following Apache\u00ae Spark DataFrame methods:\n-  print schema\n-  print top ten records\n-  count all records", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 3, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "root\n |-- PRODUCT_LINE: string (nullable = true)\n |-- GENDER: string (nullable = true)\n |-- AGE: integer (nullable = true)\n |-- MARITAL_STATUS: string (nullable = true)\n |-- PROFESSION: string (nullable = true)\n\n"
                }
            ], 
            "source": "df_data.printSchema()"
        }, 
        {
            "source": "As you can see, the data contains five fields. PRODUCT_LINE field is the one we would like to predict (label).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 4, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "+--------------------+------+---+--------------+------------+\n|        PRODUCT_LINE|GENDER|AGE|MARITAL_STATUS|  PROFESSION|\n+--------------------+------+---+--------------+------------+\n|Personal Accessories|     M| 27|        Single|Professional|\n|Personal Accessories|     F| 39|       Married|       Other|\n|Mountaineering Eq...|     F| 39|       Married|       Other|\n|Personal Accessories|     F| 56|   Unspecified| Hospitality|\n|      Golf Equipment|     M| 45|       Married|     Retired|\n|      Golf Equipment|     M| 45|       Married|     Retired|\n|   Camping Equipment|     F| 39|       Married|       Other|\n|   Camping Equipment|     F| 49|       Married|       Other|\n|  Outdoor Protection|     F| 49|       Married|       Other|\n|      Golf Equipment|     M| 47|       Married|     Retired|\n+--------------------+------+---+--------------+------------+\nonly showing top 10 rows\n\n"
                }
            ], 
            "source": "df_data.show(10)"
        }, 
        {
            "execution_count": 5, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Total number of records: 60252\n"
                }
            ], 
            "source": "println(\"Total number of records: \" + df_data.count())"
        }, 
        {
            "source": "As you can see, the data set contains 60252 records.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"model\"></a>\n## 3. Create an Apache\u00ae Spark machine learning model\n\nIn this section you will learn how to prepare data, create an Apache\u00ae Spark machine learning pipeline, and train a model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 3.1: Prepare data\n\nIn this subsection you will split your data into: train, test and predict datasets.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 6, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Number of training records: 48176\nNumber of testing records: 10860\nNumber of prediction records: 1216\n"
                }
            ], 
            "source": "val splits = df_data.randomSplit(Array(0.8, 0.18, 0.02), seed = 24L)\nval training_data = splits(0).cache()\nval test_data = splits(1)\nval prediction_data = splits(2)\n\nprintln(\"Number of training records: \" + training_data.count())\nprintln(\"Number of testing records: \" + test_data.count())\nprintln(\"Number of prediction records: \" + prediction_data.count())"
        }, 
        {
            "source": "As you can see our data has been successfully split into three datasets: \n\n-  The train data set, which is the largest group, is used for training.\n-  The test data set will be used for model evaluation.\n-  The predict data set will be used for prediction.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 3.2: Create pipeline and train a model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "In this section you will create an Apache\u00ae Spark machine learning pipeline and then train the model.\n\nIn the first step you need to import the Apache\u00ae Spark machine learning packages that will be needed in the subsequent steps.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 7, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "import org.apache.spark.ml.classification.RandomForestClassifier\nimport org.apache.spark.ml.feature.{OneHotEncoder, StringIndexer, IndexToString, VectorAssembler}\nimport org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator\nimport org.apache.spark.ml.{Model, Pipeline, PipelineStage, PipelineModel}\nimport org.apache.spark.sql.SparkSession"
        }, 
        {
            "source": "In the following step, convert all the string fields to numeric ones by using the StringIndexer transformer.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 8, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val stringIndexer_label = new StringIndexer().setInputCol(\"PRODUCT_LINE\").setOutputCol(\"label\").fit(df_data)\nval stringIndexer_prof = new StringIndexer().setInputCol(\"PROFESSION\").setOutputCol(\"PROFESSION_IX\")\nval stringIndexer_gend = new StringIndexer().setInputCol(\"GENDER\").setOutputCol(\"GENDER_IX\")\nval stringIndexer_mar = new StringIndexer().setInputCol(\"MARITAL_STATUS\").setOutputCol(\"MARITAL_STATUS_IX\")"
        }, 
        {
            "source": "In the following step, create a feature vector by combining all features together.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 9, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val vectorAssembler_features = new VectorAssembler().setInputCols(Array(\"GENDER_IX\", \"AGE\", \"MARITAL_STATUS_IX\", \"PROFESSION_IX\")).setOutputCol(\"features\")"
        }, 
        {
            "source": "Next, define estimators you want to use for classification. Random Forest is used in the following example.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 10, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val rf = new RandomForestClassifier().setLabelCol(\"label\").setFeaturesCol(\"features\").setNumTrees(10)"
        }, 
        {
            "source": "Finally, indexed labels back to original labels.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 11, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val labelConverter = new IndexToString().setInputCol(\"prediction\").setOutputCol(\"predictedLabel\").setLabels(stringIndexer_label.labels)"
        }, 
        {
            "source": "Let's build the pipeline now. A pipeline consists of transformers and an estimator.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 12, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val pipeline_rf = new Pipeline().setStages(Array(stringIndexer_label, stringIndexer_prof, stringIndexer_gend, stringIndexer_mar, vectorAssembler_features, rf, labelConverter))"
        }, 
        {
            "source": "Now, you can train your Random Forest model by using the previously defined **pipeline** and **training data**.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 13, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "root\n |-- PRODUCT_LINE: string (nullable = true)\n |-- GENDER: string (nullable = true)\n |-- AGE: integer (nullable = true)\n |-- MARITAL_STATUS: string (nullable = true)\n |-- PROFESSION: string (nullable = true)\n\n"
                }
            ], 
            "source": "training_data.printSchema()"
        }, 
        {
            "execution_count": 14, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val model_rf = pipeline_rf.fit(training_data)"
        }, 
        {
            "source": "You can check your **model accuracy** now. To evaluate the model, use **test data**.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 15, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "Accuracy = 0.58\n"
                }
            ], 
            "source": "val predictions = model_rf.transform(test_data)\nval evaluatorRF = new MulticlassClassificationEvaluator().setLabelCol(\"label\").setPredictionCol(\"prediction\").setMetricName(\"accuracy\")\nval accuracy = evaluatorRF.evaluate(predictions)\n\nprintln(f\"Accuracy = $accuracy%.2f\")"
        }, 
        {
            "source": "You can tune your model now to achieve better accuracy. For simplicity of this example tuning section is omitted.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"persistence\"></a>\n## 4. Persist model\n\nIn this section you will learn how to store your pipeline and model in Watson Machine Learning repository by using Scala client libraries.\n\nFirst, you must import client libraries.\n\n**Note**: Apache\u00ae Spark 2.1 or higher is required.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 16, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "// WML client library\n\nimport com.ibm.analytics.ngp.repository._\n\n// Helper libraries\n\nimport scalaj.http.{Http, HttpOptions}\nimport scala.util.{Success, Failure}\nimport java.util.Base64\nimport java.nio.charset.StandardCharsets\nimport play.api.libs.json._"
        }, 
        {
            "source": "Authenticate to the Watson Machine Learning service on IBM Cloud.\n\n**Tip**: Authentication information (your credentials) can be found in the [Service Credentials](https://console.bluemix.net/docs/services/service_credentials.html#service_credentials) tab of the service instance that you created on IBM Cloud. <BR>If you cannot see the **instance_id** field in **Service Credentials**, click **New credential (+)** to generate new authentication information. \n\n**Action**: Enter your Watson Machine Learning service instance credentials here.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val wml_credentials = \"\"\"{\n  \"url\": \"https://ibm-watson-ml.mybluemix.net\",\n  \"access_key\": \"***\",\n  \"username\": \"***\",\n  \"password\": \"***\",\n  \"instance_id\": \"***\"\n}\"\"\""
        }, 
        {
            "execution_count": 46, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val wml_credentials_json:JsValue = Json.parse(wml_credentials)\n\nval wml_service_path = (wml_credentials_json \\ \"url\").asOpt[String] match {\n    case Some(x) => x\n    case None => \"\"\n}\nval wml_instance_id = (wml_credentials_json \\ \"instance_id\").asOpt[String] match {\n    case Some(x) => x\n    case None => \"\"\n}\nval wml_username = (wml_credentials_json \\ \"username\").asOpt[String] match {\n    case Some(x) => x\n    case None => \"\"\n}\nval wml_password = (wml_credentials_json \\ \"password\").asOpt[String] match {\n    case Some(x) => x\n    case None => \"\"\n}"
        }, 
        {
            "execution_count": 47, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 47, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "Success(())"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "val client = MLRepositoryClient(wml_service_path)\nclient.authorize(wml_username, wml_password)"
        }, 
        {
            "source": "Create model artifact (abstraction layer).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 48, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val model_artifact = MLRepositoryArtifact(model_rf, training_data, \"WML Product Line Prediction Model\")"
        }, 
        {
            "source": "**Tip**: The MLRepositoryArtifact method expects a trained model object, training data, and a model name. (It is this model name that is displayed by the Watson Machine Learning service).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 4.1: Save pipeline and model\n\nIn this subsection you will learn how to save pipeline and model artifacts to your Watson Machine Learning instance.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 49, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val saved_model = client.models.save(model_artifact).get"
        }, 
        {
            "source": "Get saved model metadata from Watson Machine Learning.\n\n**Tip**: Use *meta.availableProps* to get the list of available props.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 50, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 50, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "Vector(trainingDataSchema, lastUpdated, label, inputDataSchema, modelVersionHref, prediction, modelType, version, modelHref, artifactPath, pipelineVersionHref, runtime, creationTime, probability)"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "saved_model.meta.availableProps"
        }, 
        {
            "execution_count": 51, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "modelType: sparkml-model-2.1\ntrainingDataSchema: {\"type\":\"struct\",\"fields\":[{\"name\":\"PRODUCT_LINE\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"GENDER\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"AGE\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"MARITAL_STATUS\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"PROFESSION\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}\ncreationTime: 2018-04-03T12:35:32.457Z\nmodelVersionHref: https://ibm-watson-ml.mybluemix.net/v2/artifacts/models/32b847bd-90a3-4db0-aa91-9fb01cf160ec/versions/605ebb83-ddde-4917-903e-4b9951c7b54c\nlabel: PRODUCT_LINE\n"
                }
            ], 
            "source": "println(\"modelType: \" + saved_model.meta.prop(\"modelType\").get)\nprintln(\"trainingDataSchema: \" + saved_model.meta.prop(\"trainingDataSchema\").get)\nprintln(\"creationTime: \" + saved_model.meta.prop(\"creationTime\").get)\nprintln(\"modelVersionHref: \" + saved_model.meta.prop(\"modelVersionHref\").get)\nprintln(\"label: \" + saved_model.meta.prop(\"label\").get)"
        }, 
        {
            "source": "**Tip**: **modelVersionHref** is our model unique indentifier in the Watson Machine Learning repository.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 4.2: Load model and make predictions\n\nIn this subsection you will learn how to load back saved model from specified instance of Watson Machine Learning.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 52, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val model_version_href = saved_model.meta.prop(\"modelVersionHref\").get\nval loaded_model_artifact = client.models.version(model_version_href).get"
        }, 
        {
            "source": "You can print for example model name to make sure that model artifact has been loaded correctly.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 53, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 53, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "WML Product Line Prediction Model"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "loaded_model_artifact.name.mkString"
        }, 
        {
            "source": "As you can see the name is correct. ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 54, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "+------+---+--------------+-----------+--------------------+\n|GENDER|AGE|MARITAL_STATUS| PROFESSION|      predictedLabel|\n+------+---+--------------+-----------+--------------------+\n|     F| 18|        Single|      Other|Personal Accessories|\n|     F| 18|        Single|     Retail|Personal Accessories|\n|     F| 19|        Single|Hospitality|   Camping Equipment|\n|     F| 19|        Single|Hospitality|   Camping Equipment|\n|     F| 19|        Single|Hospitality|   Camping Equipment|\n|     F| 19|        Single|Hospitality|   Camping Equipment|\n|     F| 19|        Single|      Other|Personal Accessories|\n|     F| 19|        Single|      Other|Personal Accessories|\n|     F| 19|        Single|      Other|Personal Accessories|\n|     F| 19|        Single|      Other|Personal Accessories|\n+------+---+--------------+-----------+--------------------+\nonly showing top 10 rows\n\n"
                }
            ], 
            "source": "loaded_model_artifact match {\n        case SparkPipelineModelLoader(Success(model)) => {\n          val predictions = model.transform(prediction_data)\n        }\n        case SparkPipelineModelLoader(Failure(e)) => \"Loading failed.\"\n        case _ => println(s\"Unexpected artifact class: ${loaded_model_artifact.getClass}\")\n    }\npredictions.select(\"GENDER\", \"AGE\", \"MARITAL_STATUS\", \"PROFESSION\", \"predictedLabel\").show(10)"
        }, 
        {
            "source": "By tabulating a count, you can see which product line is the most popular.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 55, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "+--------------------+-----+\n|      predictedLabel|count|\n+--------------------+-----+\n|   Camping Equipment| 6356|\n|      Golf Equipment|  631|\n|Mountaineering Eq...|  699|\n|Personal Accessories| 3174|\n+--------------------+-----+\n\n"
                }
            ], 
            "source": "predictions.select(\"predictedLabel\").groupBy(\"predictedLabel\").count().show()"
        }, 
        {
            "source": "You have already learned how save and load the model from Watson Machine Learning repository.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"scoring\"></a>\n## 5. Deploy and score in a Cloud\n\nIn this section you will learn how to create online scoring and to score a new data record by using the Watson Machine Learning REST API. \nFor more information about REST APIs, see the [Swagger Documentation](http://watson-ml-api.mybluemix.net/).\n\nTo work with the Watson Machine Leraning REST API you must generate an access token. To do that you can use the following sample code:", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 56, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "// Get WML service instance token\n\nval wml_auth_header = \"Basic \" + Base64.getEncoder.encodeToString((wml_username + \":\" + wml_password).getBytes(StandardCharsets.UTF_8))\nval wml_url = wml_service_path + \"/v3/identity/token\"\nval wml_response = Http(wml_url).header(\"Authorization\", wml_auth_header).asString\nval wml_token_json:JsValue = Json.parse(wml_response.body)\n\nval wml_token = (wml_token_json \\ \"token\").asOpt[String] match {\n    case Some(x) => x\n    case None => \"\"\n}"
        }, 
        {
            "source": "### 5.1: Create online scoring endpoint\n\nNow you can create an online scoring endpoint. Execute the following sample code that uses the publishedModelId value to create the scoring endpoint to the Bluemix repository.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Get published_models url from instance details", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 57, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val endpoint_instance = wml_service_path + \"/v3/wml_instances/\" + wml_instance_id\nval wml_response_instance = Http(endpoint_instance).header(\"Content-Type\", \"application/json\").header(\"Authorization\", \"Bearer \" + wml_token).option(HttpOptions.connTimeout(10000)).option(HttpOptions.readTimeout(50000)).asString"
        }, 
        {
            "execution_count": 58, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 58, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "HttpResponse({\"metadata\":{\"guid\":\"fd1db635-c9f3-4d83-afc7-32efec367e5e\",\"url\":\"https://ibm-watson-ml.mybluemix.net/v3/wml_instances/fd1db635-c9f3-4d83-afc7-32efec367e5e\",\"created_at\":\"2018-01-30T10:39:51.938Z\",\"modified_at\":\"2018-04-03T12:35:32.514Z\"},\"entity\":{\"source\":\"Bluemix\",\"published_models\":{\"url\":\"https://ibm-watson-ml.mybluemix.net/v3/wml_instances/fd1db635-c9f3-4d83-afc7-32efec367e5e/published_models\"},\"usage\":{\"expiration_date\":\"2018-05-01T00:00:00.000Z\",\"computation_time\":{\"current\":0},\"model_count\":{\"limit\":1000,\"current\":3},\"prediction_count\":{\"current\":0},\"gpu_count\":{\"limit\":8,\"current\":0},\"capacity_units\":{\"current\":0},\"deployment_count\":{\"limit\":1000,\"current\":2}},\"plan_id\":\"0f2a3c2c-456b-40f3-9b19-726d2740b11c\",\"status\":\"Act..."
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "wml_response_instance"
        }, 
        {
            "execution_count": 59, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 59, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "https://ibm-watson-ml.mybluemix.net/v3/wml_instances/fd1db635-c9f3-4d83-afc7-32efec367e5e/published_models"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "val published_models_json: JsValue = Json.parse(wml_response_instance.body)\nval published_models_url = (((published_models_json \\ \"entity\") \\\\ \"published_models\")(0) \\ \"url\").as[JsString].value\npublished_models_url"
        }, 
        {
            "source": "#### Get list of published models", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 60, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 60, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "HttpResponse({\"limit\":1000,\"resources\":[{\"metadata\":{\"guid\":\"32b847bd-90a3-4db0-aa91-9fb01cf160ec\",\"url\":\"https://ibm-watson-ml.mybluemix.net/v3/wml_instances/fd1db635-c9f3-4d83-afc7-32efec367e5e/published_models/32b847bd-90a3-4db0-aa91-9fb01cf160ec\",\"created_at\":\"2018-04-03T12:35:32.457Z\",\"modified_at\":\"2018-04-03T12:35:32.610Z\"},\"entity\":{\"runtime_environment\":\"spark-2.1\",\"learning_configuration_url\":\"https://ibm-watson-ml.mybluemix.net/v3/wml_instances/fd1db635-c9f3-4d83-afc7-32efec367e5e/published_models/32b847bd-90a3-4db0-aa91-9fb01cf160ec/learning_configuration\",\"author\":{},\"name\":\"WML Product Line Prediction Model\",\"label_col\":\"PRODUCT_LINE\",\"learning_iterations_url\":\"https://ibm-watson-ml.mybluemix.net/v3/wml_instances/fd1db635-c9f3-4d8..."
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "val wml_models = Http(published_models_url).header(\"Content-Type\", \"application/json\").header(\"Authorization\", \"Bearer \" + wml_token).option(HttpOptions.connTimeout(10000)).option(HttpOptions.readTimeout(50000)).asString\nwml_models"
        }, 
        {
            "execution_count": 61, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 61, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "https://ibm-watson-ml.mybluemix.net/v3/wml_instances/fd1db635-c9f3-4d83-afc7-32efec367e5e/published_models/32b847bd-90a3-4db0-aa91-9fb01cf160ec/deployments"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "var deployment_endpoint: String = _\nwml_models.body.split(\"\\\"\").map{ s => {if ((s contains \"deployments\") & (s contains saved_model.uid.mkString)) {deployment_endpoint = s}}}\ndeployment_endpoint"
        }, 
        {
            "source": "#### Create online deployment for published model", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 62, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val payload_name = \"Online scoring\"\nval payload_data_online = Json.stringify(Json.toJson(Map(\"type\" -> \"online\", \"name\" -> payload_name)))"
        }, 
        {
            "execution_count": 63, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val response_online = Http(deployment_endpoint).postData(payload_data_online).header(\"Content-Type\", \"application/json\").header(\"Authorization\", \"Bearer \" + wml_token).option(HttpOptions.connTimeout(50000)).option(HttpOptions.readTimeout(50000)).asString"
        }, 
        {
            "execution_count": 64, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val scoring_url_json: JsValue = Json.parse(response_online.body)\nval scoring_url = (scoring_url_json \\ \"entity\" \\ \"scoring_url\").asOpt[String] match {\n    case Some(x) => x\n    case None => \"\"\n}"
        }, 
        {
            "execution_count": 65, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "https://ibm-watson-ml.mybluemix.net/v3/wml_instances/fd1db635-c9f3-4d83-afc7-32efec367e5e/published_models/32b847bd-90a3-4db0-aa91-9fb01cf160ec/deployments/ff37fa26-1076-4808-aed4-094e2fb97d29/online"
                }
            ], 
            "source": "print(scoring_url)"
        }, 
        {
            "execution_count": 66, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val payload_scoring = Json.stringify(Json.toJson(Map(\"fields\" -> Json.toJson(List(Json.toJson(\"GENDER\"), Json.toJson(\"AGE\"), Json.toJson(\"MARITAL_STATUS\"), Json.toJson(\"PROFESSION\"))),\n                                                    \"values\" -> Json.toJson(List(List(Json.toJson(\"M\"), Json.toJson(55), Json.toJson(\"Single\"), Json.toJson(\"Executive\")))))))"
        }, 
        {
            "execution_count": 67, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [
                {
                    "execution_count": 67, 
                    "metadata": {}, 
                    "data": {
                        "text/plain": "{\"fields\":[\"GENDER\",\"AGE\",\"MARITAL_STATUS\",\"PROFESSION\"],\"values\":[[\"M\",55,\"Single\",\"Executive\"]]}"
                    }, 
                    "output_type": "execute_result"
                }
            ], 
            "source": "payload_scoring"
        }, 
        {
            "source": "Now, you can send (POST) new scoring records (new data) for which you would like to get predictions. To do that, execute the following sample code: ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": 68, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "val response_scoring = Http(scoring_url).postData(payload_scoring).header(\"Content-Type\", \"application/json\").header(\"Authorization\", \"Bearer \" + wml_token).option(HttpOptions.method(\"POST\")).option(HttpOptions.connTimeout(10000)).option(HttpOptions.readTimeout(50000)).asString"
        }, 
        {
            "execution_count": 69, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": false
            }, 
            "outputs": [
                {
                    "output_type": "stream", 
                    "name": "stdout", 
                    "text": "HttpResponse({\n  \"fields\": [\"GENDER\", \"AGE\", \"MARITAL_STATUS\", \"PROFESSION\", \"PRODUCT_LINE\", \"label\", \"PROFESSION_IX\", \"GENDER_IX\", \"MARITAL_STATUS_IX\", \"features\", \"rawPrediction\", \"probability\", \"prediction\", \"predictedLabel\"],\n  \"values\": [[\"M\", 55, \"Single\", \"Executive\", \"Camping Equipment\", 0.0, 3.0, 0.0, 1.0, [0.0, 55.0, 1.0, 3.0], [2.5050408694752604, 1.8771700964659068, 2.338808434620882, 3.1481331680893025, 0.13084743134864898], [0.250504086947526, 0.1877170096465907, 0.2338808434620882, 0.31481331680893027, 0.013084743134864898], 3.0, \"Golf Equipment\"]]\n},200,Map(Cache-Control -> Vector(private, no-cache, no-store, must-revalidate), Connection -> Vector(Keep-Alive), Content-Type -> Vector(application/json), Date -> Vector(Tue, 03 Apr 2018 12:36:39 GMT), Pragma -> Vector(no-cache), Server -> Vector(nginx/1.13.6), Status -> Vector(HTTP/1.1 200 OK), Transfer-Encoding -> Vector(chunked), X-Backside-Transport -> Vector(OK OK), X-Content-Type-Options -> Vector(nosniff), X-Envoy-Upstream-Service-Time -> Vector(14), X-Frame-Options -> Vector(DENY), X-Global-Transaction-ID -> Vector(283563769), X-Xss-Protection -> Vector(1)))"
                }
            ], 
            "source": "print(response_scoring)"
        }, 
        {
            "source": "As we can see we predict that a 55-year-old single male executive is interested in Mountaineering Equipment (prediction: 2.0).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "<a id=\"summary\"></a>\n## 6. Summary and next steps \n\nYou successfully completed this notebook! You learned how to use Apache Spark machine learning as well as Watson Machine Learning for model creation and deployment. Check out our [Online Documentation](https://dataplatform.ibm.com/docs/content/analyze-data/wml-setup.html) for more samples, tutorials, documentation, how-tos, and blog posts. \n \n### Authors\n\n**Umit Mert Cakmak** is a Data Scientist at IBM with a track record of developing enterprise-level applications that substantially increases clients' ability to turn data into actionable insights.\n\nCopyright \u00a9 2017, 2018 IBM. This notebook and its source code are released under the terms of the MIT License.", 
            "cell_type": "markdown", 
            "metadata": {}
        }
    ], 
    "metadata": {
        "kernelspec": {
            "display_name": "Scala 2.11 with Spark 2.1", 
            "name": "scala-spark21", 
            "language": "scala"
        }, 
        "language_info": {
            "mimetype": "text/x-scala", 
            "version": "2.11.8", 
            "name": "scala", 
            "pygments_lexer": "scala", 
            "file_extension": ".scala", 
            "codemirror_mode": "text/x-scala"
        }
    }, 
    "nbformat": 4
}