Amazon_Reviews_NLP_ML.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Amazon_Reviews_NLP_ML.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyOjNu2GlbwCou0cCRyk5tjK",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/susiexia/BigData_Amazon_reviews_ETL_Cloud/blob/master/Amazon_Reviews_NLP_ML.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KXDyQ0FFQaGW",
        "colab_type": "text"
      },
      "source": [
        "# Project PART 2\n",
        "Perform ETL in the cloud AND analyze data using Natural Language Processing (NLP) pipeline including Machine Learning.\n",
        "(part 1 in \"Amazon_Reriews_ETL_process.ipynb\")"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IXOBxcCBQRlR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Install Java, Spark, Findspark and download a Postgresql driver\n",
        "!apt-get install openjdk-8-jdk-headless -qq > /dev/null\n",
        "!wget -q http://www-us.apache.org/dist/spark/spark-2.4.5/spark-2.4.5-bin-hadoop2.7.tgz\n",
        "!tar xf spark-2.4.5-bin-hadoop2.7.tgz\n",
        "!pip install -q findspark\n",
        "\n",
        "\n",
        "# Set Environment Variables\n",
        "import os\n",
        "os.environ[\"JAVA_HOME\"] = \"/usr/lib/jvm/java-8-openjdk-amd64\"\n",
        "os.environ[\"SPARK_HOME\"] = \"/content/spark-2.4.5-bin-hadoop2.7\"\n",
        "\n",
        "# Start a SparkSession\n",
        "import findspark\n",
        "findspark.init()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7WBPh9V_Q2qr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create a spark session, configured with Posetgres driver\n",
        "from pyspark.sql import SparkSession\n",
        "spark = SparkSession.builder.appName('Amazon_Reviews_Analysis_NLP').getOrCreate()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KfDTHkZBO1os",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "a29860d4-b01c-4f51-ef72-7cab899240fd"
      },
      "source": [
        "\n",
        "# Read in data from S3 Bukets\n",
        "from pyspark import SparkFiles\n",
        "\n",
        "url= \"https://s3.amazonaws.com/amazon-reviews-pds/tsv/amazon_reviews_us_Beauty_v1_00.tsv.gz\"\n",
        "spark.sparkContext.addFile(url)\n",
        "df = spark.read.csv(SparkFiles.get('amazon_reviews_us_Beauty_v1_00.tsv.gz'), sep='\\t', header=True, inferSchema = True)\n",
        "df.show(n=5)"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+-----------+-----------+--------------+----------+--------------+--------------------+----------------+-----------+-------------+-----------+----+-----------------+--------------------+--------------------+-------------------+\n",
            "|marketplace|customer_id|     review_id|product_id|product_parent|       product_title|product_category|star_rating|helpful_votes|total_votes|vine|verified_purchase|     review_headline|         review_body|        review_date|\n",
            "+-----------+-----------+--------------+----------+--------------+--------------------+----------------+-----------+-------------+-----------+----+-----------------+--------------------+--------------------+-------------------+\n",
            "|         US|    1797882|R3I2DHQBR577SS|B001ANOOOE|       2102612|The Naked Bee Vit...|          Beauty|          5|            0|          0|   N|                Y|          Five Stars|Love this, excell...|2015-08-31 00:00:00|\n",
            "|         US|   18381298|R1QNE9NQFJC2Y4|B0016J22EQ|     106393691|Alba Botanica Sun...|          Beauty|          5|            0|          0|   N|                Y|Thank you Alba Bo...|The great thing a...|2015-08-31 00:00:00|\n",
            "|         US|   19242472|R3LIDG2Q4LJBAO|B00HU6UQAG|     375449471|Elysee Infusion S...|          Beauty|          5|            0|          0|   N|                Y|          Five Stars|Great Product, I'...|2015-08-31 00:00:00|\n",
            "|         US|   19551372|R3KSZHPAEVPEAL|B002HWS7RM|     255651889|Diane D722 Color,...|          Beauty|          5|            0|          0|   N|                Y|          GOOD DEAL!|I use them as sho...|2015-08-31 00:00:00|\n",
            "|         US|   14802407| RAI2OIG50KZ43|B00SM99KWU|     116158747|Biore UV Aqua Ric...|          Beauty|          5|            0|          0|   N|                Y|this soaks in qui...|This is my go-to ...|2015-08-31 00:00:00|\n",
            "+-----------+-----------+--------------+----------+--------------+--------------------+----------------+-----------+-------------+-----------+----+-----------------+--------------------+--------------------+-------------------+\n",
            "only showing top 5 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fiW-7A8rPU05",
        "colab_type": "text"
      },
      "source": [
        "# NLP pipeline"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cdR-esNaCNIN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# import ml.feature functions\n",
        "from pyspark.ml.feature import Tokenizer, StopWordsRemover, HashingTF, IDF, StringIndexer, VectorAssembler\n",
        "from pyspark.sql.functions import length"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mPDn5XuGlRim",
        "colab_type": "code",
        "outputId": "7e6c93fa-64a8-44bc-eade-96179ed0995f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# add a new column as a future feature\n",
        "df = df.withColumn('length', length(df.review_body))\n",
        "df.count()\n",
        "df.filter(df.review_body.isNotNull()).count()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "5114750"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tMv1-zL8zOoC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "amazon_df = df.filter(df.review_body.isNotNull())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vN6rnHxRl8SY",
        "colab_type": "code",
        "outputId": "160c27c0-af3a-46af-8a51-a6308ef1c4f3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "\n",
        "amazon_df = df.select('vine','review_body','length')\n",
        "amazon_df = amazon_df.dropna()\n",
        "amazon_df.count()\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "5114750"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8gRRr2a_Sc7f",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# create all the ml.features to df\n",
        "strIndexed = StringIndexer(inputCol='vine', outputCol='label')\n",
        "tokenizer = Tokenizer(inputCol='review_body', outputCol='tokened')\n",
        "stopremover = StopWordsRemover(inputCol='tokened', outputCol='removed')\n",
        "hashngTF = HashingTF(inputCol='removed', outputCol='hashed')\n",
        "idf = IDF(inputCol='hashed', outputCol='idf_token')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2t9Y2Yz-ldVv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from pyspark.ml.linalg import Vector\n",
        "# create feature vectors\n",
        "clean_up = VectorAssembler(inputCols=['idf_token', 'length'], outputCol='features')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eQ6TtJBeSdDb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# create a pipeline and list all stages in the order of being executed\n",
        "from pyspark.ml import Pipeline\n",
        "data_prep_pipeline = Pipeline(stages= [strIndexed, tokenizer, stopremover, hashngTF, idf, clean_up])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kC2A1XhRSdJ9",
        "colab_type": "code",
        "outputId": "e4dbdd5c-2b6b-4afa-b51e-604ace85e7ae",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 479
        }
      },
      "source": [
        "# fit model and transform the pipeline\n",
        "cleaner = data_prep_pipeline.fit(amazon_df)    # produce a PipelineModel\n",
        "\n",
        "# use PipelineModel to transform orginal df\n",
        "cleaned = cleaner.transform(amazon_df)\n",
        "cleaned.select('label','features').show(5)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
            "|label|features                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |\n",
            "+-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
            "|0.0  |(262145,[33496,70745,117481,127222,186480,262144],[4.759862126914744,7.224012100025549,3.858327008642361,5.143632157579753,1.7370537727958149,32.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |\n",
            "|0.0  |(262145,[12531,22346,26686,40553,53777,56749,69598,99187,99895,107499,124775,138356,143569,163287,164384,204923,208258,218401,229639,249180,250475,262144],[3.421471110579207,2.8633014421193472,6.16061567908389,2.317065694578102,4.251838425645906,5.539960518676435,3.7674043741873513,5.5250846605776776,2.2647623416012848,3.701754591225363,5.771742649536392,1.7758963222044868,13.501729126775718,5.091039961719385,5.595655566162018,3.3077651952156395,1.7493740231260482,9.098500284451234,3.039953197537011,1.2129066718497983,3.1041116948470093,190.0])                                                                                                                                                                                                                                                                                                                                                                                                                         |\n",
            "|0.0  |(262145,[4200,74473,118456,130480,138356,155725,250100,262144],[3.7063678157555686,3.2135721577813996,8.45922609383144,6.173854731298313,1.7758963222044868,3.8054151134654712,8.355065559856351,64.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |\n",
            "|0.0  |(262145,[6611,36658,72281,113458,116873,128231,132382,140750,142293,208258,211617,262144],[6.60013564990739,8.368454881221362,9.081168828099592,3.849371325696759,1.8277179016616159,3.4213274685349337,6.981107999217022,4.445939514648133,6.30933210675468,1.7493740231260482,4.640399268118853,100.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |\n",
            "|0.0  |(262145,[5381,7590,22323,22346,41862,57341,57400,66611,67734,74447,83532,90878,95225,99892,121367,154701,159715,159931,179490,187872,189603,195421,238924,239161,255198,262144],[3.2406520486581787,6.95412421176937,4.703317564836192,2.8633014421193472,6.560953636765447,4.490676280298323,2.2470748856325367,8.475033024529278,4.089836629728802,5.753084231515149,9.026066859980018,9.7812125877186,4.58545305517117,5.152177205369238,5.613555393433782,5.536580457798585,5.195856793801775,3.844626336580042,4.008984873236089,7.970035032633442,5.635718916820394,5.160523309795913,5.130024242108086,4.159258343090641,4.781945006803463,281.0])                                                                                                                                                                                                                                                                                                                                      |\n",
            "|0.0  |(262145,[113432,262144],[2.03122514911831,4.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |\n",
            "|0.0  |(262145,[8449,13957,16809,32890,54465,107332,138059,138356,141854,166544,181519,205678,262144],[4.451708423772322,3.2920076563932814,5.594971445624878,3.40895622578694,7.314932786137766,4.706103653482232,7.80415636875383,1.7758963222044868,3.3555531319215777,5.242418827620463,2.615625525641145,6.187176294451637,116.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |\n",
            "|0.0  |(262145,[67734,166027,240854,262144],[4.089836629728802,2.876618007829984,13.655879806602975,20.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |\n",
            "|0.0  |(262145,[14,2192,15775,21310,30006,31846,32890,33496,37219,44965,46505,52523,68826,72667,74397,75859,99895,113432,117280,132975,135754,138356,159715,166687,177769,188011,194740,203667,205876,212740,221257,223763,231428,232685,236821,237388,243808,245898,262144],[2.152407212906119,9.78815706007141,4.925566464061863,2.7775402039945214,3.9781747008716724,8.753077217309935,3.40895622578694,4.759862126914744,6.564137691507823,5.630690484712141,5.954001429720892,8.643024755768407,8.97220655912694,5.909651203460021,6.963175909037712,7.120880461319299,4.5295246832025695,2.03122514911831,8.392326432491279,3.652931608465215,14.43039823472139,1.7758963222044868,5.195856793801775,4.155959164891932,6.537863602137643,6.144081822143715,7.200681243262854,7.765156829296526,6.7271787631053765,4.7415673480365825,7.108138372825086,3.1356807526698027,3.1939628839324903,7.198813076522069,3.3982613768566363,4.258727972559987,6.835681507970371,7.575946611507386,423.0])|\n",
            "|0.0  |(262145,[8287,46252,52545,66246,120667,134271,174966,197339,206616,208258,225686,242361,262144],[4.115085270331143,3.9355141310315718,5.025238889346293,6.942316256985281,5.172553684698316,8.193461429374514,2.7119859866112486,4.093791609024617,6.011359265238637,1.7493740231260482,7.162125966751289,3.6578096115809,128.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |\n",
            "|0.0  |(262145,[2437,7336,55038,74975,86577,99895,100604,119364,123874,138356,144377,165816,167907,217310,222066,227662,233502,233878,239122,249180,262144],[3.665358655101906,6.956174232987525,7.678260666317047,4.5254668711665245,4.937979556749365,2.2647623416012848,4.199708531947814,12.151802409826702,4.013313393673446,1.7758963222044868,10.557290147609278,3.6729882118351354,9.666895760038702,7.046081428013718,9.962842342340377,7.2210621347446535,4.106974151860443,4.130947696868155,4.969731774003759,3.638720015549395,216.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                   |\n",
            "|0.0  |(262145,[5381,22346,31463,46903,65408,90787,141660,152097,191864,204923,218882,233654,249180,262144],[3.2406520486581787,2.8633014421193472,2.9623414669119055,6.316017488566254,5.705729918950958,6.787211916328886,9.73061157442481,5.753392369440121,3.9839565774392542,3.3077651952156395,4.11790374458489,7.8367864854357805,3.638720015549395,114.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |\n",
            "|0.0  |(262145,[5232,5381,6293,7148,12531,20719,27374,28206,28990,36200,74473,75042,83161,106491,129613,159931,179560,197339,226659,247704,262144],[4.628481183700227,3.2406520486581787,11.734067209126723,5.825652936671602,3.421471110579207,3.9653689138204364,4.442926185471987,7.4929159413331226,5.101262386540739,2.996696812874475,3.2135721577813996,3.2723042270477247,3.240007845836864,3.5748110818297287,2.765046777827325,3.844626336580042,4.427666773092163,4.093791609024617,7.782236133486313,11.053190121158591,171.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                           |\n",
            "|0.0  |(262145,[7367,12888,15786,24494,54961,62615,76764,80620,89717,99895,115917,154989,159826,165816,169540,170441,190256,212913,223999,253238,253962,262144],[4.37120529757409,2.4703825986070442,5.272332300056656,5.346243932048954,4.295798463288235,10.925696544461736,2.5916114379789343,7.12597446869603,2.988173253420803,2.2647623416012848,2.9827972813444483,5.6117348746186995,8.633096378571073,3.6729882118351354,6.9411026646033225,5.7116831946162225,3.4817419014236615,6.959051341425128,5.544757917716332,6.353608026255231,5.5765199683276325,203.0])                                                                                                                                                                                                                                                                                                                                                                                                                           |\n",
            "|0.0  |(262145,[16994,82495,117170,125372,135514,174966,243875,262144],[4.78702086299341,4.483431697228043,7.754157634995855,2.137311956535461,6.507534062702908,2.7119859866112486,5.665754545054152,58.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |\n",
            "|0.0  |(262145,[3121,8769,80023,81008,109810,115917,118433,131320,133480,177873,186480,205869,240064,249180,262144],[5.124389310485914,6.782543454491298,5.513137669765734,1.7088650642486423,5.5611961579041616,2.9827972813444483,6.196060995838604,9.798665037669824,5.9849072781816535,3.5458795654165187,1.7370537727958149,6.282505496692927,7.024756763886035,2.4258133436995966,148.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |\n",
            "|0.0  |(262145,[14,21028,27564,28533,36127,56407,65212,65234,71225,81008,89457,99895,107367,125987,128231,138356,154312,160278,165567,171911,172310,172428,181519,186480,195132,208129,208258,213813,229639,237388,256961,262144],[2.152407212906119,5.002729001551534,5.708554779586512,7.8808107966227,5.4191503581321,3.6438817049779137,3.5177615995438507,6.5564028463149695,4.164944919676907,1.7088650642486423,4.60351966003459,2.2647623416012848,3.424996801380606,10.03153887362661,3.4213274685349337,1.7758963222044868,7.382688384081887,10.3787350736108,4.336012610099551,6.326020833915653,7.242694110811824,7.076628594592875,2.615625525641145,3.4741075455916297,3.3982438324326925,6.6457183717894495,1.7493740231260482,6.285124533337453,3.039953197537011,4.258727972559987,6.141988724050523,355.0])                                                                                                                                                                         |\n",
            "|0.0  |(262145,[9129,18025,40553,51209,58236,92646,118590,262144],[4.23041461423996,7.413332339491543,2.317065694578102,8.002806001938838,4.0751719955564045,4.653568618964554,3.6410819734163513,50.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |\n",
            "|0.0  |(262145,[14,1645,9195,12888,20590,40553,40698,47032,60821,61899,84377,96290,107332,112869,113834,114250,133521,155378,159464,159931,168220,178737,181654,208258,219519,227800,242223,252165,262144],[2.152407212906119,4.091262363288062,7.324971252484624,2.4703825986070442,6.30493555257066,4.634131389156204,5.875716023563533,3.6589474493569845,11.709969657547664,3.0005831608583167,5.655809804837785,5.432744525720772,4.706103653482232,6.098232855731851,9.890811214131492,10.527658350002906,4.746531921138426,5.453854085314123,4.098457018893565,3.844626336580042,6.800295399949748,3.7519173668436365,4.295339405496275,1.7493740231260482,6.486914775500173,5.981339723599016,12.413833340529404,10.363118644184267,334.0])                                                                                                                                                                                                                                                   |\n",
            "|0.0  |(262145,[39951,53106,93604,117481,262144],[5.915577789179644,3.462518834371209,5.066272934677693,3.858327008642361,29.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |\n",
            "+-----+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
            "only showing top 20 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y6XqFt8aTloa",
        "colab_type": "text"
      },
      "source": [
        "# Run ML model and Evaluate"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-UiJG2nuTv0i",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from pyspark.ml.classification import NaiveBayes\n",
        "from pyspark.ml.evaluation import MulticlassClassificationEvaluator"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Np9mEXAmTv3Z",
        "colab_type": "code",
        "outputId": "c5e390bd-4da9-4c46-8e89-6ff842f13bde",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 479
        }
      },
      "source": [
        "# break whole data down into a training set and a testing set\n",
        "training, testing = cleaned.randomSplit([0.7,0.3])\n",
        "\n",
        "#create a Naive Bayes Model \n",
        "nb = NaiveBayes() \n",
        "predictor = nb.fit(training)     # fit training df to nb model, predictor is NaiveBayes object\n",
        "\n",
        "# transform the model with teasting data\n",
        "test_results = predictor.transform(testing)\n",
        "test_results.select('features', 'rawPrediction','probability','prediction').show(5)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------+-------------------------------------------+----------+\n",
            "|features                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |rawPrediction                            |probability                                |prediction|\n",
            "+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------+-------------------------------------------+----------+\n",
            "|(262145,[6061,85387,126768,132612,150903,202268,262144],[6.590835919102654,9.618693658220824,2.0590054269753533,6.717595322586016,4.186604119214526,4.953785056129397,33.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |[-378.7538110808805,-414.49677191676045] |[0.9999999999999998,2.9993651572503603E-16]|0.0       |\n",
            "|(262145,[4200,6057,12329,12531,35583,75107,100079,101376,102296,107160,107231,125011,128342,145480,154125,155052,161061,179827,184562,193568,202162,217836,221790,234263,249180,262144],[3.7063678157555686,12.808581946215773,9.360499244624936,3.421471110579207,13.368197734151195,5.219029367869277,3.012540415828492,6.4039439812637955,7.263962693210374,8.840989089632815,4.435110670386347,4.890577317854781,7.804635640270313,10.564837353244661,3.5739741391323694,5.066489992307346,2.2357643162232317,6.245431887085037,4.224478049516908,4.86221716994906,11.496395557249604,8.266808076786475,4.6977900152473095,3.4559548714454644,8.490346702948589,264.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |[-1964.9089182985617,-2035.0566653359242]|[1.0,3.4294189532841716E-31]               |0.0       |\n",
            "|(262145,[67435,75609,77886,91878,117481,126412,158870,165837,178935,190355,212164,221790,224761,249180,261271,262144],[10.534984390094978,10.470905533410457,4.565788007222969,3.0746252442956385,3.858327008642361,6.314828106364086,4.919631409781135,10.628303462368413,4.427617659992871,4.027838380851278,3.821036971880609,4.6977900152473095,3.893619853325841,3.638720015549395,6.2938692553432425,143.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |[-924.9733793765316,-984.0827235380505]  |[1.0,2.1337228187582882E-26]               |0.0       |\n",
            "|(262145,[28990,262144],[5.101262386540739,1.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |[-47.3547780040362,-66.25352803709377]   |[0.9999999938002075,6.199792338433357E-9]  |0.0       |\n",
            "|(262145,[28990,32108,87603,90312,99270,215378,249180,262144],[5.101262386540739,7.218128156866575,3.058000015579965,8.102920221681359,3.1107971733115107,4.398560433432799,1.2129066718497983,59.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |[-342.144911950897,-395.6710181238309]   |[1.0,5.6742367386109E-24]                  |0.0       |\n",
            "|(262145,[1129,11042,16889,16994,28990,32983,33182,42389,47977,53121,57400,71487,81008,112676,115917,135568,147136,171911,178119,195499,199496,222499,224769,249180,249943,257680,259523,262144],[6.980897627910892,6.5263151993197654,12.739589074728821,4.78702086299341,5.101262386540739,5.307578751669032,4.716342539491562,5.1633892278271105,10.040467504370913,8.896558940787626,2.2470748856325367,4.569365693830462,5.126595192745927,4.62760102521489,2.9827972813444483,4.783415990175748,2.729222160565597,6.326020833915653,5.585129980023461,7.925780023629401,5.021525995413161,4.24168744122697,3.5994351434028107,4.851626687399193,4.092350559972893,8.178022526222861,4.56278511468456,280.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |[-1707.876609964251,-1839.6211605675844] |[1.0,6.082311412855922E-58]                |0.0       |\n",
            "|(262145,[14,20030,22552,27293,28990,33209,54050,57328,76764,79737,110693,111853,115892,115917,121058,125372,147424,161061,161782,181295,196578,202268,213384,220960,241341,243094,245615,249180,262144],[2.152407212906119,11.053190121158591,3.88344101668338,4.808903366716322,5.101262386540739,3.694697410057219,7.790356482852841,6.596691823860629,2.5916114379789343,4.3094508976179435,3.433653715025092,3.924426902992667,3.9039324552713675,2.9827972813444483,7.743729066214719,2.137311956535461,6.377595989393571,2.2357643162232317,10.576152847491437,8.59328477357601,5.038901613862098,4.953785056129397,5.870366745463468,3.0635334220103516,4.739753907937228,4.440153962350456,8.404479359842691,1.2129066718497983,275.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |[-1537.5478351178494,-1637.4901863612658]|[1.0,3.940835861196776E-44]                |0.0       |\n",
            "|(262145,[1645,2437,5381,8804,11951,12329,21683,26445,28511,28990,29689,31463,36200,49403,50486,54150,54961,57155,61899,65212,73477,73859,75107,76764,78584,83384,90859,91878,93923,96486,96984,97777,99895,105358,106491,107231,116873,117550,125372,127750,128206,130480,135171,137033,140908,164390,168817,172477,174966,175119,177873,178737,185208,186593,186925,188762,188822,195132,195285,196766,201238,203627,203944,208258,212276,212685,213042,215995,231746,236136,236821,237761,242223,247394,250475,254415,262144],[4.091262363288062,3.665358655101906,3.2406520486581787,4.463533697999072,4.656260465942446,4.680249622312468,4.814046001749426,4.982566447503122,5.358917745053914,5.101262386540739,8.15598306665657,2.9623414669119055,2.996696812874475,6.036564574053768,6.130149988337805,6.688912667988993,4.295798463288235,10.883291084363195,6.001166321716633,3.5177615995438507,5.842153528153538,6.883753356422811,5.219029367869277,2.5916114379789343,5.634240558076589,10.305975719328371,5.930770543606707,3.0746252442956385,7.666500765986015,6.696006573358733,3.183627965380411,6.905363304047331,2.2647623416012848,5.297799586843812,3.5748110818297287,4.435110670386347,1.8277179016616159,8.609234074983688,2.137311956535461,5.904547868035489,8.069255562834316,6.173854731298313,9.429046061334796,6.343326308500378,5.710147277490084,5.724894526975643,5.10409197042224,6.6079907571503425,5.423971973222497,4.97416136636094,3.5458795654165187,3.7519173668436365,5.488480525924442,4.7197350813491905,5.360110993622002,5.622545790722915,2.257349995948794,3.3982438324326925,11.170973156814975,3.5220943562311797,6.061834134524107,7.138947358992133,14.361203295326112,1.7493740231260482,7.175324127875009,6.389634565158546,6.679220548973627,3.227574971674121,5.007746075926143,7.212808995388975,3.3982613768566363,4.27263709974933,8.27588889368627,3.1047481460862514,3.1041116948470093,8.36677137914025,901.0])|[-4528.814783196147,-4588.586484775903]  |[1.0,1.1002207462597217E-26]               |0.0       |\n",
            "|(262145,[2916,22346,23650,41704,45531,50091,50521,58900,62293,74079,75042,82461,103260,104288,112802,135240,136894,142107,142276,147490,173517,174966,175119,175143,182338,196766,204515,205991,229639,230444,234263,236920,240017,241791,247394,249180,258565,261622,262144],[5.662259811775232,2.8633014421193472,5.400354455547136,5.313953117507016,3.324529848516645,6.429549591726688,8.61030646114544,4.30097743676983,4.307445549352331,4.565111569767638,3.2723042270477247,4.772147152657463,3.9423025325094785,5.093012411695564,4.560370581397293,4.924677993921974,5.950842582328886,7.432311966809312,6.946372234965053,6.863161337609197,6.447526229902512,2.7119859866112486,4.97416136636094,4.264552100845975,9.788972642693837,3.5220943562311797,6.137544070717856,7.117739982835306,3.039953197537011,6.97900627503017,3.4559548714454644,8.34185314634976,3.910753173415959,8.414132791543334,3.1047481460862514,1.2129066718497983,5.084166864744288,8.35091789733627,352.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |[-2149.797066434889,-2175.995321523333]  |[0.9999999999958098,4.190273588890504E-12] |0.0       |\n",
            "|(262145,[5381,74925,94472,110360,115892,173517,178737,249180,262144],[3.2406520486581787,10.227283450752706,5.4768669764893065,8.906609276641127,3.9039324552713675,12.895052459805024,3.7519173668436365,1.2129066718497983,72.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |[-566.8185733061056,-632.6304480593178]  |[1.0,2.619786808380518E-29]                |0.0       |\n",
            "|(262145,[1998,17218,20719,32154,33182,36246,36543,36702,40553,63422,70389,82321,89663,99469,100282,103376,104350,109230,116434,116873,118642,127042,133774,138935,143199,157070,168817,170414,172805,173517,175529,182344,188123,190266,197219,204380,210812,222992,225667,230406,231428,231866,235518,245806,253348,261393,261666,262144],[4.430897157889923,8.076779109294314,3.9653689138204364,6.456201460911804,4.716342539491562,11.288756192471359,9.08979700932293,4.962466970584252,4.634131389156204,4.324945538328378,2.7722538646464248,5.900112636499978,4.836534737140397,11.458655229266757,6.777210108515935,5.581905512906819,7.400129764849609,4.372629799239475,6.7851343532547395,1.8277179016616159,5.184905088988352,7.374859942661533,4.84778473189376,5.25641965209308,5.826713593052112,7.22777453391838,5.10409197042224,4.168665560891616,6.286909404408183,6.447526229902512,4.266250248603136,3.714005221662144,5.8567414314528685,4.5799092738875755,4.197456146788302,4.003495990259177,4.957004151458148,4.508071081994444,5.929299630640172,4.43186666770615,3.1939628839324903,9.47382966396177,8.23387096771239,3.1554393940484853,7.120638535589317,6.150479211902288,10.317740560907957,503.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |[-3079.5618686670364,-3245.567375201225] |[1.0,8.030166434862669E-73]                |0.0       |\n",
            "|(262145,[16426,232989,262144],[4.108555424222221,7.499960704529464,12.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |[-125.58462314585708,-160.55483096840942]|[0.9999999999999993,6.495786041478269E-16] |0.0       |\n",
            "|(262145,[113432,254719,262144],[2.03122514911831,16.764051824466627,20.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |[-226.78716086126494,-280.2974011178843] |[1.0,5.764981676072927E-24]                |0.0       |\n",
            "|(262145,[12946,13957,107406,158870,177873,184351,203001,206582,230406,254719,259904,260748,262144],[4.421334745929975,3.2920076563932814,6.709224378663283,4.919631409781135,3.5458795654165187,5.423528280971164,4.108293687718002,6.760197108855108,4.43186666770615,8.382025912233313,12.503200296664591,8.46849400076222,133.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |[-869.6681780124914,-973.5257546174631]  |[1.0,7.856472173161296E-46]                |0.0       |\n",
            "|(262145,[86813,110693,149224,179333,205305,208258,215995,231158,253534,261610,262144],[6.478988372456156,3.433653715025092,9.203472375167294,12.312145059901882,5.460592130195972,3.4987480462520963,3.227574971674121,10.765508048706812,4.6858079513755975,5.633967027617267,106.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |[-808.8624873659428,-851.9170967530952]  |[1.0,2.0027222493300428E-19]               |0.0       |\n",
            "|(262145,[16426,30445,55639,64175,65973,84547,89717,186009,188304,213555,226981,238301,262144],[8.217110848444442,6.408324500906951,3.714550696639973,5.705729918950958,7.412684251328872,8.442757286118171,2.988173253420803,5.489237492417511,5.698177410934011,12.96273262604303,8.701227147257656,3.455521279188823,147.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |[-958.1235738922087,-1043.9617728113149] |[1.0,5.259492896894272E-38]                |0.0       |\n",
            "|(262145,[181653,262144],[13.83820136339693,9.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |[-257.31961644245837,-247.49829488702215]|[5.427885401117158E-5,0.9999457211459889]  |1.0       |\n",
            "|(262145,[11175,11275,53846,76757,101702,102787,130218,156694,171911,172772,198790,208258,214894,229639,249180,260599,262144],[6.83168731239153,5.735009611111144,10.385044242804064,8.142451060437994,5.483715354491072,3.4580573607849665,6.875579182973953,7.719663733725472,6.326020833915653,9.170995786489387,3.7327612101176997,3.4987480462520963,12.67505055359125,3.039953197537011,1.2129066718497983,12.18954273780955,145.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |[-1344.7247468783846,-1335.9548914684578]|[1.5532191187647115E-4,0.9998446780881234] |1.0       |\n",
            "|(262145,[16146,262144],[11.37010183192531,11.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |[-173.71094378633302,-177.22357010435854]|[0.9710448989912107,0.028955101008789324]  |0.0       |\n",
            "|(262145,[16266,21310,23732,27922,36200,37191,57400,115157,116562,127222,130847,158129,170380,208258,221657,229407,237599,241667,255940,262144],[6.19654091138268,2.7775402039945214,10.317740560907957,12.739589074728821,2.996696812874475,9.130474589083747,2.2470748856325367,4.540289771172604,7.304994089888236,5.143632157579753,7.006032071371388,3.687806859027875,6.354057464465043,1.7493740231260482,7.074316454834496,4.313108647446303,5.825520433690092,6.352934247453772,4.29883066294859,164.0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |[-1286.9031448417395,-1341.3133957408807]|[1.0,2.3438416939327508E-24]               |0.0       |\n",
            "+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------+-------------------------------------------+----------+\n",
            "only showing top 20 rows\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "laVYRoAVTv5-",
        "colab_type": "code",
        "outputId": "965fbefd-f9fc-4d9d-f573-58c2afdab444",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# use the Class Evaluator for a cleaner description\n",
        "acc_eval = MulticlassClassificationEvaluator()\n",
        "\n",
        "acc = acc_eval.evaluate(test_results)    # action (evaluate)\n",
        "\n",
        "print(\"Accuracy of model at predicting reviews was : %f \"% acc)"
      ],
      "execution_count": 68,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Accuracy of model at predicting reviews was : 0.959364 \n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}