HackerNews/#5 Market Basket Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Market Basket Analysis\n",
    "\n",
    "<img src = \"img/basket.jpg\" height=\"300\" width=\"300\">\n",
    "\n",
    "\n",
    "Affinity analysis is a data analysis and data mining technique that discovers co-occurrence relationships among activities performed by (or recorded about) specific individuals or groups. In general, this can be applied to any process where agents can be uniquely identified and information about their activities can be recorded. In retail, affinity analysis is used to perform market basket analysis, in which retailers seek to understand the purchase behavior of customers. This information can then be used for purposes of cross-selling and up-selling, in addition to influencing sales promotions, loyalty programs, store design, and discount plans\n",
    "\n",
    "**Example** \n",
    "\n",
    "1. Market basket analysis might tell a retailer that customers often purchase shampoo and conditioner together, so putting both items on promotion at the same time would not create a significant increase in revenue, while a promotion involving just one of the items would likely drive sales of the other.\n",
    "\n",
    "2.  one super market chain discovered in its analysis that male customers that bought diapers often bought beer as well, have put the diapers close to beer coolers, and their sales increased dramatically\n",
    "\n",
    "*src*: https://en.wikipedia.org/wiki/Affinity_analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the user-comment data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "user_stories = pd.read_csv(\"data/story_user_comment.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>story</th>\n",
       "      <th>comment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>21</td>\n",
       "      <td>14356377</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>21</td>\n",
       "      <td>15131370</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21</td>\n",
       "      <td>15196309</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>47</td>\n",
       "      <td>15601729</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>47</td>\n",
       "      <td>14023198</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  user     story  comment\n",
       "0   21  14356377        1\n",
       "1   21  15131370        1\n",
       "2   21  15196309        1\n",
       "3   47  15601729        1\n",
       "4   47  14023198        1"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_stories.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(52460, 3)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_stories.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "user_stories_sorted = user_stories.sort_values(\"user\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>story</th>\n",
       "      <th>comment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1298</th>\n",
       "      <td>0003</td>\n",
       "      <td>14384187</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21446</th>\n",
       "      <td>01000001</td>\n",
       "      <td>13816627</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43320</th>\n",
       "      <td>010001001010</td>\n",
       "      <td>14238005</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39685</th>\n",
       "      <td>01096232042</td>\n",
       "      <td>15357584</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39684</th>\n",
       "      <td>01096232042</td>\n",
       "      <td>15357584</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               user     story  comment\n",
       "1298           0003  14384187        1\n",
       "21446      01000001  13816627        1\n",
       "43320  010001001010  14238005        1\n",
       "39685   01096232042  15357584        1\n",
       "39684   01096232042  15357584        1"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_stories_sorted.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "unique_users = pd.unique(user_stories_sorted.user)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For market basket analysis, we need the baskets. Let's create the transaction matrix of various baskets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "basket = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for user in unique_users[:100]:\n",
    "    user_data = user_stories_sorted[user_stories_sorted.user == user].copy()\n",
    "    user_stories = pd.unique(user_data.story)\n",
    "    user_stories = [data.tolist() for data in user_stories]\n",
    "    basket.append(user_stories)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[14384187],\n",
       " [13816627],\n",
       " [14238005],\n",
       " [15357584],\n",
       " [13696004, 13507993, 13338592],\n",
       " [15744359],\n",
       " [14998429],\n",
       " [14446261],\n",
       " [14087381],\n",
       " [15345483, 14702676],\n",
       " [14384187],\n",
       " [13816627],\n",
       " [14238005],\n",
       " [15357584],\n",
       " [13696004, 13507993, 13338592],\n",
       " [15744359],\n",
       " [14998429],\n",
       " [14446261],\n",
       " [14087381],\n",
       " [15345483, 14702676],\n",
       " [14044517],\n",
       " [15752022, 15254952, 14039135],\n",
       " [13559662],\n",
       " [15742287],\n",
       " [15601729],\n",
       " [13361019],\n",
       " [13601451],\n",
       " [13423629],\n",
       " [14552615],\n",
       " [14308754,\n",
       "  15298833,\n",
       "  13713480,\n",
       "  15253781,\n",
       "  15316175,\n",
       "  14592745,\n",
       "  14529376,\n",
       "  14643467],\n",
       " [15200221, 15066729],\n",
       " [15675582, 14923362],\n",
       " [14997799],\n",
       " [13642662, 13559581],\n",
       " [14068280],\n",
       " [15878197, 14136081, 15234207, 13397145, 13338592],\n",
       " [15694926],\n",
       " [14417758],\n",
       " [15820161],\n",
       " [14170041, 13823979, 13597949],\n",
       " [15131370],\n",
       " [15037960],\n",
       " [14621347],\n",
       " [15756684],\n",
       " [14952787],\n",
       " [15423202, 15309989],\n",
       " [13354329],\n",
       " [15234207, 13500346, 15127633, 13755673],\n",
       " [13489100,\n",
       "  14361425,\n",
       "  14101233,\n",
       "  14744068,\n",
       "  13881535,\n",
       "  14043631,\n",
       "  13867316,\n",
       "  13520891,\n",
       "  14213220,\n",
       "  14776357],\n",
       " [15747650],\n",
       " [14923362],\n",
       " [15913250],\n",
       " [13301832],\n",
       " [13682949, 14486657],\n",
       " [15316175,\n",
       "  15050841,\n",
       "  14779881,\n",
       "  15228712,\n",
       "  15332819,\n",
       "  15704376,\n",
       "  15037960,\n",
       "  15021427,\n",
       "  15062225,\n",
       "  15633785,\n",
       "  15651058,\n",
       "  15786802,\n",
       "  14931004,\n",
       "  15511285],\n",
       " [13601451],\n",
       " [15821575],\n",
       " [14960597, 13682949],\n",
       " [14435219, 13713480, 15145176, 13682949, 14170575, 13955981],\n",
       " [14108797, 13939645, 14050049, 13319904, 15694118, 13816627, 13870199],\n",
       " [14475884],\n",
       " [15089476],\n",
       " [14023198, 15148885, 14238005],\n",
       " [15669262],\n",
       " [14973506],\n",
       " [14178397],\n",
       " [15893911],\n",
       " [15046601],\n",
       " [15089476, 13858656, 15155833, 15460851, 13844389],\n",
       " [14729400],\n",
       " [14394150, 14266423],\n",
       " [14673777, 15291961, 15037960, 14051851, 14630650, 15256603, 14495405],\n",
       " [13453189],\n",
       " [14002821],\n",
       " [13552747, 14645452, 15858197, 13561388],\n",
       " [15316175],\n",
       " [14497295],\n",
       " [15502074,\n",
       "  15237896,\n",
       "  15368104,\n",
       "  15857833,\n",
       "  15480334,\n",
       "  15367531,\n",
       "  15694118,\n",
       "  15023766],\n",
       " [15585769],\n",
       " [13769727],\n",
       " [14280569,\n",
       "  14310617,\n",
       "  15769807,\n",
       "  14105721,\n",
       "  15739744,\n",
       "  15538666,\n",
       "  14952787,\n",
       "  14023198,\n",
       "  15651510,\n",
       "  14141454,\n",
       "  14458955,\n",
       "  14560042,\n",
       "  13976938,\n",
       "  14338328,\n",
       "  14108797],\n",
       " [13653819, 15704730],\n",
       " [15793473],\n",
       " [14505378],\n",
       " [13629728],\n",
       " [13365430],\n",
       " [15543371, 15662004],\n",
       " [13301832],\n",
       " [14892980, 13673962],\n",
       " [14229412],\n",
       " [13755673],\n",
       " [15168467],\n",
       " [14962634],\n",
       " [13539552],\n",
       " [14245354, 14805265],\n",
       " [13537052],\n",
       " [14947361],\n",
       " [14049868,\n",
       "  14552615,\n",
       "  14063609,\n",
       "  14039135,\n",
       "  14560106,\n",
       "  14108849,\n",
       "  14118290,\n",
       "  14790251,\n",
       "  14452832,\n",
       "  14537747,\n",
       "  13786220],\n",
       " [14508921],\n",
       " [14477851],\n",
       " [14997941],\n",
       " [15763013],\n",
       " [13541679],\n",
       " [14291412, 14146850],\n",
       " [13417037],\n",
       " [15147660],\n",
       " [14338212],\n",
       " [13941923],\n",
       " [14196077],\n",
       " [13746830]]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "basket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "110"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(basket)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# FP Growth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pyfpgrowth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create FP Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "?pyfpgrowth.pyfpgrowth.FPTree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hn_fptree = pyfpgrowth.pyfpgrowth.FPTree(basket, 0.1, 0.1, 0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# List frequency for various items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "hn_fptree.frequent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the frequent patterns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "patterns = pyfpgrowth.find_frequent_patterns(basket, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "patterns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate the association rules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rules = pyfpgrowth.generate_association_rules(patterns, 0.7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "rules"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}