DOTA 2 Player Segmentation-Daskified.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DOTA 2 Player Segmentation through Unsupervised Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Riley Stange<br>\n",
    "Thinkful Data Science Capstone 3: Unsupervised Learning<br>\n",
    "December 2019\n",
    "\n",
    "The goal of this project is to take very detailed player gameplay statistics from Defense of The Ancients 2 (DOTA 2), and create clusters of player segmentations. DOTA 2 is a Multiplayer Online Battle Arena (MOBA) game, where two teams of 5 players have the goal of destroying the opposing team's \"ancient,\" a stationary structure that is within each team's base, at opposite sides of the map. Each team has neverending waves of \"minions,\" or computer controlled characters that continue to go down the three lanes, and, if the players were to not interfere, would end up in a deadlock at the center point of each lane. Players take turns at random choosing one character from an extensive roster of 119 (as of writing) characters, known as \"heroes.\" When a player chooses a hero, no one else in the game can play as a copy of that character, as one could in some other similar games."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from pandas.io.json import json_normalize\n",
    "\n",
    "from dask_ml.preprocessing import StandardScaler\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.cluster import KMeans, AgglomerativeClustering, DBSCAN\n",
    "from sklearn.manifold import TSNE\n",
    "from sklearn import metrics\n",
    "from dask_ml.cluster import KMeans\n",
    "\n",
    "from hdbscan import HDBSCAN\n",
    "\n",
    "from umap import UMAP\n",
    "\n",
    "import warnings\n",
    "\n",
    "from dask.distributed import Client, progress\n",
    "import dask.dataframe as dd\n",
    "import joblib\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": "<table style=\"border: 2px solid white;\">\n<tr>\n<td style=\"vertical-align: top; border: 0px solid white\">\n<h3 style=\"text-align: left;\">Client</h3>\n<ul style=\"text-align: left; list-style: none; margin: 0; padding: 0;\">\n  <li><b>Scheduler: </b>tcp://127.0.0.1:55084</li>\n  <li><b>Dashboard: </b><a href='http://127.0.0.1:8787/status' target='_blank'>http://127.0.0.1:8787/status</a>\n</ul>\n</td>\n<td style=\"vertical-align: top; border: 0px solid white\">\n<h3 style=\"text-align: left;\">Cluster</h3>\n<ul style=\"text-align: left; list-style:none; margin: 0; padding: 0;\">\n  <li><b>Workers: </b>4</li>\n  <li><b>Cores: </b>8</li>\n  <li><b>Memory: </b>8.00 GB</li>\n</ul>\n</td>\n</tr>\n</table>",
      "text/plain": "<Client: 'tcp://127.0.0.1:55084' processes=4 threads=8, memory=8.00 GB>"
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client = Client(n_workers=4, threads_per_worker=2, memory_limit='2GB')\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Exploration and Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data was gathered from [OpenDota's second data dump](https://blog.opendota.com/2017/03/24/datadump2/), downloaded from their small subsample that they linked, which is originally comprised of a 4GB file, containing match information, and a 1GB file containing the skill rank of the matches, with the match id, so that it can be merged with the match data for further filtering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Process to Import Data Before Dask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "This is how I loaded and filtered my data from a 4GB file that would've otherwise crashed my machine.\n",
    "```\n",
    "i = 1\n",
    "for chunk in pd.read_csv(\"~\\projects\\Dota 2\\player_matches_small.csv\", chunksize=2500000, low_memory=False):\n",
    "    chunk = chunk.loc[chunk['ability_uses'].notnull()].drop(\"account_id\", 1)\n",
    "    chunk.to_csv(\"~\\projects\\Dota 2\\player_stats{}.csv\".format(i), index=False)\n",
    "    i += 1\n",
    "```\n",
    "I then appended these files into one dataframe\n",
    "```\n",
    "i = 16\n",
    "\n",
    "player_stats = pd.DataFrame()\n",
    "\n",
    "while i > 0:\n",
    "    player_stats = player_stats.append(\n",
    "        pd.read_csv(\n",
    "            \"~\\projects\\Dota 2\\player_stats{}.csv\".format(i)\n",
    "        )\n",
    "    )\n",
    "    i -= 1\n",
    "```\n",
    "I then did the same for the file containing match_id and skill bracket, 2 being the highest, which is what I desire for my project.\n",
    "```\n",
    "i = 1\n",
    "for chunk in pd.read_csv(\"~\\projects\\Dota 2\\match_skill.csv\", chunksize=2500000):\n",
    "    chunk = chunk.loc[chunk['skill'] == 2]\n",
    "    chunk.to_csv(\"~\\projects\\Dota 2\\match_skill{}.csv\".format(i), index=False)\n",
    "    i += 1\n",
    "```\n",
    "Now with the data\n",
    "```\n",
    "i = 53\n",
    "\n",
    "high_skill = pd.DataFrame()\n",
    "\n",
    "while i > 0:\n",
    "    high_skill = high_skill.append(\n",
    "        pd.read_csv(\n",
    "            \"~\\projects\\Dota 2\\match_skill{}.csv\".format(i)\n",
    "        )\n",
    "    )\n",
    "    i -= 1\n",
    "```\n",
    "I then performed an inner merge function to combine the two dataframes into one, using match_id as the key\n",
    "```\n",
    "skilled_player_matches = pd.merge(left=high_skill, right=player_stats, on=\"match_id\")\n",
    "```\n",
    "And for later use, I wrote the result of the dataframe to a new CSV file, allowing me to dispose of all the small CSV files I generated earlier.\n",
    "```\n",
    "skilled_player_matches.to_csv(\"~\\projects\\Dota 2\\skilled_player_matches.csv\", index=False)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Process to Import Data With Dask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above method works, but it is clumsy and does not lend itself well to reproducibility. \n",
    "\n",
    "First, to prevent Dask related issues, I must implicitly state the datatype of each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "dtypes = {'match_id': 'int64',\n",
    " 'skill': 'int64',\n",
    " 'player_slot': 'int64',\n",
    " 'hero_id': 'int64',\n",
    " 'item_0': 'int64',\n",
    " 'item_1': 'int64',\n",
    " 'item_2': 'int64',\n",
    " 'item_3': 'int64',\n",
    " 'item_4': 'int64',\n",
    " 'item_5': 'int64',\n",
    " 'kills': 'int64',\n",
    " 'deaths': 'int64',\n",
    " 'assists': 'int64',\n",
    " 'leaver_status': 'float64',\n",
    " 'gold': 'float64',\n",
    " 'last_hits': 'int64',\n",
    " 'denies': 'int64',\n",
    " 'gold_per_min': 'int64',\n",
    " 'xp_per_min': 'int64',\n",
    " 'gold_spent': 'float64',\n",
    " 'hero_damage': 'float64',\n",
    " 'tower_damage': 'float64',\n",
    " 'hero_healing': 'float64',\n",
    " 'level': 'int64',\n",
    " 'additional_units': 'object',\n",
    " 'stuns': 'float64',\n",
    " 'max_hero_hit': 'object',\n",
    " 'times': 'object',\n",
    " 'gold_t': 'object',\n",
    " 'lh_t': 'object',\n",
    " 'xp_t': 'object',\n",
    " 'obs_log': 'object',\n",
    " 'sen_log': 'object',\n",
    " 'purchase_log': 'object',\n",
    " 'kills_log': 'object',\n",
    " 'buyback_log': 'object',\n",
    " 'lane_pos': 'object',\n",
    " 'obs': 'object',\n",
    " 'sen': 'object',\n",
    " 'actions': 'object',\n",
    " 'pings': 'object',\n",
    " 'purchase': 'object',\n",
    " 'gold_reasons': 'object',\n",
    " 'xp_reasons': 'object',\n",
    " 'killed': 'object',\n",
    " 'item_uses': 'object',\n",
    " 'ability_uses': 'object',\n",
    " 'hero_hits': 'object',\n",
    " 'damage': 'object',\n",
    " 'damage_taken': 'object',\n",
    " 'damage_inflictor': 'object',\n",
    " 'runes': 'object',\n",
    " 'killed_by': 'object',\n",
    " 'kill_streaks': 'object',\n",
    " 'multi_kills': 'object',\n",
    " 'life_state': 'object'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "player_stats = dd.read_csv(\"player_matches_small.csv\", dtype=dtypes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now I filter out the data I do not need, like rows that contain null values in the \"ability uses\" column, and dropping \"account_id\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "player_stats = player_stats.loc[player_stats[\"ability_uses\"].notnull()].drop(columns=\"account_id\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to read in the seperate CSV that contains data that will allow me to remove observations that are not of the highest skilled players (2). Dropping the data that does not contain \"2\" values under \"skill\" will help to eliminate unnecessary memory overhead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "match_skill = dd.read_csv(\"match_skill.csv\")\n",
    "match_skill = match_skill.loc[match_skill[\"skill\"] == 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the Dask tasks are defined in their proper order, I can compute it into a Pandas dataframe, which I will explain later why this is suitable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "skilled_player_stats = dd.merge(left=match_skill, right=player_stats, on=\"match_id\").compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 9289 entries, 0 to 129\n",
      "Data columns (total 56 columns):\n",
      "match_id            9289 non-null int64\n",
      "skill               9289 non-null int64\n",
      "player_slot         9289 non-null int64\n",
      "hero_id             9289 non-null int64\n",
      "item_0              9289 non-null int64\n",
      "item_1              9289 non-null int64\n",
      "item_2              9289 non-null int64\n",
      "item_3              9289 non-null int64\n",
      "item_4              9289 non-null int64\n",
      "item_5              9289 non-null int64\n",
      "kills               9289 non-null int64\n",
      "deaths              9289 non-null int64\n",
      "assists             9289 non-null int64\n",
      "leaver_status       9289 non-null float64\n",
      "gold                9289 non-null float64\n",
      "last_hits           9289 non-null int64\n",
      "denies              9289 non-null int64\n",
      "gold_per_min        9289 non-null int64\n",
      "xp_per_min          9289 non-null int64\n",
      "gold_spent          9289 non-null float64\n",
      "hero_damage         9289 non-null float64\n",
      "tower_damage        9289 non-null float64\n",
      "hero_healing        9289 non-null float64\n",
      "level               9289 non-null int64\n",
      "additional_units    68 non-null object\n",
      "stuns               6877 non-null float64\n",
      "max_hero_hit        9289 non-null object\n",
      "times               9289 non-null object\n",
      "gold_t              9289 non-null object\n",
      "lh_t                9289 non-null object\n",
      "xp_t                9289 non-null object\n",
      "obs_log             9289 non-null object\n",
      "sen_log             9289 non-null object\n",
      "purchase_log        9289 non-null object\n",
      "kills_log           9289 non-null object\n",
      "buyback_log         9289 non-null object\n",
      "lane_pos            9289 non-null object\n",
      "obs                 9289 non-null object\n",
      "sen                 9289 non-null object\n",
      "actions             9249 non-null object\n",
      "pings               9289 non-null object\n",
      "purchase            9289 non-null object\n",
      "gold_reasons        9289 non-null object\n",
      "xp_reasons          9289 non-null object\n",
      "killed              9289 non-null object\n",
      "item_uses           9289 non-null object\n",
      "ability_uses        9289 non-null object\n",
      "hero_hits           9289 non-null object\n",
      "damage              9289 non-null object\n",
      "damage_taken        9289 non-null object\n",
      "damage_inflictor    9289 non-null object\n",
      "runes               9289 non-null object\n",
      "killed_by           9289 non-null object\n",
      "kill_streaks        9289 non-null object\n",
      "multi_kills         9289 non-null object\n",
      "life_state          9249 non-null object\n",
      "dtypes: float64(7), int64(18), object(31)\n",
      "memory usage: 96.5 MB\n"
     ]
    }
   ],
   "source": [
    "skilled_player_stats.info(memory_usage=\"deep\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the filtered data is such a small size, I will retain it as a Pandas dataframe until I get to the model training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>match_id</th>\n",
       "      <th>skill</th>\n",
       "      <th>player_slot</th>\n",
       "      <th>hero_id</th>\n",
       "      <th>item_0</th>\n",
       "      <th>item_1</th>\n",
       "      <th>item_2</th>\n",
       "      <th>item_3</th>\n",
       "      <th>item_4</th>\n",
       "      <th>item_5</th>\n",
       "      <th>...</th>\n",
       "      <th>last_hits</th>\n",
       "      <th>denies</th>\n",
       "      <th>gold_per_min</th>\n",
       "      <th>xp_per_min</th>\n",
       "      <th>gold_spent</th>\n",
       "      <th>hero_damage</th>\n",
       "      <th>tower_damage</th>\n",
       "      <th>hero_healing</th>\n",
       "      <th>level</th>\n",
       "      <th>stuns</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>9.289000e+03</td>\n",
       "      <td>9289.0</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>9289.000000</td>\n",
       "      <td>6877.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>2.315296e+09</td>\n",
       "      <td>2.0</td>\n",
       "      <td>66.147056</td>\n",
       "      <td>52.104963</td>\n",
       "      <td>99.561847</td>\n",
       "      <td>102.909140</td>\n",
       "      <td>97.861018</td>\n",
       "      <td>93.113037</td>\n",
       "      <td>87.903972</td>\n",
       "      <td>77.570244</td>\n",
       "      <td>...</td>\n",
       "      <td>140.677037</td>\n",
       "      <td>5.338250</td>\n",
       "      <td>420.980299</td>\n",
       "      <td>446.880289</td>\n",
       "      <td>14689.853052</td>\n",
       "      <td>11878.722360</td>\n",
       "      <td>1337.827538</td>\n",
       "      <td>459.928625</td>\n",
       "      <td>18.154161</td>\n",
       "      <td>40.877252</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3.915754e+07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>64.016537</td>\n",
       "      <td>31.723346</td>\n",
       "      <td>73.206330</td>\n",
       "      <td>71.746296</td>\n",
       "      <td>71.560717</td>\n",
       "      <td>72.146970</td>\n",
       "      <td>72.549821</td>\n",
       "      <td>73.718667</td>\n",
       "      <td>...</td>\n",
       "      <td>109.909288</td>\n",
       "      <td>5.967534</td>\n",
       "      <td>141.416588</td>\n",
       "      <td>145.307008</td>\n",
       "      <td>7029.270891</td>\n",
       "      <td>6918.410586</td>\n",
       "      <td>1813.633859</td>\n",
       "      <td>1213.586262</td>\n",
       "      <td>4.593760</td>\n",
       "      <td>40.756453</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.662940e+09</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>106.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>75.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-2.893970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>2.317870e+09</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>46.000000</td>\n",
       "      <td>46.000000</td>\n",
       "      <td>41.000000</td>\n",
       "      <td>40.000000</td>\n",
       "      <td>34.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>57.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>317.000000</td>\n",
       "      <td>340.000000</td>\n",
       "      <td>9750.000000</td>\n",
       "      <td>6819.000000</td>\n",
       "      <td>114.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>15.000000</td>\n",
       "      <td>10.865600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>2.317901e+09</td>\n",
       "      <td>2.0</td>\n",
       "      <td>128.000000</td>\n",
       "      <td>52.000000</td>\n",
       "      <td>92.000000</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>92.000000</td>\n",
       "      <td>71.000000</td>\n",
       "      <td>63.000000</td>\n",
       "      <td>48.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>117.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>402.000000</td>\n",
       "      <td>444.000000</td>\n",
       "      <td>13600.000000</td>\n",
       "      <td>10474.000000</td>\n",
       "      <td>560.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>18.000000</td>\n",
       "      <td>30.712200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>2.317936e+09</td>\n",
       "      <td>2.0</td>\n",
       "      <td>130.000000</td>\n",
       "      <td>75.000000</td>\n",
       "      <td>164.000000</td>\n",
       "      <td>158.000000</td>\n",
       "      <td>152.000000</td>\n",
       "      <td>149.000000</td>\n",
       "      <td>143.000000</td>\n",
       "      <td>135.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>194.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>511.000000</td>\n",
       "      <td>547.000000</td>\n",
       "      <td>18555.000000</td>\n",
       "      <td>15558.000000</td>\n",
       "      <td>1841.000000</td>\n",
       "      <td>224.000000</td>\n",
       "      <td>22.000000</td>\n",
       "      <td>56.881600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>2.317991e+09</td>\n",
       "      <td>2.0</td>\n",
       "      <td>132.000000</td>\n",
       "      <td>113.000000</td>\n",
       "      <td>254.000000</td>\n",
       "      <td>254.000000</td>\n",
       "      <td>254.000000</td>\n",
       "      <td>254.000000</td>\n",
       "      <td>254.000000</td>\n",
       "      <td>254.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>1068.000000</td>\n",
       "      <td>81.000000</td>\n",
       "      <td>1154.000000</td>\n",
       "      <td>1006.000000</td>\n",
       "      <td>67260.000000</td>\n",
       "      <td>61295.000000</td>\n",
       "      <td>13747.000000</td>\n",
       "      <td>17123.000000</td>\n",
       "      <td>25.000000</td>\n",
       "      <td>422.876000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           match_id   skill  player_slot      hero_id       item_0  \\\n",
       "count  9.289000e+03  9289.0  9289.000000  9289.000000  9289.000000   \n",
       "mean   2.315296e+09     2.0    66.147056    52.104963    99.561847   \n",
       "std    3.915754e+07     0.0    64.016537    31.723346    73.206330   \n",
       "min    1.662940e+09     2.0     0.000000     1.000000     0.000000   \n",
       "25%    2.317870e+09     2.0     2.000000    23.000000    46.000000   \n",
       "50%    2.317901e+09     2.0   128.000000    52.000000    92.000000   \n",
       "75%    2.317936e+09     2.0   130.000000    75.000000   164.000000   \n",
       "max    2.317991e+09     2.0   132.000000   113.000000   254.000000   \n",
       "\n",
       "            item_1       item_2       item_3       item_4       item_5  ...  \\\n",
       "count  9289.000000  9289.000000  9289.000000  9289.000000  9289.000000  ...   \n",
       "mean    102.909140    97.861018    93.113037    87.903972    77.570244  ...   \n",
       "std      71.746296    71.560717    72.146970    72.549821    73.718667  ...   \n",
       "min       0.000000     0.000000     0.000000     0.000000     0.000000  ...   \n",
       "25%      46.000000    41.000000    40.000000    34.000000     1.000000  ...   \n",
       "50%     100.000000    92.000000    71.000000    63.000000    48.000000  ...   \n",
       "75%     158.000000   152.000000   149.000000   143.000000   135.000000  ...   \n",
       "max     254.000000   254.000000   254.000000   254.000000   254.000000  ...   \n",
       "\n",
       "         last_hits       denies  gold_per_min   xp_per_min    gold_spent  \\\n",
       "count  9289.000000  9289.000000   9289.000000  9289.000000   9289.000000   \n",
       "mean    140.677037     5.338250    420.980299   446.880289  14689.853052   \n",
       "std     109.909288     5.967534    141.416588   145.307008   7029.270891   \n",
       "min       0.000000     0.000000    106.000000     0.000000     75.000000   \n",
       "25%      57.000000     1.000000    317.000000   340.000000   9750.000000   \n",
       "50%     117.000000     3.000000    402.000000   444.000000  13600.000000   \n",
       "75%     194.000000     7.000000    511.000000   547.000000  18555.000000   \n",
       "max    1068.000000    81.000000   1154.000000  1006.000000  67260.000000   \n",
       "\n",
       "        hero_damage  tower_damage  hero_healing        level        stuns  \n",
       "count   9289.000000   9289.000000   9289.000000  9289.000000  6877.000000  \n",
       "mean   11878.722360   1337.827538    459.928625    18.154161    40.877252  \n",
       "std     6918.410586   1813.633859   1213.586262     4.593760    40.756453  \n",
       "min        0.000000      0.000000      0.000000     1.000000    -2.893970  \n",
       "25%     6819.000000    114.000000      0.000000    15.000000    10.865600  \n",
       "50%    10474.000000    560.000000      0.000000    18.000000    30.712200  \n",
       "75%    15558.000000   1841.000000    224.000000    22.000000    56.881600  \n",
       "max    61295.000000  13747.000000  17123.000000    25.000000   422.876000  \n",
       "\n",
       "[8 rows x 25 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skilled_player_stats.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset is comprised of more than half seemingly non-numerical features. Lets look at the last 32 columns to see what they contain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>additional_units</th>\n",
       "      <th>stuns</th>\n",
       "      <th>max_hero_hit</th>\n",
       "      <th>times</th>\n",
       "      <th>gold_t</th>\n",
       "      <th>lh_t</th>\n",
       "      <th>xp_t</th>\n",
       "      <th>obs_log</th>\n",
       "      <th>sen_log</th>\n",
       "      <th>purchase_log</th>\n",
       "      <th>kills_log</th>\n",
       "      <th>buyback_log</th>\n",
       "      <th>lane_pos</th>\n",
       "      <th>obs</th>\n",
       "      <th>sen</th>\n",
       "      <th>actions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>20.569000</td>\n",
       "      <td>{\"type\":\"max_hero_hit\",\"time\":1170,\"max\":true,...</td>\n",
       "      <td>{0,60,120,180,240,300,360,420,480,540,600,660,...</td>\n",
       "      <td>{0,100,282,381,948,1127,1227,1366,1466,1566,18...</td>\n",
       "      <td>{0,0,2,2,5,5,5,6,6,6,6,10,11,15,15,15,15,18,19...</td>\n",
       "      <td>{0,51,339,514,929,1102,1153,1359,1359,1503,153...</td>\n",
       "      <td>{\"{\\\"time\\\":-9,\\\"key\\\":[100,158]}\",\"{\\\"time\\\":...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"{\\\"time\\\":-81,\\\"key\\\":\\\"tango\\\"}\",\"{\\\"time\\\"...</td>\n",
       "      <td>{\"{\\\"time\\\":189,\\\"key\\\":\\\"npc_dota_hero_furion...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"70\":{\"164\":1,\"166\":1,\"168\":1,\"170\":5},\"72\":{...</td>\n",
       "      <td>{\"100\":{\"158\":1},\"140\":{\"106\":1},\"166\":{\"102\":1}}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"1\":3491,\"2\":90,\"4\":382,\"5\":11,\"6\":107,\"7\":5,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>38.754900</td>\n",
       "      <td>{\"type\":\"max_hero_hit\",\"time\":1053,\"max\":true,...</td>\n",
       "      <td>{0,60,120,180,240,300,360,420,480,540,600,660,...</td>\n",
       "      <td>{0,100,201,300,578,678,1266,1438,1538,1638,189...</td>\n",
       "      <td>{0,0,0,0,1,1,1,4,4,4,4,4,5,5,5,5,8,8,8,15,20,2...</td>\n",
       "      <td>{0,56,231,313,712,774,1216,1339,1640,1887,2083...</td>\n",
       "      <td>{\"{\\\"time\\\":143,\\\"key\\\":[118,138]}\",\"{\\\"time\\\"...</td>\n",
       "      <td>{\"{\\\"time\\\":201,\\\"key\\\":[154,88]}\",\"{\\\"time\\\":...</td>\n",
       "      <td>{\"{\\\"time\\\":-41,\\\"key\\\":\\\"ward_observer\\\"}\",\"{...</td>\n",
       "      <td>{\"{\\\"time\\\":305,\\\"key\\\":\\\"npc_dota_hero_slark\\...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"70\":{\"74\":1,\"76\":1,\"78\":8},\"74\":{\"78\":1,\"158...</td>\n",
       "      <td>{\"94\":{\"160\":1},\"100\":{\"128\":1},\"112\":{\"146\":1...</td>\n",
       "      <td>{\"90\":{\"160\":1},\"94\":{\"160\":1},\"110\":{\"146\":1}...</td>\n",
       "      <td>{\"1\":1461,\"2\":11,\"4\":107,\"5\":61,\"6\":116,\"7\":16...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>85.917400</td>\n",
       "      <td>{\"type\":\"max_hero_hit\",\"time\":1589,\"max\":true,...</td>\n",
       "      <td>{0,60,120,180,240,300,360,420,480,540,600,660,...</td>\n",
       "      <td>{0,280,460,765,1231,1808,2198,2457,2557,3047,4...</td>\n",
       "      <td>{0,2,4,9,17,24,31,35,35,44,53,55,65,67,72,77,8...</td>\n",
       "      <td>{0,317,760,1131,1673,2173,2637,2864,2864,3674,...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"{\\\"time\\\":-65,\\\"key\\\":\\\"circlet\\\"}\",\"{\\\"time...</td>\n",
       "      <td>{\"{\\\"time\\\":293,\\\"key\\\":\\\"npc_dota_hero_furion...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"72\":{\"76\":50,\"78\":1},\"74\":{\"78\":32,\"80\":1,\"8...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"1\":3203,\"2\":89,\"3\":83,\"4\":714,\"5\":68,\"6\":46,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>13.024900</td>\n",
       "      <td>{\"type\":\"max_hero_hit\",\"time\":1951,\"max\":true,...</td>\n",
       "      <td>{0,60,120,180,240,300,360,420,480,540,600,660,...</td>\n",
       "      <td>{0,146,413,553,1022,1314,1882,2146,2494,2680,3...</td>\n",
       "      <td>{0,1,5,6,11,14,19,23,29,31,34,43,48,49,55,62,6...</td>\n",
       "      <td>{0,51,339,463,878,1175,1717,1923,2506,2650,279...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"{\\\"time\\\":-80,\\\"key\\\":\\\"quelling_blade\\\"}\",\"...</td>\n",
       "      <td>{\"{\\\"time\\\":306,\\\"key\\\":\\\"npc_dota_hero_omnikn...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"70\":{\"74\":10,\"76\":8},\"72\":{\"74\":2,\"76\":4,\"16...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"1\":4110,\"2\":249,\"4\":1903,\"5\":103,\"6\":23,\"7\":...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.033418</td>\n",
       "      <td>{\"type\":\"max_hero_hit\",\"time\":2130,\"max\":true,...</td>\n",
       "      <td>{0,60,120,180,240,300,360,420,480,540,600,660,...</td>\n",
       "      <td>{0,138,363,631,1113,1297,1605,1995,2262,2528,3...</td>\n",
       "      <td>{0,1,4,8,11,13,18,25,29,33,42,48,54,56,65,77,8...</td>\n",
       "      <td>{0,87,262,716,1185,1505,1979,2433,2672,2878,39...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"{\\\"time\\\":-85,\\\"key\\\":\\\"tango\\\"}\",\"{\\\"time\\\"...</td>\n",
       "      <td>{\"{\\\"time\\\":227,\\\"key\\\":\\\"npc_dota_hero_windru...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"70\":{\"76\":8},\"72\":{\"76\":1,\"78\":7},\"74\":{\"76\"...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"1\":4306,\"2\":96,\"3\":1,\"4\":625,\"5\":19,\"6\":20,\"...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>125</th>\n",
       "      <td>NaN</td>\n",
       "      <td>26.494800</td>\n",
       "      <td>{\"type\":\"max_hero_hit\",\"time\":1100,\"max\":true,...</td>\n",
       "      <td>{0,60,120,180,240,300,360,420,480,540,600,660,...</td>\n",
       "      <td>{0,181,405,620,853,1023,1166,1634,1773,2056,23...</td>\n",
       "      <td>{0,2,5,8,11,12,13,14,15,18,21,21,22,22,23,28,3...</td>\n",
       "      <td>{0,144,319,742,1030,1195,1494,1894,2089,2406,2...</td>\n",
       "      <td>{\"{\\\"time\\\":1834,\\\"key\\\":[176,142]}\",\"{\\\"time\\...</td>\n",
       "      <td>{\"{\\\"time\\\":2086,\\\"key\\\":[176,92]}\"}</td>\n",
       "      <td>{\"{\\\"time\\\":-81,\\\"key\\\":\\\"ring_of_protection\\\"...</td>\n",
       "      <td>{\"{\\\"time\\\":400,\\\"key\\\":\\\"npc_dota_hero_phanto...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"148\":{\"118\":1,\"120\":2},\"150\":{\"108\":2,\"110\":...</td>\n",
       "      <td>{\"90\":{\"174\":1},\"144\":{\"168\":1},\"176\":{\"142\":1}}</td>\n",
       "      <td>{\"176\":{\"92\":1}}</td>\n",
       "      <td>{\"1\":4148,\"4\":538,\"5\":37,\"6\":49,\"7\":3,\"8\":64,\"...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>126</th>\n",
       "      <td>NaN</td>\n",
       "      <td>57.974100</td>\n",
       "      <td>{\"type\":\"max_hero_hit\",\"time\":2051,\"max\":true,...</td>\n",
       "      <td>{0,60,120,180,240,300,360,420,480,540,600,660,...</td>\n",
       "      <td>{0,100,200,300,400,500,600,868,968,1068,1168,1...</td>\n",
       "      <td>{0,0,0,0,0,0,0,0,0,0,0,1,1,2,2,2,4,6,8,8,11,11...</td>\n",
       "      <td>{0,82,257,339,390,493,565,843,1038,1100,1277,1...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"{\\\"time\\\":-74,\\\"key\\\":\\\"orb_of_venom\\\"}\",\"{\\...</td>\n",
       "      <td>{\"{\\\"time\\\":1101,\\\"key\\\":\\\"npc_dota_hero_legio...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"68\":{\"160\":1},\"70\":{\"156\":2,\"158\":1,\"160\":2,...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"1\":1363,\"2\":1,\"3\":30,\"4\":310,\"5\":6,\"6\":44,\"7...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>NaN</td>\n",
       "      <td>15.179000</td>\n",
       "      <td>{\"type\":\"max_hero_hit\",\"time\":2180,\"max\":true,...</td>\n",
       "      <td>{0,60,120,180,240,300,360,420,480,540,600,660,...</td>\n",
       "      <td>{0,100,200,300,400,500,600,700,800,942,1042,13...</td>\n",
       "      <td>{0,0,0,0,0,0,0,0,0,1,1,4,7,8,8,8,8,8,8,8,8,11,...</td>\n",
       "      <td>{0,0,164,246,246,374,617,750,1000,1144,1177,15...</td>\n",
       "      <td>{\"{\\\"time\\\":115,\\\"key\\\":[80,150]}\",\"{\\\"time\\\":...</td>\n",
       "      <td>{\"{\\\"time\\\":1273,\\\"key\\\":[148,114]}\",\"{\\\"time\\...</td>\n",
       "      <td>{\"{\\\"time\\\":-81,\\\"key\\\":\\\"courier\\\"}\",\"{\\\"time...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"70\":{\"164\":2,\"166\":2},\"72\":{\"164\":2,\"168\":2,...</td>\n",
       "      <td>{\"80\":{\"150\":1},\"100\":{\"142\":1},\"110\":{\"148\":2...</td>\n",
       "      <td>{\"88\":{\"160\":1},\"122\":{\"136\":1},\"128\":{\"144\":1...</td>\n",
       "      <td>{\"1\":3392,\"4\":389,\"5\":31,\"6\":76,\"7\":2,\"8\":22,\"...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{\"type\":\"max_hero_hit\",\"time\":2190,\"max\":true,...</td>\n",
       "      <td>{0,60,120,180,240,300,360,420,480,540,600,660,...</td>\n",
       "      <td>{0,182,319,500,764,1028,1408,1623,1956,2141,25...</td>\n",
       "      <td>{0,2,3,5,9,13,19,22,27,29,36,38,44,52,60,69,75...</td>\n",
       "      <td>{0,196,515,783,1278,1618,1861,2180,2430,2677,2...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"{\\\"time\\\":-70,\\\"key\\\":\\\"stout_shield\\\"}\",\"{\\...</td>\n",
       "      <td>{\"{\\\"time\\\":2061,\\\"key\\\":\\\"npc_dota_hero_trean...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"72\":{\"160\":2,\"162\":6},\"74\":{\"158\":2,\"160\":2,...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"1\":3875,\"3\":4,\"4\":1074,\"5\":6,\"6\":2,\"7\":4,\"8\"...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{\"type\":\"max_hero_hit\",\"time\":2057,\"max\":true,...</td>\n",
       "      <td>{0,60,120,180,240,300,360,420,480,540,600,660,...</td>\n",
       "      <td>{0,319,1180,1581,1883,2233,2667,3173,3886,4101...</td>\n",
       "      <td>{0,3,10,16,20,26,34,46,58,60,69,70,80,98,109,1...</td>\n",
       "      <td>{0,348,1043,1506,1986,2378,2832,3647,4675,4907...</td>\n",
       "      <td>{\"{\\\"time\\\":-42,\\\"key\\\":[124,130]}\"}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"{\\\"time\\\":-85,\\\"key\\\":\\\"circlet\\\"}\",\"{\\\"time...</td>\n",
       "      <td>{\"{\\\"time\\\":95,\\\"key\\\":\\\"npc_dota_hero_invoker...</td>\n",
       "      <td>{\"{\\\"time\\\":2228}\"}</td>\n",
       "      <td>{\"80\":{\"156\":1,\"158\":1,\"162\":1,\"164\":1},\"82\":{...</td>\n",
       "      <td>{\"124\":{\"130\":1}}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"1\":3255,\"4\":619,\"5\":271,\"6\":74,\"7\":2,\"8\":583...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>9289 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    additional_units      stuns  \\\n",
       "0                NaN  20.569000   \n",
       "1                NaN  38.754900   \n",
       "2                NaN  85.917400   \n",
       "3                NaN  13.024900   \n",
       "4                NaN   0.033418   \n",
       "..               ...        ...   \n",
       "125              NaN  26.494800   \n",
       "126              NaN  57.974100   \n",
       "127              NaN  15.179000   \n",
       "128              NaN        NaN   \n",
       "129              NaN        NaN   \n",
       "\n",
       "                                          max_hero_hit  \\\n",
       "0    {\"type\":\"max_hero_hit\",\"time\":1170,\"max\":true,...   \n",
       "1    {\"type\":\"max_hero_hit\",\"time\":1053,\"max\":true,...   \n",
       "2    {\"type\":\"max_hero_hit\",\"time\":1589,\"max\":true,...   \n",
       "3    {\"type\":\"max_hero_hit\",\"time\":1951,\"max\":true,...   \n",
       "4    {\"type\":\"max_hero_hit\",\"time\":2130,\"max\":true,...   \n",
       "..                                                 ...   \n",
       "125  {\"type\":\"max_hero_hit\",\"time\":1100,\"max\":true,...   \n",
       "126  {\"type\":\"max_hero_hit\",\"time\":2051,\"max\":true,...   \n",
       "127  {\"type\":\"max_hero_hit\",\"time\":2180,\"max\":true,...   \n",
       "128  {\"type\":\"max_hero_hit\",\"time\":2190,\"max\":true,...   \n",
       "129  {\"type\":\"max_hero_hit\",\"time\":2057,\"max\":true,...   \n",
       "\n",
       "                                                 times  \\\n",
       "0    {0,60,120,180,240,300,360,420,480,540,600,660,...   \n",
       "1    {0,60,120,180,240,300,360,420,480,540,600,660,...   \n",
       "2    {0,60,120,180,240,300,360,420,480,540,600,660,...   \n",
       "3    {0,60,120,180,240,300,360,420,480,540,600,660,...   \n",
       "4    {0,60,120,180,240,300,360,420,480,540,600,660,...   \n",
       "..                                                 ...   \n",
       "125  {0,60,120,180,240,300,360,420,480,540,600,660,...   \n",
       "126  {0,60,120,180,240,300,360,420,480,540,600,660,...   \n",
       "127  {0,60,120,180,240,300,360,420,480,540,600,660,...   \n",
       "128  {0,60,120,180,240,300,360,420,480,540,600,660,...   \n",
       "129  {0,60,120,180,240,300,360,420,480,540,600,660,...   \n",
       "\n",
       "                                                gold_t  \\\n",
       "0    {0,100,282,381,948,1127,1227,1366,1466,1566,18...   \n",
       "1    {0,100,201,300,578,678,1266,1438,1538,1638,189...   \n",
       "2    {0,280,460,765,1231,1808,2198,2457,2557,3047,4...   \n",
       "3    {0,146,413,553,1022,1314,1882,2146,2494,2680,3...   \n",
       "4    {0,138,363,631,1113,1297,1605,1995,2262,2528,3...   \n",
       "..                                                 ...   \n",
       "125  {0,181,405,620,853,1023,1166,1634,1773,2056,23...   \n",
       "126  {0,100,200,300,400,500,600,868,968,1068,1168,1...   \n",
       "127  {0,100,200,300,400,500,600,700,800,942,1042,13...   \n",
       "128  {0,182,319,500,764,1028,1408,1623,1956,2141,25...   \n",
       "129  {0,319,1180,1581,1883,2233,2667,3173,3886,4101...   \n",
       "\n",
       "                                                  lh_t  \\\n",
       "0    {0,0,2,2,5,5,5,6,6,6,6,10,11,15,15,15,15,18,19...   \n",
       "1    {0,0,0,0,1,1,1,4,4,4,4,4,5,5,5,5,8,8,8,15,20,2...   \n",
       "2    {0,2,4,9,17,24,31,35,35,44,53,55,65,67,72,77,8...   \n",
       "3    {0,1,5,6,11,14,19,23,29,31,34,43,48,49,55,62,6...   \n",
       "4    {0,1,4,8,11,13,18,25,29,33,42,48,54,56,65,77,8...   \n",
       "..                                                 ...   \n",
       "125  {0,2,5,8,11,12,13,14,15,18,21,21,22,22,23,28,3...   \n",
       "126  {0,0,0,0,0,0,0,0,0,0,0,1,1,2,2,2,4,6,8,8,11,11...   \n",
       "127  {0,0,0,0,0,0,0,0,0,1,1,4,7,8,8,8,8,8,8,8,8,11,...   \n",
       "128  {0,2,3,5,9,13,19,22,27,29,36,38,44,52,60,69,75...   \n",
       "129  {0,3,10,16,20,26,34,46,58,60,69,70,80,98,109,1...   \n",
       "\n",
       "                                                  xp_t  \\\n",
       "0    {0,51,339,514,929,1102,1153,1359,1359,1503,153...   \n",
       "1    {0,56,231,313,712,774,1216,1339,1640,1887,2083...   \n",
       "2    {0,317,760,1131,1673,2173,2637,2864,2864,3674,...   \n",
       "3    {0,51,339,463,878,1175,1717,1923,2506,2650,279...   \n",
       "4    {0,87,262,716,1185,1505,1979,2433,2672,2878,39...   \n",
       "..                                                 ...   \n",
       "125  {0,144,319,742,1030,1195,1494,1894,2089,2406,2...   \n",
       "126  {0,82,257,339,390,493,565,843,1038,1100,1277,1...   \n",
       "127  {0,0,164,246,246,374,617,750,1000,1144,1177,15...   \n",
       "128  {0,196,515,783,1278,1618,1861,2180,2430,2677,2...   \n",
       "129  {0,348,1043,1506,1986,2378,2832,3647,4675,4907...   \n",
       "\n",
       "                                               obs_log  \\\n",
       "0    {\"{\\\"time\\\":-9,\\\"key\\\":[100,158]}\",\"{\\\"time\\\":...   \n",
       "1    {\"{\\\"time\\\":143,\\\"key\\\":[118,138]}\",\"{\\\"time\\\"...   \n",
       "2                                                   {}   \n",
       "3                                                   {}   \n",
       "4                                                   {}   \n",
       "..                                                 ...   \n",
       "125  {\"{\\\"time\\\":1834,\\\"key\\\":[176,142]}\",\"{\\\"time\\...   \n",
       "126                                                 {}   \n",
       "127  {\"{\\\"time\\\":115,\\\"key\\\":[80,150]}\",\"{\\\"time\\\":...   \n",
       "128                                                 {}   \n",
       "129               {\"{\\\"time\\\":-42,\\\"key\\\":[124,130]}\"}   \n",
       "\n",
       "                                               sen_log  \\\n",
       "0                                                   {}   \n",
       "1    {\"{\\\"time\\\":201,\\\"key\\\":[154,88]}\",\"{\\\"time\\\":...   \n",
       "2                                                   {}   \n",
       "3                                                   {}   \n",
       "4                                                   {}   \n",
       "..                                                 ...   \n",
       "125               {\"{\\\"time\\\":2086,\\\"key\\\":[176,92]}\"}   \n",
       "126                                                 {}   \n",
       "127  {\"{\\\"time\\\":1273,\\\"key\\\":[148,114]}\",\"{\\\"time\\...   \n",
       "128                                                 {}   \n",
       "129                                                 {}   \n",
       "\n",
       "                                          purchase_log  \\\n",
       "0    {\"{\\\"time\\\":-81,\\\"key\\\":\\\"tango\\\"}\",\"{\\\"time\\\"...   \n",
       "1    {\"{\\\"time\\\":-41,\\\"key\\\":\\\"ward_observer\\\"}\",\"{...   \n",
       "2    {\"{\\\"time\\\":-65,\\\"key\\\":\\\"circlet\\\"}\",\"{\\\"time...   \n",
       "3    {\"{\\\"time\\\":-80,\\\"key\\\":\\\"quelling_blade\\\"}\",\"...   \n",
       "4    {\"{\\\"time\\\":-85,\\\"key\\\":\\\"tango\\\"}\",\"{\\\"time\\\"...   \n",
       "..                                                 ...   \n",
       "125  {\"{\\\"time\\\":-81,\\\"key\\\":\\\"ring_of_protection\\\"...   \n",
       "126  {\"{\\\"time\\\":-74,\\\"key\\\":\\\"orb_of_venom\\\"}\",\"{\\...   \n",
       "127  {\"{\\\"time\\\":-81,\\\"key\\\":\\\"courier\\\"}\",\"{\\\"time...   \n",
       "128  {\"{\\\"time\\\":-70,\\\"key\\\":\\\"stout_shield\\\"}\",\"{\\...   \n",
       "129  {\"{\\\"time\\\":-85,\\\"key\\\":\\\"circlet\\\"}\",\"{\\\"time...   \n",
       "\n",
       "                                             kills_log          buyback_log  \\\n",
       "0    {\"{\\\"time\\\":189,\\\"key\\\":\\\"npc_dota_hero_furion...                   {}   \n",
       "1    {\"{\\\"time\\\":305,\\\"key\\\":\\\"npc_dota_hero_slark\\...                   {}   \n",
       "2    {\"{\\\"time\\\":293,\\\"key\\\":\\\"npc_dota_hero_furion...                   {}   \n",
       "3    {\"{\\\"time\\\":306,\\\"key\\\":\\\"npc_dota_hero_omnikn...                   {}   \n",
       "4    {\"{\\\"time\\\":227,\\\"key\\\":\\\"npc_dota_hero_windru...                   {}   \n",
       "..                                                 ...                  ...   \n",
       "125  {\"{\\\"time\\\":400,\\\"key\\\":\\\"npc_dota_hero_phanto...                   {}   \n",
       "126  {\"{\\\"time\\\":1101,\\\"key\\\":\\\"npc_dota_hero_legio...                   {}   \n",
       "127                                                 {}                   {}   \n",
       "128  {\"{\\\"time\\\":2061,\\\"key\\\":\\\"npc_dota_hero_trean...                   {}   \n",
       "129  {\"{\\\"time\\\":95,\\\"key\\\":\\\"npc_dota_hero_invoker...  {\"{\\\"time\\\":2228}\"}   \n",
       "\n",
       "                                              lane_pos  \\\n",
       "0    {\"70\":{\"164\":1,\"166\":1,\"168\":1,\"170\":5},\"72\":{...   \n",
       "1    {\"70\":{\"74\":1,\"76\":1,\"78\":8},\"74\":{\"78\":1,\"158...   \n",
       "2    {\"72\":{\"76\":50,\"78\":1},\"74\":{\"78\":32,\"80\":1,\"8...   \n",
       "3    {\"70\":{\"74\":10,\"76\":8},\"72\":{\"74\":2,\"76\":4,\"16...   \n",
       "4    {\"70\":{\"76\":8},\"72\":{\"76\":1,\"78\":7},\"74\":{\"76\"...   \n",
       "..                                                 ...   \n",
       "125  {\"148\":{\"118\":1,\"120\":2},\"150\":{\"108\":2,\"110\":...   \n",
       "126  {\"68\":{\"160\":1},\"70\":{\"156\":2,\"158\":1,\"160\":2,...   \n",
       "127  {\"70\":{\"164\":2,\"166\":2},\"72\":{\"164\":2,\"168\":2,...   \n",
       "128  {\"72\":{\"160\":2,\"162\":6},\"74\":{\"158\":2,\"160\":2,...   \n",
       "129  {\"80\":{\"156\":1,\"158\":1,\"162\":1,\"164\":1},\"82\":{...   \n",
       "\n",
       "                                                   obs  \\\n",
       "0    {\"100\":{\"158\":1},\"140\":{\"106\":1},\"166\":{\"102\":1}}   \n",
       "1    {\"94\":{\"160\":1},\"100\":{\"128\":1},\"112\":{\"146\":1...   \n",
       "2                                                   {}   \n",
       "3                                                   {}   \n",
       "4                                                   {}   \n",
       "..                                                 ...   \n",
       "125   {\"90\":{\"174\":1},\"144\":{\"168\":1},\"176\":{\"142\":1}}   \n",
       "126                                                 {}   \n",
       "127  {\"80\":{\"150\":1},\"100\":{\"142\":1},\"110\":{\"148\":2...   \n",
       "128                                                 {}   \n",
       "129                                  {\"124\":{\"130\":1}}   \n",
       "\n",
       "                                                   sen  \\\n",
       "0                                                   {}   \n",
       "1    {\"90\":{\"160\":1},\"94\":{\"160\":1},\"110\":{\"146\":1}...   \n",
       "2                                                   {}   \n",
       "3                                                   {}   \n",
       "4                                                   {}   \n",
       "..                                                 ...   \n",
       "125                                   {\"176\":{\"92\":1}}   \n",
       "126                                                 {}   \n",
       "127  {\"88\":{\"160\":1},\"122\":{\"136\":1},\"128\":{\"144\":1...   \n",
       "128                                                 {}   \n",
       "129                                                 {}   \n",
       "\n",
       "                                               actions  \n",
       "0    {\"1\":3491,\"2\":90,\"4\":382,\"5\":11,\"6\":107,\"7\":5,...  \n",
       "1    {\"1\":1461,\"2\":11,\"4\":107,\"5\":61,\"6\":116,\"7\":16...  \n",
       "2    {\"1\":3203,\"2\":89,\"3\":83,\"4\":714,\"5\":68,\"6\":46,...  \n",
       "3    {\"1\":4110,\"2\":249,\"4\":1903,\"5\":103,\"6\":23,\"7\":...  \n",
       "4    {\"1\":4306,\"2\":96,\"3\":1,\"4\":625,\"5\":19,\"6\":20,\"...  \n",
       "..                                                 ...  \n",
       "125  {\"1\":4148,\"4\":538,\"5\":37,\"6\":49,\"7\":3,\"8\":64,\"...  \n",
       "126  {\"1\":1363,\"2\":1,\"3\":30,\"4\":310,\"5\":6,\"6\":44,\"7...  \n",
       "127  {\"1\":3392,\"4\":389,\"5\":31,\"6\":76,\"7\":2,\"8\":22,\"...  \n",
       "128  {\"1\":3875,\"3\":4,\"4\":1074,\"5\":6,\"6\":2,\"7\":4,\"8\"...  \n",
       "129  {\"1\":3255,\"4\":619,\"5\":271,\"6\":74,\"7\":2,\"8\":583...  \n",
       "\n",
       "[9289 rows x 16 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skilled_player_stats[skilled_player_stats.columns[24:40]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pings</th>\n",
       "      <th>purchase</th>\n",
       "      <th>gold_reasons</th>\n",
       "      <th>xp_reasons</th>\n",
       "      <th>killed</th>\n",
       "      <th>item_uses</th>\n",
       "      <th>ability_uses</th>\n",
       "      <th>hero_hits</th>\n",
       "      <th>damage</th>\n",
       "      <th>damage_taken</th>\n",
       "      <th>damage_inflictor</th>\n",
       "      <th>runes</th>\n",
       "      <th>killed_by</th>\n",
       "      <th>kill_streaks</th>\n",
       "      <th>multi_kills</th>\n",
       "      <th>life_state</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>{\"0\":3}</td>\n",
       "      <td>{\"tango\":1,\"branches\":3,\"enchanted_mango\":1,\"w...</td>\n",
       "      <td>{\"0\":897,\"1\":-1075,\"6\":137,\"11\":4181,\"12\":2748...</td>\n",
       "      <td>{\"0\":526,\"1\":4800,\"2\":7057}</td>\n",
       "      <td>{\"npc_dota_creep_badguys_melee\":35,\"npc_dota_h...</td>\n",
       "      <td>{\"tango\":4,\"ward_observer\":3,\"enchanted_mango\"...</td>\n",
       "      <td>{\"ogre_magi_ignite\":40,\"ogre_magi_fireblast\":1...</td>\n",
       "      <td>{\"ogre_magi_ignite\":148,\"undefined\":44,\"ogre_m...</td>\n",
       "      <td>{\"npc_dota_hero_slark\":4519,\"npc_dota_creep_ba...</td>\n",
       "      <td>{\"npc_dota_creep_badguys_melee\":724,\"npc_dota_...</td>\n",
       "      <td>{\"ogre_magi_ignite\":4744,\"undefined\":2123,\"ogr...</td>\n",
       "      <td>{\"3\":1,\"5\":3}</td>\n",
       "      <td>{\"npc_dota_hero_omniknight\":1,\"npc_dota_hero_z...</td>\n",
       "      <td>{\"3\":1}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"0\":2163,\"1\":10,\"2\":148}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>{\"0\":38}</td>\n",
       "      <td>{\"ward_observer\":10,\"courier\":1,\"clarity\":4,\"t...</td>\n",
       "      <td>{\"0\":816,\"11\":3860,\"12\":2525,\"13\":1520,\"14\":200}</td>\n",
       "      <td>{\"0\":318,\"1\":6018,\"2\":7607}</td>\n",
       "      <td>{\"npc_dota_creep_goodguys_melee\":2,\"npc_dota_c...</td>\n",
       "      <td>{\"courier\":1,\"ward_observer\":6,\"ward_sentry\":5...</td>\n",
       "      <td>{\"treant_living_armor\":54,\"treant_natures_guis...</td>\n",
       "      <td>{\"treant_leech_seed\":61,\"undefined\":8}</td>\n",
       "      <td>{\"npc_dota_creep_goodguys_melee\":382,\"npc_dota...</td>\n",
       "      <td>{\"npc_dota_hero_windrunner\":382,\"npc_dota_cree...</td>\n",
       "      <td>{\"treant_leech_seed\":1580,\"undefined\":468}</td>\n",
       "      <td>{\"2\":1,\"3\":1,\"4\":1,\"5\":1,\"6\":2}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"0\":2321}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>{\"0\":45}</td>\n",
       "      <td>{\"circlet\":1,\"mantle\":1,\"null_talisman\":1,\"rec...</td>\n",
       "      <td>{\"0\":969,\"1\":-1317,\"6\":330,\"11\":5120,\"12\":7047...</td>\n",
       "      <td>{\"0\":506,\"1\":10992,\"2\":13979,\"3\":894}</td>\n",
       "      <td>{\"npc_dota_creep_badguys_melee\":117,\"npc_dota_...</td>\n",
       "      <td>{\"flask\":1,\"tango_single\":1,\"bottle\":43,\"tpscr...</td>\n",
       "      <td>{\"invoker_exort\":560,\"invoker_invoke\":85,\"invo...</td>\n",
       "      <td>{\"undefined\":95,\"invoker_cold_snap\":36,\"invoke...</td>\n",
       "      <td>{\"npc_dota_creep_goodguys_melee\":336,\"npc_dota...</td>\n",
       "      <td>{\"npc_dota_hero_zuus\":8990,\"npc_dota_creep_bad...</td>\n",
       "      <td>{\"undefined\":8374,\"invoker_cold_snap\":1012,\"in...</td>\n",
       "      <td>{\"4\":2,\"5\":4}</td>\n",
       "      <td>{\"npc_dota_hero_furion\":1,\"npc_dota_hero_slark...</td>\n",
       "      <td>{\"3\":2,\"4\":2,\"5\":2,\"6\":2,\"7\":2,\"8\":2,\"9\":1}</td>\n",
       "      <td>{\"2\":1}</td>\n",
       "      <td>{\"0\":2142,\"1\":10,\"2\":169}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>{\"0\":7}</td>\n",
       "      <td>{\"quelling_blade\":1,\"ring_of_protection\":1,\"ta...</td>\n",
       "      <td>{\"0\":728,\"1\":-956,\"11\":4747,\"12\":2783,\"13\":631...</td>\n",
       "      <td>{\"0\":196,\"1\":6218,\"2\":11610}</td>\n",
       "      <td>{\"npc_dota_creep_badguys_ranged\":15,\"npc_dota_...</td>\n",
       "      <td>{\"tango\":12,\"faerie_fire\":2,\"tpscroll\":8,\"quel...</td>\n",
       "      <td>{\"beastmaster_call_of_the_wild\":23,\"beastmaste...</td>\n",
       "      <td>{\"undefined\":172,\"beastmaster_wild_axes\":21,\"b...</td>\n",
       "      <td>{\"npc_dota_hero_omniknight\":1380,\"npc_dota_her...</td>\n",
       "      <td>{\"npc_dota_hero_omniknight\":1052,\"npc_dota_her...</td>\n",
       "      <td>{\"undefined\":3530,\"beastmaster_wild_axes\":1930...</td>\n",
       "      <td>{\"5\":1}</td>\n",
       "      <td>{\"npc_dota_hero_omniknight\":1,\"npc_dota_hero_s...</td>\n",
       "      <td>{\"3\":1}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"0\":2183,\"1\":12,\"2\":126}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>{\"0\":4}</td>\n",
       "      <td>{\"tango\":2,\"slippers\":1,\"clarity\":2,\"flask\":1,...</td>\n",
       "      <td>{\"0\":710,\"1\":-269,\"6\":1000,\"11\":5084,\"12\":5063...</td>\n",
       "      <td>{\"0\":156,\"1\":9888,\"2\":18448,\"3\":894}</td>\n",
       "      <td>{\"npc_dota_creep_badguys_melee\":114,\"npc_dota_...</td>\n",
       "      <td>{\"tango\":8,\"flask\":1,\"clarity\":2,\"phase_boots\"...</td>\n",
       "      <td>{\"juggernaut_blade_fury\":17,\"juggernaut_healin...</td>\n",
       "      <td>{\"juggernaut_blade_fury\":166,\"undefined\":46,\"j...</td>\n",
       "      <td>{\"npc_dota_creep_badguys_melee\":63382,\"npc_dot...</td>\n",
       "      <td>{\"npc_dota_creep_badguys_melee\":1205,\"npc_dota...</td>\n",
       "      <td>{\"juggernaut_blade_fury\":3586,\"undefined\":9240...</td>\n",
       "      <td>{\"1\":2,\"5\":1,\"6\":1}</td>\n",
       "      <td>{\"npc_dota_hero_zuus\":1}</td>\n",
       "      <td>{\"3\":1,\"4\":1,\"5\":1,\"6\":1,\"7\":1,\"8\":1,\"9\":1}</td>\n",
       "      <td>{\"2\":1}</td>\n",
       "      <td>{\"0\":2282,\"1\":3,\"2\":36}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>125</th>\n",
       "      <td>{\"0\":41}</td>\n",
       "      <td>{\"ring_of_protection\":1,\"stout_shield\":1,\"tang...</td>\n",
       "      <td>{\"0\":943,\"1\":-1854,\"6\":100,\"11\":240,\"12\":1430,...</td>\n",
       "      <td>{\"0\":498,\"1\":2761,\"2\":9659}</td>\n",
       "      <td>{\"npc_dota_creep_goodguys_melee\":85,\"npc_dota_...</td>\n",
       "      <td>{\"tango\":4,\"faerie_fire\":1,\"branches\":1,\"magic...</td>\n",
       "      <td>{\"centaur_hoof_stomp\":25,\"centaur_double_edge\"...</td>\n",
       "      <td>{\"centaur_hoof_stomp\":11,\"undefined\":15,\"centa...</td>\n",
       "      <td>{\"npc_dota_creep_goodguys_melee\":44278,\"npc_do...</td>\n",
       "      <td>{\"npc_dota_hero_lich\":2366,\"npc_dota_hero_phan...</td>\n",
       "      <td>{\"centaur_hoof_stomp\":1549,\"undefined\":842,\"ce...</td>\n",
       "      <td>{\"5\":4}</td>\n",
       "      <td>{\"npc_dota_hero_lich\":2,\"npc_dota_hero_phantom...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"0\":2264,\"1\":20,\"2\":240}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>126</th>\n",
       "      <td>{\"0\":24}</td>\n",
       "      <td>{\"orb_of_venom\":1,\"stout_shield\":1,\"tango\":1,\"...</td>\n",
       "      <td>{\"0\":781,\"1\":-3019,\"6\":1887,\"11\":483,\"12\":1465...</td>\n",
       "      <td>{\"0\":245,\"1\":1803,\"2\":9000}</td>\n",
       "      <td>{\"npc_dota_creep_goodguys_melee\":26,\"npc_dota_...</td>\n",
       "      <td>{\"tango\":4,\"magic_stick\":14,\"tango_single\":1,\"...</td>\n",
       "      <td>{\"spirit_breaker_charge_of_darkness\":24,\"spiri...</td>\n",
       "      <td>{\"undefined\":22,\"orb_of_venom\":51,\"spirit_brea...</td>\n",
       "      <td>{\"npc_dota_hero_phantom_assassin\":1355,\"npc_do...</td>\n",
       "      <td>{\"npc_dota_hero_phantom_assassin\":4834,\"npc_do...</td>\n",
       "      <td>{\"undefined\":1137,\"orb_of_venom\":102,\"spirit_b...</td>\n",
       "      <td>{\"3\":2,\"5\":2}</td>\n",
       "      <td>{\"npc_dota_hero_phantom_assassin\":2,\"npc_dota_...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"0\":2094,\"1\":31,\"2\":399}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>{\"0\":14}</td>\n",
       "      <td>{\"courier\":1,\"ward_observer\":14,\"tango\":1,\"fla...</td>\n",
       "      <td>{\"0\":745,\"1\":-1763,\"11\":240,\"12\":471,\"13\":897}</td>\n",
       "      <td>{\"0\":266,\"1\":1141,\"2\":5482}</td>\n",
       "      <td>{\"npc_dota_creep_badguys_ranged\":3,\"npc_dota_c...</td>\n",
       "      <td>{\"courier\":1,\"ward_observer\":8,\"tango\":4,\"bran...</td>\n",
       "      <td>{\"rubick_telekinesis\":8,\"rubick_telekinesis_la...</td>\n",
       "      <td>{\"undefined\":26,\"rubick_fade_bolt\":5}</td>\n",
       "      <td>{\"npc_dota_hero_treant\":571,\"npc_dota_hero_leg...</td>\n",
       "      <td>{\"npc_dota_hero_treant\":512,\"npc_dota_creep_go...</td>\n",
       "      <td>{\"undefined\":981,\"rubick_fade_bolt\":565}</td>\n",
       "      <td>{\"4\":1,\"5\":1}</td>\n",
       "      <td>{\"npc_dota_hero_treant\":1,\"npc_dota_hero_invok...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"0\":2270,\"1\":21,\"2\":233}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>{\"0\":33}</td>\n",
       "      <td>{\"stout_shield\":1,\"tango\":3,\"branches\":2,\"flas...</td>\n",
       "      <td>{\"0\":913,\"1\":-509,\"6\":350,\"11\":240,\"12\":2036,\"...</td>\n",
       "      <td>{\"0\":498,\"1\":3541,\"2\":12450}</td>\n",
       "      <td>{\"npc_dota_creep_goodguys_melee\":89,\"npc_dota_...</td>\n",
       "      <td>{\"tango\":4,\"branches\":2,\"power_treads\":9,\"faer...</td>\n",
       "      <td>{\"slark_pounce\":16,\"slark_dark_pact\":17,\"slark...</td>\n",
       "      <td>{\"undefined\":56,\"slark_pounce\":7,\"slark_dark_p...</td>\n",
       "      <td>{\"npc_dota_creep_goodguys_melee\":28193,\"npc_do...</td>\n",
       "      <td>{\"npc_dota_creep_goodguys_melee\":862,\"npc_dota...</td>\n",
       "      <td>{\"undefined\":4404,\"slark_pounce\":1012,\"slark_d...</td>\n",
       "      <td>{\"5\":3}</td>\n",
       "      <td>{\"npc_dota_hero_phantom_assassin\":1}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"0\":2456,\"1\":1,\"2\":67}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>{\"0\":15}</td>\n",
       "      <td>{\"circlet\":1,\"mantle\":1,\"null_talisman\":1,\"rec...</td>\n",
       "      <td>{\"0\":857,\"1\":-2814,\"2\":-406,\"6\":62,\"11\":240,\"1...</td>\n",
       "      <td>{\"0\":283,\"1\":4110,\"2\":14572}</td>\n",
       "      <td>{\"npc_dota_creep_goodguys_melee\":135,\"npc_dota...</td>\n",
       "      <td>{\"ward_observer\":1,\"tango_single\":2,\"bottle\":2...</td>\n",
       "      <td>{\"tinker_laser\":23,\"tinker_heat_seeking_missil...</td>\n",
       "      <td>{\"undefined\":22,\"tinker_laser\":14,\"tinker_heat...</td>\n",
       "      <td>{\"npc_dota_hero_treant\":8688,\"npc_dota_creep_b...</td>\n",
       "      <td>{\"npc_dota_creep_goodguys_ranged\":685,\"npc_dot...</td>\n",
       "      <td>{\"undefined\":1539,\"tinker_laser\":2960,\"tinker_...</td>\n",
       "      <td>{\"0\":1,\"1\":1,\"5\":3,\"6\":2}</td>\n",
       "      <td>{\"npc_dota_hero_invoker\":2,\"npc_dota_hero_trea...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"0\":2191,\"1\":14,\"2\":319}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>9289 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        pings                                           purchase  \\\n",
       "0     {\"0\":3}  {\"tango\":1,\"branches\":3,\"enchanted_mango\":1,\"w...   \n",
       "1    {\"0\":38}  {\"ward_observer\":10,\"courier\":1,\"clarity\":4,\"t...   \n",
       "2    {\"0\":45}  {\"circlet\":1,\"mantle\":1,\"null_talisman\":1,\"rec...   \n",
       "3     {\"0\":7}  {\"quelling_blade\":1,\"ring_of_protection\":1,\"ta...   \n",
       "4     {\"0\":4}  {\"tango\":2,\"slippers\":1,\"clarity\":2,\"flask\":1,...   \n",
       "..        ...                                                ...   \n",
       "125  {\"0\":41}  {\"ring_of_protection\":1,\"stout_shield\":1,\"tang...   \n",
       "126  {\"0\":24}  {\"orb_of_venom\":1,\"stout_shield\":1,\"tango\":1,\"...   \n",
       "127  {\"0\":14}  {\"courier\":1,\"ward_observer\":14,\"tango\":1,\"fla...   \n",
       "128  {\"0\":33}  {\"stout_shield\":1,\"tango\":3,\"branches\":2,\"flas...   \n",
       "129  {\"0\":15}  {\"circlet\":1,\"mantle\":1,\"null_talisman\":1,\"rec...   \n",
       "\n",
       "                                          gold_reasons  \\\n",
       "0    {\"0\":897,\"1\":-1075,\"6\":137,\"11\":4181,\"12\":2748...   \n",
       "1     {\"0\":816,\"11\":3860,\"12\":2525,\"13\":1520,\"14\":200}   \n",
       "2    {\"0\":969,\"1\":-1317,\"6\":330,\"11\":5120,\"12\":7047...   \n",
       "3    {\"0\":728,\"1\":-956,\"11\":4747,\"12\":2783,\"13\":631...   \n",
       "4    {\"0\":710,\"1\":-269,\"6\":1000,\"11\":5084,\"12\":5063...   \n",
       "..                                                 ...   \n",
       "125  {\"0\":943,\"1\":-1854,\"6\":100,\"11\":240,\"12\":1430,...   \n",
       "126  {\"0\":781,\"1\":-3019,\"6\":1887,\"11\":483,\"12\":1465...   \n",
       "127     {\"0\":745,\"1\":-1763,\"11\":240,\"12\":471,\"13\":897}   \n",
       "128  {\"0\":913,\"1\":-509,\"6\":350,\"11\":240,\"12\":2036,\"...   \n",
       "129  {\"0\":857,\"1\":-2814,\"2\":-406,\"6\":62,\"11\":240,\"1...   \n",
       "\n",
       "                                xp_reasons  \\\n",
       "0              {\"0\":526,\"1\":4800,\"2\":7057}   \n",
       "1              {\"0\":318,\"1\":6018,\"2\":7607}   \n",
       "2    {\"0\":506,\"1\":10992,\"2\":13979,\"3\":894}   \n",
       "3             {\"0\":196,\"1\":6218,\"2\":11610}   \n",
       "4     {\"0\":156,\"1\":9888,\"2\":18448,\"3\":894}   \n",
       "..                                     ...   \n",
       "125            {\"0\":498,\"1\":2761,\"2\":9659}   \n",
       "126            {\"0\":245,\"1\":1803,\"2\":9000}   \n",
       "127            {\"0\":266,\"1\":1141,\"2\":5482}   \n",
       "128           {\"0\":498,\"1\":3541,\"2\":12450}   \n",
       "129           {\"0\":283,\"1\":4110,\"2\":14572}   \n",
       "\n",
       "                                                killed  \\\n",
       "0    {\"npc_dota_creep_badguys_melee\":35,\"npc_dota_h...   \n",
       "1    {\"npc_dota_creep_goodguys_melee\":2,\"npc_dota_c...   \n",
       "2    {\"npc_dota_creep_badguys_melee\":117,\"npc_dota_...   \n",
       "3    {\"npc_dota_creep_badguys_ranged\":15,\"npc_dota_...   \n",
       "4    {\"npc_dota_creep_badguys_melee\":114,\"npc_dota_...   \n",
       "..                                                 ...   \n",
       "125  {\"npc_dota_creep_goodguys_melee\":85,\"npc_dota_...   \n",
       "126  {\"npc_dota_creep_goodguys_melee\":26,\"npc_dota_...   \n",
       "127  {\"npc_dota_creep_badguys_ranged\":3,\"npc_dota_c...   \n",
       "128  {\"npc_dota_creep_goodguys_melee\":89,\"npc_dota_...   \n",
       "129  {\"npc_dota_creep_goodguys_melee\":135,\"npc_dota...   \n",
       "\n",
       "                                             item_uses  \\\n",
       "0    {\"tango\":4,\"ward_observer\":3,\"enchanted_mango\"...   \n",
       "1    {\"courier\":1,\"ward_observer\":6,\"ward_sentry\":5...   \n",
       "2    {\"flask\":1,\"tango_single\":1,\"bottle\":43,\"tpscr...   \n",
       "3    {\"tango\":12,\"faerie_fire\":2,\"tpscroll\":8,\"quel...   \n",
       "4    {\"tango\":8,\"flask\":1,\"clarity\":2,\"phase_boots\"...   \n",
       "..                                                 ...   \n",
       "125  {\"tango\":4,\"faerie_fire\":1,\"branches\":1,\"magic...   \n",
       "126  {\"tango\":4,\"magic_stick\":14,\"tango_single\":1,\"...   \n",
       "127  {\"courier\":1,\"ward_observer\":8,\"tango\":4,\"bran...   \n",
       "128  {\"tango\":4,\"branches\":2,\"power_treads\":9,\"faer...   \n",
       "129  {\"ward_observer\":1,\"tango_single\":2,\"bottle\":2...   \n",
       "\n",
       "                                          ability_uses  \\\n",
       "0    {\"ogre_magi_ignite\":40,\"ogre_magi_fireblast\":1...   \n",
       "1    {\"treant_living_armor\":54,\"treant_natures_guis...   \n",
       "2    {\"invoker_exort\":560,\"invoker_invoke\":85,\"invo...   \n",
       "3    {\"beastmaster_call_of_the_wild\":23,\"beastmaste...   \n",
       "4    {\"juggernaut_blade_fury\":17,\"juggernaut_healin...   \n",
       "..                                                 ...   \n",
       "125  {\"centaur_hoof_stomp\":25,\"centaur_double_edge\"...   \n",
       "126  {\"spirit_breaker_charge_of_darkness\":24,\"spiri...   \n",
       "127  {\"rubick_telekinesis\":8,\"rubick_telekinesis_la...   \n",
       "128  {\"slark_pounce\":16,\"slark_dark_pact\":17,\"slark...   \n",
       "129  {\"tinker_laser\":23,\"tinker_heat_seeking_missil...   \n",
       "\n",
       "                                             hero_hits  \\\n",
       "0    {\"ogre_magi_ignite\":148,\"undefined\":44,\"ogre_m...   \n",
       "1               {\"treant_leech_seed\":61,\"undefined\":8}   \n",
       "2    {\"undefined\":95,\"invoker_cold_snap\":36,\"invoke...   \n",
       "3    {\"undefined\":172,\"beastmaster_wild_axes\":21,\"b...   \n",
       "4    {\"juggernaut_blade_fury\":166,\"undefined\":46,\"j...   \n",
       "..                                                 ...   \n",
       "125  {\"centaur_hoof_stomp\":11,\"undefined\":15,\"centa...   \n",
       "126  {\"undefined\":22,\"orb_of_venom\":51,\"spirit_brea...   \n",
       "127              {\"undefined\":26,\"rubick_fade_bolt\":5}   \n",
       "128  {\"undefined\":56,\"slark_pounce\":7,\"slark_dark_p...   \n",
       "129  {\"undefined\":22,\"tinker_laser\":14,\"tinker_heat...   \n",
       "\n",
       "                                                damage  \\\n",
       "0    {\"npc_dota_hero_slark\":4519,\"npc_dota_creep_ba...   \n",
       "1    {\"npc_dota_creep_goodguys_melee\":382,\"npc_dota...   \n",
       "2    {\"npc_dota_creep_goodguys_melee\":336,\"npc_dota...   \n",
       "3    {\"npc_dota_hero_omniknight\":1380,\"npc_dota_her...   \n",
       "4    {\"npc_dota_creep_badguys_melee\":63382,\"npc_dot...   \n",
       "..                                                 ...   \n",
       "125  {\"npc_dota_creep_goodguys_melee\":44278,\"npc_do...   \n",
       "126  {\"npc_dota_hero_phantom_assassin\":1355,\"npc_do...   \n",
       "127  {\"npc_dota_hero_treant\":571,\"npc_dota_hero_leg...   \n",
       "128  {\"npc_dota_creep_goodguys_melee\":28193,\"npc_do...   \n",
       "129  {\"npc_dota_hero_treant\":8688,\"npc_dota_creep_b...   \n",
       "\n",
       "                                          damage_taken  \\\n",
       "0    {\"npc_dota_creep_badguys_melee\":724,\"npc_dota_...   \n",
       "1    {\"npc_dota_hero_windrunner\":382,\"npc_dota_cree...   \n",
       "2    {\"npc_dota_hero_zuus\":8990,\"npc_dota_creep_bad...   \n",
       "3    {\"npc_dota_hero_omniknight\":1052,\"npc_dota_her...   \n",
       "4    {\"npc_dota_creep_badguys_melee\":1205,\"npc_dota...   \n",
       "..                                                 ...   \n",
       "125  {\"npc_dota_hero_lich\":2366,\"npc_dota_hero_phan...   \n",
       "126  {\"npc_dota_hero_phantom_assassin\":4834,\"npc_do...   \n",
       "127  {\"npc_dota_hero_treant\":512,\"npc_dota_creep_go...   \n",
       "128  {\"npc_dota_creep_goodguys_melee\":862,\"npc_dota...   \n",
       "129  {\"npc_dota_creep_goodguys_ranged\":685,\"npc_dot...   \n",
       "\n",
       "                                      damage_inflictor  \\\n",
       "0    {\"ogre_magi_ignite\":4744,\"undefined\":2123,\"ogr...   \n",
       "1           {\"treant_leech_seed\":1580,\"undefined\":468}   \n",
       "2    {\"undefined\":8374,\"invoker_cold_snap\":1012,\"in...   \n",
       "3    {\"undefined\":3530,\"beastmaster_wild_axes\":1930...   \n",
       "4    {\"juggernaut_blade_fury\":3586,\"undefined\":9240...   \n",
       "..                                                 ...   \n",
       "125  {\"centaur_hoof_stomp\":1549,\"undefined\":842,\"ce...   \n",
       "126  {\"undefined\":1137,\"orb_of_venom\":102,\"spirit_b...   \n",
       "127           {\"undefined\":981,\"rubick_fade_bolt\":565}   \n",
       "128  {\"undefined\":4404,\"slark_pounce\":1012,\"slark_d...   \n",
       "129  {\"undefined\":1539,\"tinker_laser\":2960,\"tinker_...   \n",
       "\n",
       "                               runes  \\\n",
       "0                      {\"3\":1,\"5\":3}   \n",
       "1    {\"2\":1,\"3\":1,\"4\":1,\"5\":1,\"6\":2}   \n",
       "2                      {\"4\":2,\"5\":4}   \n",
       "3                            {\"5\":1}   \n",
       "4                {\"1\":2,\"5\":1,\"6\":1}   \n",
       "..                               ...   \n",
       "125                          {\"5\":4}   \n",
       "126                    {\"3\":2,\"5\":2}   \n",
       "127                    {\"4\":1,\"5\":1}   \n",
       "128                          {\"5\":3}   \n",
       "129        {\"0\":1,\"1\":1,\"5\":3,\"6\":2}   \n",
       "\n",
       "                                             killed_by  \\\n",
       "0    {\"npc_dota_hero_omniknight\":1,\"npc_dota_hero_z...   \n",
       "1                                                   {}   \n",
       "2    {\"npc_dota_hero_furion\":1,\"npc_dota_hero_slark...   \n",
       "3    {\"npc_dota_hero_omniknight\":1,\"npc_dota_hero_s...   \n",
       "4                             {\"npc_dota_hero_zuus\":1}   \n",
       "..                                                 ...   \n",
       "125  {\"npc_dota_hero_lich\":2,\"npc_dota_hero_phantom...   \n",
       "126  {\"npc_dota_hero_phantom_assassin\":2,\"npc_dota_...   \n",
       "127  {\"npc_dota_hero_treant\":1,\"npc_dota_hero_invok...   \n",
       "128               {\"npc_dota_hero_phantom_assassin\":1}   \n",
       "129  {\"npc_dota_hero_invoker\":2,\"npc_dota_hero_trea...   \n",
       "\n",
       "                                    kill_streaks multi_kills  \\\n",
       "0                                        {\"3\":1}          {}   \n",
       "1                                             {}          {}   \n",
       "2    {\"3\":2,\"4\":2,\"5\":2,\"6\":2,\"7\":2,\"8\":2,\"9\":1}     {\"2\":1}   \n",
       "3                                        {\"3\":1}          {}   \n",
       "4    {\"3\":1,\"4\":1,\"5\":1,\"6\":1,\"7\":1,\"8\":1,\"9\":1}     {\"2\":1}   \n",
       "..                                           ...         ...   \n",
       "125                                           {}          {}   \n",
       "126                                           {}          {}   \n",
       "127                                           {}          {}   \n",
       "128                                           {}          {}   \n",
       "129                                           {}          {}   \n",
       "\n",
       "                    life_state  \n",
       "0    {\"0\":2163,\"1\":10,\"2\":148}  \n",
       "1                   {\"0\":2321}  \n",
       "2    {\"0\":2142,\"1\":10,\"2\":169}  \n",
       "3    {\"0\":2183,\"1\":12,\"2\":126}  \n",
       "4      {\"0\":2282,\"1\":3,\"2\":36}  \n",
       "..                         ...  \n",
       "125  {\"0\":2264,\"1\":20,\"2\":240}  \n",
       "126  {\"0\":2094,\"1\":31,\"2\":399}  \n",
       "127  {\"0\":2270,\"1\":21,\"2\":233}  \n",
       "128    {\"0\":2456,\"1\":1,\"2\":67}  \n",
       "129  {\"0\":2191,\"1\":14,\"2\":319}  \n",
       "\n",
       "[9289 rows x 16 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skilled_player_stats[skilled_player_stats.columns[-16:]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, all 31 of these object columns contain dictionaries with various numerical values. It is unclear how deep some of these dictionaries go, so time to do some exploring to find out just how much dimensionality this dataset could potentially have! I can already see that the \"times\" and any column with the \"_t\" suffix is actually a list of numerical values, and would probably create far too much dimensionality."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Informed Approach"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a column called \"leaver_status,\" a value greater than 0 indicates that the player left the game at a certain point. '\n",
    "\n",
    "The different numbers indicate how the player left the game. 1 is where they disconnected, but reconnected within 5 minutes of disconnection, 2 is where they disconnected and didn't come back within 5 minutes of disconnecting, 3 is where they actually pressed the \"Quit\" button, and 4 is where they stayed connected to the game, but didn't actively participate. \n",
    "\n",
    "Since the game is team based, if a team is lacking one or more players, it will make that team perform worse, and the opposing team perform better, causing the data to not be an accurate representation of a player's usual behavior and performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0    9004\n",
       "1.0     187\n",
       "3.0      48\n",
       "2.0      41\n",
       "4.0       9\n",
       "Name: leaver_status, dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skilled_player_stats['leaver_status'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So I will filter out all players that were in a game where at least one player left(or was inactive) at any point in the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "skilled_player_stats_informed = skilled_player_stats[\n",
    "    ~skilled_player_stats[\"match_id\"].isin( # No matches...\n",
    "    skilled_player_stats[\n",
    "        skilled_player_stats[\"leaver_status\"] > 0][\"match_id\"])].reset_index(drop=True) # where a player left at any point"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7794"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(skilled_player_stats_informed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filtered out over 1400 players, but still have a decent amount to work with.\n",
    "\n",
    "Going to start with dropping the same columns that I dropped in the naive approach.\n",
    "\n",
    "**Daskified Version Note**: I omitted the naive approach section of this notebook as it ended up being inferior, and I mainly wanted to demonstrate how Dask can assist in reading in the large amount of data and paralellization of clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "skilled_player_stats_informed = skilled_player_stats_informed.drop(\n",
    "    columns=[\"leaver_status\", \"actions\", \"additional_units\", \"times\", \n",
    "             \"gold_t\", \"lh_t\", \"xp_t\", \"max_hero_hit\", \"obs_log\", \n",
    "             \"sen_log\", \"purchase_log\", \"kills_log\", \"buyback_log\", \n",
    "             \"life_state\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item_0</th>\n",
       "      <th>item_1</th>\n",
       "      <th>item_2</th>\n",
       "      <th>item_3</th>\n",
       "      <th>item_4</th>\n",
       "      <th>item_5</th>\n",
       "      <th>kills</th>\n",
       "      <th>deaths</th>\n",
       "      <th>assists</th>\n",
       "      <th>gold</th>\n",
       "      <th>last_hits</th>\n",
       "      <th>denies</th>\n",
       "      <th>gold_per_min</th>\n",
       "      <th>xp_per_min</th>\n",
       "      <th>gold_spent</th>\n",
       "      <th>hero_damage</th>\n",
       "      <th>tower_damage</th>\n",
       "      <th>hero_healing</th>\n",
       "      <th>level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100</td>\n",
       "      <td>90</td>\n",
       "      <td>46</td>\n",
       "      <td>178</td>\n",
       "      <td>214</td>\n",
       "      <td>181</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>12</td>\n",
       "      <td>3648.0</td>\n",
       "      <td>58</td>\n",
       "      <td>6</td>\n",
       "      <td>369</td>\n",
       "      <td>340</td>\n",
       "      <td>9800.0</td>\n",
       "      <td>9320.0</td>\n",
       "      <td>362.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11</td>\n",
       "      <td>231</td>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>46</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>822.0</td>\n",
       "      <td>36</td>\n",
       "      <td>3</td>\n",
       "      <td>328</td>\n",
       "      <td>384</td>\n",
       "      <td>11725.0</td>\n",
       "      <td>1699.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>1519.0</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>48</td>\n",
       "      <td>100</td>\n",
       "      <td>203</td>\n",
       "      <td>108</td>\n",
       "      <td>110</td>\n",
       "      <td>65</td>\n",
       "      <td>20</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>1740.0</td>\n",
       "      <td>188</td>\n",
       "      <td>4</td>\n",
       "      <td>709</td>\n",
       "      <td>726</td>\n",
       "      <td>24140.0</td>\n",
       "      <td>20841.0</td>\n",
       "      <td>4494.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>81</td>\n",
       "      <td>48</td>\n",
       "      <td>11</td>\n",
       "      <td>194</td>\n",
       "      <td>46</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>12</td>\n",
       "      <td>3797.0</td>\n",
       "      <td>158</td>\n",
       "      <td>0</td>\n",
       "      <td>490</td>\n",
       "      <td>496</td>\n",
       "      <td>13650.0</td>\n",
       "      <td>6316.0</td>\n",
       "      <td>2112.0</td>\n",
       "      <td>454.0</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>50</td>\n",
       "      <td>249</td>\n",
       "      <td>145</td>\n",
       "      <td>135</td>\n",
       "      <td>168</td>\n",
       "      <td>108</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>858.0</td>\n",
       "      <td>312</td>\n",
       "      <td>21</td>\n",
       "      <td>735</td>\n",
       "      <td>810</td>\n",
       "      <td>27425.0</td>\n",
       "      <td>14669.0</td>\n",
       "      <td>4621.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   item_0  item_1  item_2  item_3  item_4  item_5  kills  deaths  assists  \\\n",
       "0     100      90      46     178     214     181      4       5       12   \n",
       "1      11     231      37       1       0      46      1       0       16   \n",
       "2      48     100     203     108     110      65     20       3        8   \n",
       "3       1      81      48      11     194      46      4       4       12   \n",
       "4      50     249     145     135     168     108     11       1        6   \n",
       "\n",
       "     gold  last_hits  denies  gold_per_min  xp_per_min  gold_spent  \\\n",
       "0  3648.0         58       6           369         340      9800.0   \n",
       "1   822.0         36       3           328         384     11725.0   \n",
       "2  1740.0        188       4           709         726     24140.0   \n",
       "3  3797.0        158       0           490         496     13650.0   \n",
       "4   858.0        312      21           735         810     27425.0   \n",
       "\n",
       "   hero_damage  tower_damage  hero_healing  level  \n",
       "0       9320.0         362.0           0.0     15  \n",
       "1       1699.0          94.0        1519.0     16  \n",
       "2      20841.0        4494.0           0.0     22  \n",
       "3       6316.0        2112.0         454.0     18  \n",
       "4      14669.0        4621.0           0.0     23  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skilled_player_stats_informed[skilled_player_stats_informed.columns[4:23]\n",
    "                             ].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "item_0-5 is useful as it tells what items each player had equipped at the end of their game, and the other columns so far seem to be useful and do not need to be dropped or engineered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>stuns</th>\n",
       "      <th>lane_pos</th>\n",
       "      <th>obs</th>\n",
       "      <th>sen</th>\n",
       "      <th>pings</th>\n",
       "      <th>purchase</th>\n",
       "      <th>gold_reasons</th>\n",
       "      <th>xp_reasons</th>\n",
       "      <th>killed</th>\n",
       "      <th>item_uses</th>\n",
       "      <th>ability_uses</th>\n",
       "      <th>hero_hits</th>\n",
       "      <th>damage</th>\n",
       "      <th>damage_taken</th>\n",
       "      <th>damage_inflictor</th>\n",
       "      <th>runes</th>\n",
       "      <th>killed_by</th>\n",
       "      <th>kill_streaks</th>\n",
       "      <th>multi_kills</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>20.569000</td>\n",
       "      <td>{\"70\":{\"164\":1,\"166\":1,\"168\":1,\"170\":5},\"72\":{...</td>\n",
       "      <td>{\"100\":{\"158\":1},\"140\":{\"106\":1},\"166\":{\"102\":1}}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"0\":3}</td>\n",
       "      <td>{\"tango\":1,\"branches\":3,\"enchanted_mango\":1,\"w...</td>\n",
       "      <td>{\"0\":897,\"1\":-1075,\"6\":137,\"11\":4181,\"12\":2748...</td>\n",
       "      <td>{\"0\":526,\"1\":4800,\"2\":7057}</td>\n",
       "      <td>{\"npc_dota_creep_badguys_melee\":35,\"npc_dota_h...</td>\n",
       "      <td>{\"tango\":4,\"ward_observer\":3,\"enchanted_mango\"...</td>\n",
       "      <td>{\"ogre_magi_ignite\":40,\"ogre_magi_fireblast\":1...</td>\n",
       "      <td>{\"ogre_magi_ignite\":148,\"undefined\":44,\"ogre_m...</td>\n",
       "      <td>{\"npc_dota_hero_slark\":4519,\"npc_dota_creep_ba...</td>\n",
       "      <td>{\"npc_dota_creep_badguys_melee\":724,\"npc_dota_...</td>\n",
       "      <td>{\"ogre_magi_ignite\":4744,\"undefined\":2123,\"ogr...</td>\n",
       "      <td>{\"3\":1,\"5\":3}</td>\n",
       "      <td>{\"npc_dota_hero_omniknight\":1,\"npc_dota_hero_z...</td>\n",
       "      <td>{\"3\":1}</td>\n",
       "      <td>{}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38.754900</td>\n",
       "      <td>{\"70\":{\"74\":1,\"76\":1,\"78\":8},\"74\":{\"78\":1,\"158...</td>\n",
       "      <td>{\"94\":{\"160\":1},\"100\":{\"128\":1},\"112\":{\"146\":1...</td>\n",
       "      <td>{\"90\":{\"160\":1},\"94\":{\"160\":1},\"110\":{\"146\":1}...</td>\n",
       "      <td>{\"0\":38}</td>\n",
       "      <td>{\"ward_observer\":10,\"courier\":1,\"clarity\":4,\"t...</td>\n",
       "      <td>{\"0\":816,\"11\":3860,\"12\":2525,\"13\":1520,\"14\":200}</td>\n",
       "      <td>{\"0\":318,\"1\":6018,\"2\":7607}</td>\n",
       "      <td>{\"npc_dota_creep_goodguys_melee\":2,\"npc_dota_c...</td>\n",
       "      <td>{\"courier\":1,\"ward_observer\":6,\"ward_sentry\":5...</td>\n",
       "      <td>{\"treant_living_armor\":54,\"treant_natures_guis...</td>\n",
       "      <td>{\"treant_leech_seed\":61,\"undefined\":8}</td>\n",
       "      <td>{\"npc_dota_creep_goodguys_melee\":382,\"npc_dota...</td>\n",
       "      <td>{\"npc_dota_hero_windrunner\":382,\"npc_dota_cree...</td>\n",
       "      <td>{\"treant_leech_seed\":1580,\"undefined\":468}</td>\n",
       "      <td>{\"2\":1,\"3\":1,\"4\":1,\"5\":1,\"6\":2}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>85.917400</td>\n",
       "      <td>{\"72\":{\"76\":50,\"78\":1},\"74\":{\"78\":32,\"80\":1,\"8...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"0\":45}</td>\n",
       "      <td>{\"circlet\":1,\"mantle\":1,\"null_talisman\":1,\"rec...</td>\n",
       "      <td>{\"0\":969,\"1\":-1317,\"6\":330,\"11\":5120,\"12\":7047...</td>\n",
       "      <td>{\"0\":506,\"1\":10992,\"2\":13979,\"3\":894}</td>\n",
       "      <td>{\"npc_dota_creep_badguys_melee\":117,\"npc_dota_...</td>\n",
       "      <td>{\"flask\":1,\"tango_single\":1,\"bottle\":43,\"tpscr...</td>\n",
       "      <td>{\"invoker_exort\":560,\"invoker_invoke\":85,\"invo...</td>\n",
       "      <td>{\"undefined\":95,\"invoker_cold_snap\":36,\"invoke...</td>\n",
       "      <td>{\"npc_dota_creep_goodguys_melee\":336,\"npc_dota...</td>\n",
       "      <td>{\"npc_dota_hero_zuus\":8990,\"npc_dota_creep_bad...</td>\n",
       "      <td>{\"undefined\":8374,\"invoker_cold_snap\":1012,\"in...</td>\n",
       "      <td>{\"4\":2,\"5\":4}</td>\n",
       "      <td>{\"npc_dota_hero_furion\":1,\"npc_dota_hero_slark...</td>\n",
       "      <td>{\"3\":2,\"4\":2,\"5\":2,\"6\":2,\"7\":2,\"8\":2,\"9\":1}</td>\n",
       "      <td>{\"2\":1}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>13.024900</td>\n",
       "      <td>{\"70\":{\"74\":10,\"76\":8},\"72\":{\"74\":2,\"76\":4,\"16...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"0\":7}</td>\n",
       "      <td>{\"quelling_blade\":1,\"ring_of_protection\":1,\"ta...</td>\n",
       "      <td>{\"0\":728,\"1\":-956,\"11\":4747,\"12\":2783,\"13\":631...</td>\n",
       "      <td>{\"0\":196,\"1\":6218,\"2\":11610}</td>\n",
       "      <td>{\"npc_dota_creep_badguys_ranged\":15,\"npc_dota_...</td>\n",
       "      <td>{\"tango\":12,\"faerie_fire\":2,\"tpscroll\":8,\"quel...</td>\n",
       "      <td>{\"beastmaster_call_of_the_wild\":23,\"beastmaste...</td>\n",
       "      <td>{\"undefined\":172,\"beastmaster_wild_axes\":21,\"b...</td>\n",
       "      <td>{\"npc_dota_hero_omniknight\":1380,\"npc_dota_her...</td>\n",
       "      <td>{\"npc_dota_hero_omniknight\":1052,\"npc_dota_her...</td>\n",
       "      <td>{\"undefined\":3530,\"beastmaster_wild_axes\":1930...</td>\n",
       "      <td>{\"5\":1}</td>\n",
       "      <td>{\"npc_dota_hero_omniknight\":1,\"npc_dota_hero_s...</td>\n",
       "      <td>{\"3\":1}</td>\n",
       "      <td>{}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.033418</td>\n",
       "      <td>{\"70\":{\"76\":8},\"72\":{\"76\":1,\"78\":7},\"74\":{\"76\"...</td>\n",
       "      <td>{}</td>\n",
       "      <td>{}</td>\n",
       "      <td>{\"0\":4}</td>\n",
       "      <td>{\"tango\":2,\"slippers\":1,\"clarity\":2,\"flask\":1,...</td>\n",
       "      <td>{\"0\":710,\"1\":-269,\"6\":1000,\"11\":5084,\"12\":5063...</td>\n",
       "      <td>{\"0\":156,\"1\":9888,\"2\":18448,\"3\":894}</td>\n",
       "      <td>{\"npc_dota_creep_badguys_melee\":114,\"npc_dota_...</td>\n",
       "      <td>{\"tango\":8,\"flask\":1,\"clarity\":2,\"phase_boots\"...</td>\n",
       "      <td>{\"juggernaut_blade_fury\":17,\"juggernaut_healin...</td>\n",
       "      <td>{\"juggernaut_blade_fury\":166,\"undefined\":46,\"j...</td>\n",
       "      <td>{\"npc_dota_creep_badguys_melee\":63382,\"npc_dot...</td>\n",
       "      <td>{\"npc_dota_creep_badguys_melee\":1205,\"npc_dota...</td>\n",
       "      <td>{\"juggernaut_blade_fury\":3586,\"undefined\":9240...</td>\n",
       "      <td>{\"1\":2,\"5\":1,\"6\":1}</td>\n",
       "      <td>{\"npc_dota_hero_zuus\":1}</td>\n",
       "      <td>{\"3\":1,\"4\":1,\"5\":1,\"6\":1,\"7\":1,\"8\":1,\"9\":1}</td>\n",
       "      <td>{\"2\":1}</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       stuns                                           lane_pos  \\\n",
       "0  20.569000  {\"70\":{\"164\":1,\"166\":1,\"168\":1,\"170\":5},\"72\":{...   \n",
       "1  38.754900  {\"70\":{\"74\":1,\"76\":1,\"78\":8},\"74\":{\"78\":1,\"158...   \n",
       "2  85.917400  {\"72\":{\"76\":50,\"78\":1},\"74\":{\"78\":32,\"80\":1,\"8...   \n",
       "3  13.024900  {\"70\":{\"74\":10,\"76\":8},\"72\":{\"74\":2,\"76\":4,\"16...   \n",
       "4   0.033418  {\"70\":{\"76\":8},\"72\":{\"76\":1,\"78\":7},\"74\":{\"76\"...   \n",
       "\n",
       "                                                 obs  \\\n",
       "0  {\"100\":{\"158\":1},\"140\":{\"106\":1},\"166\":{\"102\":1}}   \n",
       "1  {\"94\":{\"160\":1},\"100\":{\"128\":1},\"112\":{\"146\":1...   \n",
       "2                                                 {}   \n",
       "3                                                 {}   \n",
       "4                                                 {}   \n",
       "\n",
       "                                                 sen     pings  \\\n",
       "0                                                 {}   {\"0\":3}   \n",
       "1  {\"90\":{\"160\":1},\"94\":{\"160\":1},\"110\":{\"146\":1}...  {\"0\":38}   \n",
       "2                                                 {}  {\"0\":45}   \n",
       "3                                                 {}   {\"0\":7}   \n",
       "4                                                 {}   {\"0\":4}   \n",
       "\n",
       "                                            purchase  \\\n",
       "0  {\"tango\":1,\"branches\":3,\"enchanted_mango\":1,\"w...   \n",
       "1  {\"ward_observer\":10,\"courier\":1,\"clarity\":4,\"t...   \n",
       "2  {\"circlet\":1,\"mantle\":1,\"null_talisman\":1,\"rec...   \n",
       "3  {\"quelling_blade\":1,\"ring_of_protection\":1,\"ta...   \n",
       "4  {\"tango\":2,\"slippers\":1,\"clarity\":2,\"flask\":1,...   \n",
       "\n",
       "                                        gold_reasons  \\\n",
       "0  {\"0\":897,\"1\":-1075,\"6\":137,\"11\":4181,\"12\":2748...   \n",
       "1   {\"0\":816,\"11\":3860,\"12\":2525,\"13\":1520,\"14\":200}   \n",
       "2  {\"0\":969,\"1\":-1317,\"6\":330,\"11\":5120,\"12\":7047...   \n",
       "3  {\"0\":728,\"1\":-956,\"11\":4747,\"12\":2783,\"13\":631...   \n",
       "4  {\"0\":710,\"1\":-269,\"6\":1000,\"11\":5084,\"12\":5063...   \n",
       "\n",
       "                              xp_reasons  \\\n",
       "0            {\"0\":526,\"1\":4800,\"2\":7057}   \n",
       "1            {\"0\":318,\"1\":6018,\"2\":7607}   \n",
       "2  {\"0\":506,\"1\":10992,\"2\":13979,\"3\":894}   \n",
       "3           {\"0\":196,\"1\":6218,\"2\":11610}   \n",
       "4   {\"0\":156,\"1\":9888,\"2\":18448,\"3\":894}   \n",
       "\n",
       "                                              killed  \\\n",
       "0  {\"npc_dota_creep_badguys_melee\":35,\"npc_dota_h...   \n",
       "1  {\"npc_dota_creep_goodguys_melee\":2,\"npc_dota_c...   \n",
       "2  {\"npc_dota_creep_badguys_melee\":117,\"npc_dota_...   \n",
       "3  {\"npc_dota_creep_badguys_ranged\":15,\"npc_dota_...   \n",
       "4  {\"npc_dota_creep_badguys_melee\":114,\"npc_dota_...   \n",
       "\n",
       "                                           item_uses  \\\n",
       "0  {\"tango\":4,\"ward_observer\":3,\"enchanted_mango\"...   \n",
       "1  {\"courier\":1,\"ward_observer\":6,\"ward_sentry\":5...   \n",
       "2  {\"flask\":1,\"tango_single\":1,\"bottle\":43,\"tpscr...   \n",
       "3  {\"tango\":12,\"faerie_fire\":2,\"tpscroll\":8,\"quel...   \n",
       "4  {\"tango\":8,\"flask\":1,\"clarity\":2,\"phase_boots\"...   \n",
       "\n",
       "                                        ability_uses  \\\n",
       "0  {\"ogre_magi_ignite\":40,\"ogre_magi_fireblast\":1...   \n",
       "1  {\"treant_living_armor\":54,\"treant_natures_guis...   \n",
       "2  {\"invoker_exort\":560,\"invoker_invoke\":85,\"invo...   \n",
       "3  {\"beastmaster_call_of_the_wild\":23,\"beastmaste...   \n",
       "4  {\"juggernaut_blade_fury\":17,\"juggernaut_healin...   \n",
       "\n",
       "                                           hero_hits  \\\n",
       "0  {\"ogre_magi_ignite\":148,\"undefined\":44,\"ogre_m...   \n",
       "1             {\"treant_leech_seed\":61,\"undefined\":8}   \n",
       "2  {\"undefined\":95,\"invoker_cold_snap\":36,\"invoke...   \n",
       "3  {\"undefined\":172,\"beastmaster_wild_axes\":21,\"b...   \n",
       "4  {\"juggernaut_blade_fury\":166,\"undefined\":46,\"j...   \n",
       "\n",
       "                                              damage  \\\n",
       "0  {\"npc_dota_hero_slark\":4519,\"npc_dota_creep_ba...   \n",
       "1  {\"npc_dota_creep_goodguys_melee\":382,\"npc_dota...   \n",
       "2  {\"npc_dota_creep_goodguys_melee\":336,\"npc_dota...   \n",
       "3  {\"npc_dota_hero_omniknight\":1380,\"npc_dota_her...   \n",
       "4  {\"npc_dota_creep_badguys_melee\":63382,\"npc_dot...   \n",
       "\n",
       "                                        damage_taken  \\\n",
       "0  {\"npc_dota_creep_badguys_melee\":724,\"npc_dota_...   \n",
       "1  {\"npc_dota_hero_windrunner\":382,\"npc_dota_cree...   \n",
       "2  {\"npc_dota_hero_zuus\":8990,\"npc_dota_creep_bad...   \n",
       "3  {\"npc_dota_hero_omniknight\":1052,\"npc_dota_her...   \n",
       "4  {\"npc_dota_creep_badguys_melee\":1205,\"npc_dota...   \n",
       "\n",
       "                                    damage_inflictor  \\\n",
       "0  {\"ogre_magi_ignite\":4744,\"undefined\":2123,\"ogr...   \n",
       "1         {\"treant_leech_seed\":1580,\"undefined\":468}   \n",
       "2  {\"undefined\":8374,\"invoker_cold_snap\":1012,\"in...   \n",
       "3  {\"undefined\":3530,\"beastmaster_wild_axes\":1930...   \n",
       "4  {\"juggernaut_blade_fury\":3586,\"undefined\":9240...   \n",
       "\n",
       "                             runes  \\\n",
       "0                    {\"3\":1,\"5\":3}   \n",
       "1  {\"2\":1,\"3\":1,\"4\":1,\"5\":1,\"6\":2}   \n",
       "2                    {\"4\":2,\"5\":4}   \n",
       "3                          {\"5\":1}   \n",
       "4              {\"1\":2,\"5\":1,\"6\":1}   \n",
       "\n",
       "                                           killed_by  \\\n",
       "0  {\"npc_dota_hero_omniknight\":1,\"npc_dota_hero_z...   \n",
       "1                                                 {}   \n",
       "2  {\"npc_dota_hero_furion\":1,\"npc_dota_hero_slark...   \n",
       "3  {\"npc_dota_hero_omniknight\":1,\"npc_dota_hero_s...   \n",
       "4                           {\"npc_dota_hero_zuus\":1}   \n",
       "\n",
       "                                  kill_streaks multi_kills  \n",
       "0                                      {\"3\":1}          {}  \n",
       "1                                           {}          {}  \n",
       "2  {\"3\":2,\"4\":2,\"5\":2,\"6\":2,\"7\":2,\"8\":2,\"9\":1}     {\"2\":1}  \n",
       "3                                      {\"3\":1}          {}  \n",
       "4  {\"3\":1,\"4\":1,\"5\":1,\"6\":1,\"7\":1,\"8\":1,\"9\":1}     {\"2\":1}  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "skilled_player_stats_informed[skilled_player_stats_informed.columns[23:43]\n",
    "                             ].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering and Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"pings\" column can easily be converted to just a single column of the amount of times a player marked a location of interest on the map, showing how active of a communicator they are."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "skilled_player_stats_informed[\"pings\"] = json_normalize(skilled_player_stats_informed[\"pings\"].map(eval)\n",
    "                                                       )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "skilled_player_stats_informed = pd.concat([\n",
    "    skilled_player_stats_informed,\n",
    "    json_normalize(skilled_player_stats_informed[\"gold_reasons\"].map(eval)).add_prefix(\"gold_reasons_\")\n",
    "    ], axis=1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "skilled_player_stats_informed = pd.concat([\n",
    "    skilled_player_stats_informed,\n",
    "    json_normalize(skilled_player_stats_informed[\"xp_reasons\"].map(eval)).add_prefix(\"xp_reasons_\")\n",
    "    ], axis=1\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\"damage_taken\" is interesting, where it seems like a lot of dimensionality, it can be filtered to only have columns containing neutral monsters, since there is a certain type of play style that will take a lot more damage from these neutral monsters than others, so I will sum up all the values from these columns containing the damage taken from neutral monsters, and make them into one single column, called \"neutral_damage_taken.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "0       1009.0\n1        525.0\n2        159.0\n3       6192.0\n4       3045.0\n         ...  \n7789     816.0\n7790     516.0\n7791      48.0\n7792    3360.0\n7793    1516.0\nName: neutral_damage_taken, Length: 7794, dtype: float64"
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "neutral_cols = [col for col in json_normalize(skilled_player_stats_informed[\"damage_taken\"].map(eval)\n",
    "                                             ) if col.startswith('npc_dota_neutral')]\n",
    "\n",
    "skilled_player_stats_informed[\"neutral_damage_taken\"] = json_normalize(skilled_player_stats_informed[\"damage_taken\"].map(eval)\n",
    "                                                                      )[neutral_cols].sum(axis=1, skipna=True)\n",
    "del neutral_cols\n",
    "\n",
    "skilled_player_stats_informed[\"neutral_damage_taken\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I will create another feature in a similar way that I did with \"neutral_damage_taken,\" except by how much damage a player does to enemy minions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "0       23369.0\n1        6932.0\n2       71579.0\n3       28463.0\n4       82082.0\n         ...   \n7789    53928.0\n7790    19797.0\n7791     7211.0\n7792    36437.0\n7793    93117.0\nName: damage_to_minion, Length: 7794, dtype: float64"
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "minion_cols = [col for col in json_normalize(skilled_player_stats_informed[\"damage\"].map(eval)\n",
    "                                            ) if col.startswith('npc_dota_creep')]\n",
    "\n",
    "skilled_player_stats_informed['damage_to_minion'] = json_normalize(skilled_player_stats_informed[\"damage\"].map(eval)\n",
    "                                                                  )[minion_cols].sum(axis=1, skipna=True)\n",
    "del minion_cols\n",
    "\n",
    "skilled_player_stats_informed['damage_to_minion']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for \"runes.\" Runes are power-up items that can be found on the map at certain locations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "skilled_player_stats_informed = pd.concat([\n",
    "    skilled_player_stats_informed,\n",
    "    json_normalize(skilled_player_stats_informed[\"runes\"].map(eval)).add_prefix(\"rune_\")\n",
    "], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for the final two columns that are kind of related: \"kill_streaks\" and \"multi_kills\"\n",
    "A \"kill_streak\" is started when a player kills another player at least twice without dying.\n",
    "A \"multi_kill\" is when a player kills more than one player in a short period of time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These both should add some value in differentiating very high performing offensive players without adding too much dimensionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "skilled_player_stats_informed = pd.concat([\n",
    "    skilled_player_stats_informed,\n",
    "    json_normalize(skilled_player_stats_informed[\"kill_streaks\"].map(eval)\n",
    "                  ).add_prefix(\"kill_streak_\"),\n",
    "    json_normalize(skilled_player_stats_informed[\"multi_kills\"].map(eval)\n",
    "                  ).add_prefix(\"multi_kill_\")\n",
    "], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to drop all the columns I won't need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "skilled_player_stats_informed = skilled_player_stats_informed.drop(\n",
    "    columns=[\n",
    "        \"lane_pos\", \"obs\", \"sen\", \"purchase\", \"killed\", \"item_uses\", \n",
    "        \"ability_uses\", \"hero_hits\", \"damage\", \"damage_taken\", \n",
    "        \"damage_inflictor\", \"runes\", \"killed_by\", \"kill_streaks\",\n",
    "        \"multi_kills\", \"gold_reasons\", \"xp_reasons\", \"skill\"\n",
    "    ]\n",
    ").fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 7794 entries, 0 to 7793\n",
      "Data columns (total 59 columns):\n",
      "match_id                7794 non-null int64\n",
      "player_slot             7794 non-null int64\n",
      "hero_id                 7794 non-null int64\n",
      "item_0                  7794 non-null int64\n",
      "item_1                  7794 non-null int64\n",
      "item_2                  7794 non-null int64\n",
      "item_3                  7794 non-null int64\n",
      "item_4                  7794 non-null int64\n",
      "item_5                  7794 non-null int64\n",
      "kills                   7794 non-null int64\n",
      "deaths                  7794 non-null int64\n",
      "assists                 7794 non-null int64\n",
      "gold                    7794 non-null float64\n",
      "last_hits               7794 non-null int64\n",
      "denies                  7794 non-null int64\n",
      "gold_per_min            7794 non-null int64\n",
      "xp_per_min              7794 non-null int64\n",
      "gold_spent              7794 non-null float64\n",
      "hero_damage             7794 non-null float64\n",
      "tower_damage            7794 non-null float64\n",
      "hero_healing            7794 non-null float64\n",
      "level                   7794 non-null int64\n",
      "stuns                   7794 non-null float64\n",
      "pings                   7794 non-null float64\n",
      "gold_reasons_0          7794 non-null float64\n",
      "gold_reasons_1          7794 non-null float64\n",
      "gold_reasons_6          7794 non-null float64\n",
      "gold_reasons_11         7794 non-null float64\n",
      "gold_reasons_12         7794 non-null float64\n",
      "gold_reasons_13         7794 non-null int64\n",
      "gold_reasons_14         7794 non-null float64\n",
      "gold_reasons_15         7794 non-null float64\n",
      "gold_reasons_2          7794 non-null float64\n",
      "gold_reasons_5          7794 non-null float64\n",
      "xp_reasons_0            7794 non-null float64\n",
      "xp_reasons_1            7794 non-null float64\n",
      "xp_reasons_2            7794 non-null int64\n",
      "xp_reasons_3            7794 non-null float64\n",
      "neutral_damage_taken    7794 non-null float64\n",
      "damage_to_minion        7794 non-null float64\n",
      "rune_3                  7794 non-null float64\n",
      "rune_5                  7794 non-null float64\n",
      "rune_2                  7794 non-null float64\n",
      "rune_4                  7794 non-null float64\n",
      "rune_6                  7794 non-null float64\n",
      "rune_1                  7794 non-null float64\n",
      "rune_0                  7794 non-null float64\n",
      "kill_streak_3           7794 non-null float64\n",
      "kill_streak_4           7794 non-null float64\n",
      "kill_streak_5           7794 non-null float64\n",
      "kill_streak_6           7794 non-null float64\n",
      "kill_streak_7           7794 non-null float64\n",
      "kill_streak_8           7794 non-null float64\n",
      "kill_streak_9           7794 non-null float64\n",
      "kill_streak_10          7794 non-null float64\n",
      "multi_kill_2            7794 non-null float64\n",
      "multi_kill_3            7794 non-null float64\n",
      "multi_kill_4            7794 non-null float64\n",
      "multi_kill_5            7794 non-null float64\n",
      "dtypes: float64(40), int64(19)\n",
      "memory usage: 3.5 MB\n"
     ]
    }
   ],
   "source": [
    "skilled_player_stats_informed.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Back to Dask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that feature engineering and cleaning is complete, I will convert the Pandas dataframe into a dask dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "skilled_player_stats = dd.from_pandas(skilled_player_stats_informed, npartitions=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scaling with Dask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the dataframe is so small, I will have it persist in memory to improve performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": "<div><strong>Dask DataFrame Structure:</strong></div>\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>item_0</th>\n      <th>item_1</th>\n      <th>item_2</th>\n      <th>item_3</th>\n      <th>item_4</th>\n      <th>item_5</th>\n      <th>kills</th>\n      <th>deaths</th>\n      <th>assists</th>\n      <th>gold</th>\n      <th>last_hits</th>\n      <th>denies</th>\n      <th>gold_per_min</th>\n      <th>xp_per_min</th>\n      <th>gold_spent</th>\n      <th>hero_damage</th>\n      <th>tower_damage</th>\n      <th>hero_healing</th>\n      <th>level</th>\n      <th>stuns</th>\n      <th>pings</th>\n      <th>gold_reasons_0</th>\n      <th>gold_reasons_1</th>\n      <th>gold_reasons_6</th>\n      <th>gold_reasons_11</th>\n      <th>gold_reasons_12</th>\n      <th>gold_reasons_13</th>\n      <th>gold_reasons_14</th>\n      <th>gold_reasons_15</th>\n      <th>gold_reasons_2</th>\n      <th>gold_reasons_5</th>\n      <th>xp_reasons_0</th>\n      <th>xp_reasons_1</th>\n      <th>xp_reasons_2</th>\n      <th>xp_reasons_3</th>\n      <th>neutral_damage_taken</th>\n      <th>damage_to_minion</th>\n      <th>rune_3</th>\n      <th>rune_5</th>\n      <th>rune_2</th>\n      <th>rune_4</th>\n      <th>rune_6</th>\n      <th>rune_1</th>\n      <th>rune_0</th>\n      <th>kill_streak_3</th>\n      <th>kill_streak_4</th>\n      <th>kill_streak_5</th>\n      <th>kill_streak_6</th>\n      <th>kill_streak_7</th>\n      <th>kill_streak_8</th>\n      <th>kill_streak_9</th>\n      <th>kill_streak_10</th>\n      <th>multi_kill_2</th>\n      <th>multi_kill_3</th>\n      <th>multi_kill_4</th>\n      <th>multi_kill_5</th>\n    </tr>\n    <tr>\n      <th>npartitions=4</th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n      <td>float64</td>\n    </tr>\n    <tr>\n      <th>1949</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>3898</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>5847</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>7793</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n<div>Dask Name: truediv, 4 tasks</div>",
      "text/plain": "Dask DataFrame Structure:\n                item_0   item_1   item_2   item_3   item_4   item_5    kills   deaths  assists     gold last_hits   denies gold_per_min xp_per_min gold_spent hero_damage tower_damage hero_healing    level    stuns    pings gold_reasons_0 gold_reasons_1 gold_reasons_6 gold_reasons_11 gold_reasons_12 gold_reasons_13 gold_reasons_14 gold_reasons_15 gold_reasons_2 gold_reasons_5 xp_reasons_0 xp_reasons_1 xp_reasons_2 xp_reasons_3 neutral_damage_taken damage_to_minion   rune_3   rune_5   rune_2   rune_4   rune_6   rune_1   rune_0 kill_streak_3 kill_streak_4 kill_streak_5 kill_streak_6 kill_streak_7 kill_streak_8 kill_streak_9 kill_streak_10 multi_kill_2 multi_kill_3 multi_kill_4 multi_kill_5\nnpartitions=4                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n0              float64  float64  float64  float64  float64  float64  float64  float64  float64  float64   float64  float64      float64    float64    float64     float64      float64      float64  float64  float64  float64        float64        float64        float64         float64         float64         float64         float64         float64        float64        float64      float64      float64      float64      float64              float64          float64  float64  float64  float64  float64  float64  float64  float64       float64       float64       float64       float64       float64       float64       float64        float64      float64      float64      float64      float64\n1949               ...      ...      ...      ...      ...      ...      ...      ...      ...      ...       ...      ...          ...        ...        ...         ...          ...          ...      ...      ...      ...            ...            ...            ...             ...             ...             ...             ...             ...            ...            ...          ...          ...          ...          ...                  ...              ...      ...      ...      ...      ...      ...      ...      ...           ...           ...           ...           ...           ...           ...           ...            ...          ...          ...          ...          ...\n3898               ...      ...      ...      ...      ...      ...      ...      ...      ...      ...       ...      ...          ...        ...        ...         ...          ...          ...      ...      ...      ...            ...            ...            ...             ...             ...             ...             ...             ...            ...            ...          ...          ...          ...          ...                  ...              ...      ...      ...      ...      ...      ...      ...      ...           ...           ...           ...           ...           ...           ...           ...            ...          ...          ...          ...          ...\n5847               ...      ...      ...      ...      ...      ...      ...      ...      ...      ...       ...      ...          ...        ...        ...         ...          ...          ...      ...      ...      ...            ...            ...            ...             ...             ...             ...             ...             ...            ...            ...          ...          ...          ...          ...                  ...              ...      ...      ...      ...      ...      ...      ...      ...           ...           ...           ...           ...           ...           ...           ...            ...          ...          ...          ...          ...\n7793               ...      ...      ...      ...      ...      ...      ...      ...      ...      ...       ...      ...          ...        ...        ...         ...          ...          ...      ...      ...      ...            ...            ...            ...             ...             ...             ...             ...             ...            ...            ...          ...          ...          ...          ...                  ...              ...      ...      ...      ...      ...      ...      ...      ...           ...           ...           ...           ...           ...           ...           ...            ...          ...          ...          ...          ...\nDask Name: truediv, 4 tasks"
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler = StandardScaler()\n",
    "X_std = scaler.fit_transform(\n",
    "    skilled_player_stats.drop(\n",
    "        columns=[\"match_id\", \"player_slot\", \"hero_id\"]))\n",
    "X_std.persist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### K-Means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, finally onto some clustering. First with K-means."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the dimensionality isn't that bad, I will skip straight to the clustering. I will do 5 clusters, since there are 5 defined roles in this type of game."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Without Dask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "2.69 s ± 226 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
    }
   ],
   "source": [
    "kmeans = KMeans(n_clusters=5, n_jobs=-1)\n",
    "\n",
    "%timeit kmeans.fit(X_std)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = kmeans.predict(X_std)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.08549047511180916\n",
      "0.08549047511180916\n",
      "0.08549047511180916\n",
      "0.08549047511180916\n",
      "0.08549047511180916\n",
      "0.08549047511180916\n",
      "0.08549047511180916\n",
      "0.08549047511180916\n",
      "1.26 s ± 64.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit print(metrics.silhouette_score(X_std, y_pred, metric='euclidean'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With Dask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "9.26 s ± 2.83 s per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
    }
   ],
   "source": [
    "from dask_ml.cluster import KMeans as dask_KMeans\n",
    "\n",
    "kmeans = dask_KMeans(5)\n",
    "%timeit kmeans.fit(X_std)"
   ]
  },
  {
   "cell_type": "markdown",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Very strange, more than 3x slower than sklearn's KMeans with n_jobs=-1. I'll try and see if any improvement is found with using sklearn KMeans with joblib.parallel_backend(\"dask\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": "674 ms ± 31 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
    }
   ],
   "source": [
    "kmeans = KMeans(n_clusters=5)\n",
    "with joblib.parallel_backend(\"dask\"):\n",
    "    %timeit kmeans.fit(X_std)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Impressive. Completes ~3.72x faster than using n_jobs=-1 without Dask. I wonder why anyone even bothers with dask_ml, if it is so slow compared to using joblib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.08549047511180108\n",
      "0.08549047511180108\n",
      "0.08549047511180108\n",
      "0.08549047511180108\n",
      "0.08549047511180108\n",
      "0.08549047511180108\n",
      "0.08549047511180108\n",
      "0.08549047511180108\n",
      "2.17 s ± 59.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "with joblib.parallel_backend(\"dask\"):    \n",
    "    %timeit print(metrics.silhouette_score(X_std, y_pred, metric='euclidean'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, now I know that printing metrics does not lend itself well to Dask, I will not do it again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### t-SNE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Without Dask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20.8 s ± 528 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "tsne = TSNE(n_components=2, perplexity=40, n_iter=300, verbose=0)\n",
    "%timeit tsne_results = pd.DataFrame(tsne.fit_transform(X_std))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With Dask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18.6 s ± 697 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "with joblib.parallel_backend(\"dask\"):\n",
    "    %timeit tsne_results = pd.DataFrame(tsne.fit_transform(X_std))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not nearly as impressive of an improvement like with KMeans, but still significant enough, and perhaps will improve with a larger data and worker pool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### UMAP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Without Dask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15.1 s ± 117 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "umap = UMAP(n_neighbors=15, min_dist=0.6)\n",
    "%timeit umap_results = pd.DataFrame(umap.fit_transform(X_std))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In terms of pure speed, K-Means wins, but I am not satisfied with the results. It mistook the far off outlier group discovered by UMAP to be a part of the closer lower part of the main cluster that is just starting to break away."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With Dask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14.7 s ± 344 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "with joblib.parallel_backend(\"dask\"):\n",
    "    %timeit umap_results = pd.DataFrame(umap.fit_transform(X_std))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not worth doing with Dask. I saw no activity on my dashboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Agglomerative"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Without Dask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.56 s ± 77.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "agg_cluster = AgglomerativeClustering(n_clusters=5)\n",
    "\n",
    "%timeit y_pred = agg_cluster.fit_predict(X_std)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.022524470671924212\n"
     ]
    }
   ],
   "source": [
    "print(metrics.silhouette_score(X_std, y_pred, metric='euclidean'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With Dask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.34 s ± 93.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "with joblib.parallel_backend(\"dask\"):\n",
    "    %timeit y_pred = agg_cluster.fit_predict(X_std)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, does not seem to provide an improvement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HDBSCAN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Without Dask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.4 s ± 93.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "hdbscan = HDBSCAN()\n",
    "\n",
    "%timeit y_pred = hdbscan.fit_predict(X_std)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After seeing the visualizations for UMAP, I really liked HDBSCAN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With Dask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.34 s ± 107 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "with joblib.parallel_backend(\"dask\"):\n",
    "    %timeit y_pred = hdbscan.fit_predict(X_std)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, another example of a model not being worth it in Dask."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll cut it short here, as the remainder of my Capstone involves tasks that do not benefit from Dask as I continued forward with UMAP fed into Agglomerative Clustering for visualization, both of which did not benefit from Dask. However, Dask was extremely useful in terms of reading in the data without the clumsy, irreproducible approach I employed at first."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.5 64-bit ('Anaconda3': conda)",
   "language": "python",
   "name": "python37564bitanaconda3conda6437deb97293482a8080b30b8c3837a6"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}