Application 1 - ProvStore Documents.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Application 1: ProvStore Documents\n",
    "Identifying owners of provenance documents from their provenance network metrics\n",
    "\n",
    "* **Goal**: To determine if the provenance network analytics method can identify the owner of a provenance document from its provenance network metrics.\n",
    "* **Training data**: In order to ensure that there are sufficient samples to represent a user's provenance documents the Training phase, we limit our experiment to users who have at least 20 documents. There are fourteen such users (the authors were excluded to avoid bias), who we named $u_{1}, u_{2}, \\ldots, u_{14}$. Their numbers of documents range between 21 and 6,745, with the total number of documents in the data set is 13,870.\n",
    "* **Classification labels**: $\\mathcal{L} = \\left\\{ u_1, u_2, \\ldots, u_{14} \\right\\} $, where $l_{x} = u_i$ if the provenance document $x$ belongs to user $u_i$. Hence, there are 14 labels in total.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading data\n",
    "For each provenance document, we calculate the 22 provenance network metrics. The dataset provided contains those metrics values for 13,870 provenance documents along with the owner identifier (i.e. $u_{1}, u_{2}, \\ldots, u_{14}$)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>entities</th>\n",
       "      <th>agents</th>\n",
       "      <th>activities</th>\n",
       "      <th>nodes</th>\n",
       "      <th>edges</th>\n",
       "      <th>diameter</th>\n",
       "      <th>assortativity</th>\n",
       "      <th>acc</th>\n",
       "      <th>acc_e</th>\n",
       "      <th>...</th>\n",
       "      <th>mfd_e_a</th>\n",
       "      <th>mfd_e_ag</th>\n",
       "      <th>mfd_a_e</th>\n",
       "      <th>mfd_a_a</th>\n",
       "      <th>mfd_a_ag</th>\n",
       "      <th>mfd_ag_e</th>\n",
       "      <th>mfd_ag_a</th>\n",
       "      <th>mfd_ag_ag</th>\n",
       "      <th>mfd_der</th>\n",
       "      <th>powerlaw_alpha</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>u_3</td>\n",
       "      <td>17</td>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>31</td>\n",
       "      <td>49</td>\n",
       "      <td>6</td>\n",
       "      <td>-0.196362</td>\n",
       "      <td>0.444709</td>\n",
       "      <td>0.466667</td>\n",
       "      <td>...</td>\n",
       "      <td>5</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>u_2</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>u_2</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>u_2</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>u_2</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  label  entities  agents  activities  nodes  edges  diameter  assortativity  \\\n",
       "0   u_3        17       5           9     31     49         6      -0.196362   \n",
       "1   u_2         7       0           2      9      0        -1      -1.000000   \n",
       "2   u_2         7       0           2      9      0        -1      -1.000000   \n",
       "3   u_2         7       0           2      9      0        -1      -1.000000   \n",
       "4   u_2         7       0           2      9      0        -1      -1.000000   \n",
       "\n",
       "        acc     acc_e       ...        mfd_e_a  mfd_e_ag  mfd_a_e  mfd_a_a  \\\n",
       "0  0.444709  0.466667       ...              5         8        4        2   \n",
       "1  0.000000  0.000000       ...              0         0        0        0   \n",
       "2  0.000000  0.000000       ...              0         0        0        0   \n",
       "3  0.000000  0.000000       ...              0         0        0        0   \n",
       "4  0.000000  0.000000       ...              0         0        0        0   \n",
       "\n",
       "   mfd_a_ag  mfd_ag_e  mfd_ag_a  mfd_ag_ag  mfd_der  powerlaw_alpha  \n",
       "0         5         0         0          0        3            -1.0  \n",
       "1         0         0         0          0       -1            -1.0  \n",
       "2         0         0         0          0       -1            -1.0  \n",
       "3         0         0         0          0       -1            -1.0  \n",
       "4         0         0         0          0       -1            -1.0  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"provstore/data.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>entities</th>\n",
       "      <th>agents</th>\n",
       "      <th>activities</th>\n",
       "      <th>nodes</th>\n",
       "      <th>edges</th>\n",
       "      <th>diameter</th>\n",
       "      <th>assortativity</th>\n",
       "      <th>acc</th>\n",
       "      <th>acc_e</th>\n",
       "      <th>acc_a</th>\n",
       "      <th>...</th>\n",
       "      <th>mfd_e_a</th>\n",
       "      <th>mfd_e_ag</th>\n",
       "      <th>mfd_a_e</th>\n",
       "      <th>mfd_a_a</th>\n",
       "      <th>mfd_a_ag</th>\n",
       "      <th>mfd_ag_e</th>\n",
       "      <th>mfd_ag_a</th>\n",
       "      <th>mfd_ag_ag</th>\n",
       "      <th>mfd_der</th>\n",
       "      <th>powerlaw_alpha</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.00000</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.000000</td>\n",
       "      <td>13870.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>9.913338</td>\n",
       "      <td>2.08695</td>\n",
       "      <td>1.836193</td>\n",
       "      <td>13.836482</td>\n",
       "      <td>19.212689</td>\n",
       "      <td>0.868926</td>\n",
       "      <td>-0.628690</td>\n",
       "      <td>0.347835</td>\n",
       "      <td>0.341142</td>\n",
       "      <td>0.323606</td>\n",
       "      <td>...</td>\n",
       "      <td>1.312761</td>\n",
       "      <td>1.754939</td>\n",
       "      <td>1.073540</td>\n",
       "      <td>0.709229</td>\n",
       "      <td>0.752127</td>\n",
       "      <td>0.017448</td>\n",
       "      <td>0.014924</td>\n",
       "      <td>0.030353</td>\n",
       "      <td>2.185436</td>\n",
       "      <td>-0.916534</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>28.931915</td>\n",
       "      <td>2.27716</td>\n",
       "      <td>18.570823</td>\n",
       "      <td>43.352894</td>\n",
       "      <td>134.640366</td>\n",
       "      <td>1.943905</td>\n",
       "      <td>0.376718</td>\n",
       "      <td>0.394531</td>\n",
       "      <td>0.409577</td>\n",
       "      <td>0.395727</td>\n",
       "      <td>...</td>\n",
       "      <td>1.769329</td>\n",
       "      <td>1.314874</td>\n",
       "      <td>1.622606</td>\n",
       "      <td>1.343363</td>\n",
       "      <td>1.077628</td>\n",
       "      <td>0.200902</td>\n",
       "      <td>0.152351</td>\n",
       "      <td>0.209759</td>\n",
       "      <td>5.211118</td>\n",
       "      <td>0.612437</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>4.000000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.592949</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>5.000000</td>\n",
       "      <td>3.00000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>13.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>-0.350000</td>\n",
       "      <td>0.674147</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>...</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>-1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1188.000000</td>\n",
       "      <td>51.00000</td>\n",
       "      <td>1580.000000</td>\n",
       "      <td>2776.000000</td>\n",
       "      <td>6853.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>52.000000</td>\n",
       "      <td>44.000000</td>\n",
       "      <td>51.000000</td>\n",
       "      <td>52.000000</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>8.184413</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           entities       agents    activities         nodes         edges  \\\n",
       "count  13870.000000  13870.00000  13870.000000  13870.000000  13870.000000   \n",
       "mean       9.913338      2.08695      1.836193     13.836482     19.212689   \n",
       "std       28.931915      2.27716     18.570823     43.352894    134.640366   \n",
       "min        0.000000      0.00000      0.000000      1.000000      0.000000   \n",
       "25%        2.000000      1.00000      0.000000      5.000000      5.000000   \n",
       "50%        4.000000      1.00000      1.000000      7.000000      9.000000   \n",
       "75%        5.000000      3.00000      2.000000     10.000000     13.000000   \n",
       "max     1188.000000     51.00000   1580.000000   2776.000000   6853.000000   \n",
       "\n",
       "           diameter  assortativity           acc         acc_e         acc_a  \\\n",
       "count  13870.000000   13870.000000  13870.000000  13870.000000  13870.000000   \n",
       "mean       0.868926      -0.628690      0.347835      0.341142      0.323606   \n",
       "std        1.943905       0.376718      0.394531      0.409577      0.395727   \n",
       "min       -1.000000      -1.000000      0.000000      0.000000      0.000000   \n",
       "25%       -1.000000      -1.000000      0.000000      0.000000      0.000000   \n",
       "50%        1.000000      -0.592949      0.000000      0.000000      0.000000   \n",
       "75%        2.000000      -0.350000      0.674147      0.750000      0.666667   \n",
       "max       10.000000       1.000000      1.000000      1.000000      1.000000   \n",
       "\n",
       "            ...             mfd_e_a      mfd_e_ag       mfd_a_e       mfd_a_a  \\\n",
       "count       ...        13870.000000  13870.000000  13870.000000  13870.000000   \n",
       "mean        ...            1.312761      1.754939      1.073540      0.709229   \n",
       "std         ...            1.769329      1.314874      1.622606      1.343363   \n",
       "min         ...            0.000000      0.000000      0.000000      0.000000   \n",
       "25%         ...            0.000000      1.000000      0.000000      0.000000   \n",
       "50%         ...            1.000000      2.000000      0.000000      0.000000   \n",
       "75%         ...            2.000000      2.000000      2.000000      1.000000   \n",
       "max         ...           52.000000     44.000000     51.000000     52.000000   \n",
       "\n",
       "           mfd_a_ag      mfd_ag_e      mfd_ag_a     mfd_ag_ag       mfd_der  \\\n",
       "count  13870.000000  13870.000000  13870.000000  13870.000000  13870.000000   \n",
       "mean       0.752127      0.017448      0.014924      0.030353      2.185436   \n",
       "std        1.077628      0.200902      0.152351      0.209759      5.211118   \n",
       "min        0.000000      0.000000      0.000000      0.000000     -1.000000   \n",
       "25%        0.000000      0.000000      0.000000      0.000000      1.000000   \n",
       "50%        1.000000      0.000000      0.000000      0.000000      2.000000   \n",
       "75%        1.000000      0.000000      0.000000      0.000000      2.000000   \n",
       "max       43.000000      4.000000      5.000000      6.000000    303.000000   \n",
       "\n",
       "       powerlaw_alpha  \n",
       "count    13870.000000  \n",
       "mean        -0.916534  \n",
       "std          0.612437  \n",
       "min         -1.000000  \n",
       "25%         -1.000000  \n",
       "50%         -1.000000  \n",
       "75%         -1.000000  \n",
       "max          8.184413  \n",
       "\n",
       "[8 rows x 22 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u_3     6745\n",
       "u_8     4449\n",
       "u_5     1327\n",
       "u_2      487\n",
       "u_12     312\n",
       "u_14     150\n",
       "u_9      141\n",
       "u_6       71\n",
       "u_7       66\n",
       "u_4       34\n",
       "u_1       25\n",
       "u_13      21\n",
       "u_10      21\n",
       "u_11      21\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The number of each label in the dataset\n",
    "df.label.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from analytics import balance_smote, test_classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Balancing the data**\n",
    "\n",
    "With an unbalanced like the above, the resulted trained classifier will typically be skewed towards the majority labels. In order to mitigate this, we balance the dataset using the [SMOTE Oversampling Method](https://www.jair.org/media/953/live-953-2037-jair.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original data shapes: (13870, 22) (13870,)\n",
      "Balanced data shapes: (94430, 22) (94430,)\n"
     ]
    }
   ],
   "source": [
    "df = balance_smote(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Cross Validation tests**: We now run the cross validation tests on the dataset (`df`) using all the features (`combined`), only the generic network metrics (`generic`), and only the provenance-specific network metrics (`provenance`). Please refer to [Cross Validation Code.ipynb](Cross%20Validation%20Code.ipynb) for the detailed description of the cross validation code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 98.13% ±0.0080 <-- combined\n",
      "Accuracy: 92.32% ±0.0157 <-- generic\n",
      "Accuracy: 98.11% ±0.0079 <-- provenance\n"
     ]
    }
   ],
   "source": [
    "results, importances = test_classification(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Result**: The outputs above is the accuracy of the classifier in identifying the owner of a provenance document from ProvStore using all provenance network metrics (i.e. `combined`), only generic network metrics, and only provenance-specific network metrics.\n",
    "\n",
    "The individual accuracy scores are stored in `results` and the importance of every feature in each test in `imps` (both are pandas Dataframe objects)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving experiments' results (optional)\n",
    "\n",
    "Optionally, we can save the test results to save time the next time we want to re-explore them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "results.to_pickle(\"provstore/results.pkl\")\n",
    "importances.to_pickle(\"provstore/importances.pkl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next time, we can reload the results as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((3000, 2), (1000, 22))"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "results = pd.read_pickle(\"provstore/results.pkl\")\n",
    "importances = pd.read_pickle(\"provstore/importances.pkl\")\n",
    "results.shape, importances.shape  # showing the shape of the data (for checking)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The importance of features\n",
    "\n",
    "We plot the importance of each feature (from the `combined` test, which used all 22 features)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "sns.set_style(\"whitegrid\")\n",
    "sns.set_context(\"talk\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Rename the columns with Math notation for consistency with the metrics symbols in the paper\n",
    "feature_name_maths_mapping = {\n",
    "    \"entities\": \"$n_e$\", \"agents\": \"$n_{ag}$\", \"activities\": \"$n_a$\", \"nodes\": \"$n$\", \"edges\": \"$e$\",\n",
    "    \"diameter\": \"$d$\", \"assortativity\": \"$r$\", \"acc\": \"$\\\\mathsf{ACC}$\",\n",
    "    \"acc_e\": \"$\\\\mathsf{ACC}_e$\",  \"acc_a\": \"$\\\\mathsf{ACC}_a$\",  \"acc_ag\": \"$\\\\mathsf{ACC}_{ag}$\",\n",
    "    \"mfd_e_e\": \"$\\\\mathrm{mfd}_{e \\\\rightarrow e}$\", \"mfd_e_a\": \"$\\\\mathrm{mfd}_{e \\\\rightarrow a}$\",\n",
    "    \"mfd_e_ag\": \"$\\\\mathrm{mfd}_{e \\\\rightarrow ag}$\", \"mfd_a_e\": \"$\\\\mathrm{mfd}_{a \\\\rightarrow e}$\",\n",
    "    \"mfd_a_a\": \"$\\\\mathrm{mfd}_{a \\\\rightarrow a}$\", \"mfd_a_ag\": \"$\\\\mathrm{mfd}_{a \\\\rightarrow ag}$\",\n",
    "    \"mfd_ag_e\": \"$\\\\mathrm{mfd}_{ag \\\\rightarrow e}$\", \"mfd_ag_a\": \"$\\\\mathrm{mfd}_{ag \\\\rightarrow a}$\",\n",
    "    \"mfd_ag_ag\": \"$\\\\mathrm{mfd}_{ag \\\\rightarrow ag}$\", \"mfd_der\": \"$\\\\mathrm{mfd}_\\\\mathit{der}$\", \"powerlaw_alpha\": \"$\\\\alpha$\"\n",
    "}\n",
    "importances.rename(columns=feature_name_maths_mapping, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEuCAYAAABmlhI1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XtYVPW6B/DvDIIzCGpFUj6WIp0iTUMwDMMbhe5toyZB\nulEwPaZQJOk2o067yC6Wx+0NE9tImxR3XmgrKqaZKZqE11OKZI/KqKAFEYUKM4Cwzh9uRkcuM2vW\nDIthfT/P4/Mwa37v/N41M7yMv3nXWipBEAQQEZGiqOVOgIiIWh+LPxGRArH4ExEpEIs/EZECsfgT\nESkQiz8RkQJ1kDsBaxw7dkzuFIiInFJgYGCT260q/gUFBXjrrbdw9uxZ9OzZE++88w78/f2bHf/e\ne+/B1dUVr732mmnb119/jWXLluHSpUu499578corryAsLEzyDhARUdNa+uBscdmnuroasbGxCA8P\nx5EjRxAdHY24uDhUVlY2Gvv7778jMTERa9euNduu1+sxb948vPHGGzh27Bhef/11zJs3D+fOnbNh\nd4iISCqLxT8vLw9qtRpRUVFwdXVFREQEvLy8kJOT02hsVFQUXFxcMGrUKLPtly5dwnPPPYfg4GCo\nVCqEhITAx8cHJ0+etN+eEBGR1Swu++j1evj6+ppt8/HxQWFhYaOx6enp8Pb2RmJiotn2kJAQhISE\nmG4XFRXhzJkz8PPzszVvIiKSwGLxr6qqglarNdum0WhgNBobjfX29rY4YUlJCV544QWMHz9eVPFv\naj4iIrKNxeKv1WobFV6j0Qh3d3fRkxUUFCA2NhbDhw9HUlKSqNhTp06Jno+IiJpmsfj37t0bGRkZ\nZtv0ej10Op2oifbv34/Zs2fjpZdewrRp08RlCaBv376iY4iIlKylD80Wi39wcDBqamqwdu1aTJw4\nEVlZWSgrKzNbw7fkzJkzmDVrFt5//308/fTTVsfdSqPRNNqWm5uLJUuWAABmz56NwYMH2/TYRERK\nY7Hbx83NDampqcjOzkZQUBAyMjKQkpICd3d3TJ8+HatWrbI4yZo1a2A0GvHmm29iwIABpn8bNmyw\nOXFBEJCcnIzy8nKUl5cjOTkZvDQBEZF1VM5wMZdjx441OsjLYDA0Wnravn17oy+niYiUqqna2cAp\nTu/Q4NeUm989CIKAu7Sd8JvhxsFmd2k74eo/M3FNpTKNuTtucqvnSETkDJz2xG4qlQrTBgSjq0aL\nrhotpg24cQAZERFZ5lSf/G8X2P0+BHafKHcaREROx2k/+RMRke1Y/ImIFIjFn4hIgVj8iYgUiMWf\niEiBWPyJiBSIxZ+ISIFY/ImIFIjFn4hIgVj8iYgUiMWfqJ3Jzc1FZGQkIiMjkZubK3c61Eax+BO1\nI7zOBVnLqU/sRkTA+C/2mX4WamtwtbTUdLu0tBTjN+yGytXNtG3zs8NbMTtqq/jJn6g96eAKlUdn\n002VR2egg6uMCVFbxeJP1I6oVCpohv4ZKncPqNw9bvzM61xQE7jsQ9TOuPZ6EK7PPyh3GtTG8ZM/\nEZECsfgTESkQiz8RkQKx+BMRKRCLPxGRArH4ExEpEIs/EZECWVX8CwoKEBERAX9/f4wbNw7ff/99\ni+Pfe+89fPTRR2bbcnNzodPp4O/vj6ioKOj1etuzJiIiSSwW/+rqasTGxiI8PBxHjhxBdHQ04uLi\nUFlZ2Wjs77//jsTERKxdu9Zse1lZGeLj4zFnzhwcPnwYgwcPRnx8PE84RUQkE4vFPy8vD2q1GlFR\nUXB1dUVERAS8vLyQk5PTaGxUVBRcXFwwatQos+1fffUVHn74YYSGhsLNzQ1xcXEoLS3FyZMn7bcn\nRERkNYund9Dr9fD19TXb5uPjg8LCwkZj09PT4e3tjcTERLPthYWFZo/h4uKC++67D4WFhejfv79V\niRqNRqvGSY0hau/4e0GAFcW/qqoKWq3WbJtGo2nyDeTt7d3kYxgMBnh4eJht02q1MBgMVid66tQp\n3G/16JsxRGSOvxcEWFH8tVpto0JvNBrh7u5u9SRNPYbBYBD1GH379sXVwz9aPb4hhqjdO58najh/\nL5SjpT/0Fot/7969kZGRYbZNr9dDp9NZnUDv3r2xc+dO0+26ujpcvHgRDzzwgNWPodFocNXq0Tdj\niMgcfy8IsOIL3+DgYNTU1GDt2rWora1FZmYmysrKEBISYvUkYWFhyM/Px1dffYWamhqkpKTgnnvu\nQZ8+fSQlT0REtrFY/N3c3JCamors7GwEBQUhIyMDKSkpcHd3x/Tp07Fq1SqLk9x9991YuXIlVqxY\ngUGDBiE3NxfJycm8yAQRkUysupiLn58f1q9f32j76tWrmxz/4YcfNtr2+OOPY+vWrSLTIyIiR+Dp\nHYiIFIjFn4hIgVj8iYgUiMWfiEiBWPyJiBSIxZ+ISIFY/ImIFIjFn4hIgVj8iYgUiMWfiEiBWPyJ\niBSIxZ+ISIFY/ImIFIjFn4hIgVj8iYgUiMWfiEiBWPyJiBSIxZ+ISIFY/ImIFIjFn4hIgVj8iYgU\niMWfiEiBWPyJiBSIxZ9aRW5uLiIjIxEZGYnc3Fy50yFSPBZ/cjhBEJCcnIzy8nKUl5cjOTkZgiDI\nnRaRollV/AsKChAREQF/f3+MGzcO33//fZPj0tPTMWTIEAQEBGDu3Lmoqqoy3bdp0yY8+eSTCAwM\nxMSJE5Gfn2+fPaA2z2g0orS01HS7tLQURqNRxoyIyGLxr66uRmxsLMLDw3HkyBFER0cjLi4OlZWV\nZuP27t2LtLQ0rFmzBjk5OaioqMDChQsBAKdPn8aiRYuwevVqHDlyBKGhoUhISHDMHhERkUUWi39e\nXh7UajWioqLg6uqKiIgIeHl5IScnx2xcVlYWIiIi4OPjA09PTyQkJCArKwt1dXW4cOEC6uvrUVdX\nB0EQoFarodFoHLZTRETUsg6WBuj1evj6+ppt8/HxQWFhodm2wsJChIWFmY2pqqpCSUkJQkJC0KtX\nLzz99NNwcXFBp06dsGbNGlGJ2rJMwKUFebyR/YzZ7braxuv7iVnPwMVVZbr9wdNbHJ4X3cDfCwKs\nKP5VVVXQarVm2zQaTaM3kMFgMPs03xDTsP2BBx7AW2+9hQcffBCpqamIj49Hdna21f8DOHXqFO63\naqR5DDkHvlath881AVYUf61W26jQG41GuLu7m23TaDSorq423TYYDACATp06YcWKFbjnnnvQr18/\nAMBLL72EjRs3Ijc3F6GhoVYl2rdvX1w9/KNVY2+NIRmcN7+p7gC4egC1127cdvW4se1WfK0kOJ8n\najifa+Vo6Q+9xeLfu3dvZGRkmG3T6/XQ6XRm23x9fc2WgvR6PTp37oxu3brh8uXLZv97UKlUcHFx\ngYuLi9U7odFocNXq0TdjSH4qlQo9hrqgaF8dAKDHUBeoVCqzMXytWg+fawKs+MI3ODgYNTU1WLt2\nLWpra5GZmYmysjKEhISYjRs7diw2bNiAM2fO4Nq1a1i+fDl0Oh3UajWGDx+OzMxMnDp1CtevX8c/\n//lP1NXVITAw0GE7ZgkPOmpdnXup0fd5V/R93hWde/HwEiK5Wfzk7+bmhtTUVCQlJWHx4sXo2bMn\nUlJS4O7ujunTp2PgwIGIjY1FaGgoiouLMXPmTFy5cgXDhg3DvHnzAAATJkzAlStX8PLLL+PKlSt4\n+OGHsXr1anh4eDh8B5ty60FHAJCcnIzg4OBGn0aJiNori8UfAPz8/LB+/fpG21evXm12OyYmBjEx\nMY3GqVQqzJgxAzNmzLAxTelKUj40/Wy8fr3RQUcXV7wPTYebT4d3XGKr5kdE1JoU+f/vji4uuEt7\nc93zLq0GHUV8/0BE5OwUWfxVKhWmDvBDV40bumrcMHWAH5d8iEhRrFr2aY8Cut+Nld2HyZ0GEZEs\nFPnJn4hI6Vj8iYgUiMWfiEiBWPyJiBSIxd8GPDqYiJwdi79IvCQhEbUHim31FONc8jjTz8brAkpL\nb17FrLS0FAVLx0HTwfw4Ad+Xs1otP2p/cnNzsWTJEgDA7NmzMXjwYJkzovaGn/xF6ugC3KG9Wejv\n0KrQkQcHkx3xf5fUGlj8RVKpVJj0aEd07qhC5443fubRwWRPvOA9tQYu+9ig/z0dsOjPfOpaixKW\nQMZm3lwmFAQBKg8PCNduXP1G5eGB57bvMvuQsTViXKPHIBKDFYzaNCWeflulUsFtyAjU5OwBALgN\nGdGu91cpSpYeFjXe+5UgB2VyA4s/tWnNLYHcfl3p9qZDLx906DVd7jSoHeOaP5ED8FgQautY/Ins\njN065Ay47ENtzp+zYk0/CzX1je4fvz0BKrebn1u+HLeqVfJqiS5znelnobYWxtuWqnTrP4PK1dUs\nZnvEpFbLj+h2/ORPZG8dOkDl0cl0U+XRCejAz1nUtrD4U9vmqgI8bimcHh1ubGvDVCoVXIcEA+5a\nwF0L1yHtuzuJnBM/jlCbplKp0GHE3bi+58YySocRdztFIXXpdT+0ve6XOw2iZrH4U5un9vGA23QP\nudMgale47ENEpEAs/kRECsTiT0SkQFYV/4KCAkRERMDf3x/jxo3D999/3+S49PR0DBkyBAEBAZg7\ndy6qqqpM9x09ehTjx4/HgAEDMGbMGHz33Xf22QMiIhLNYvGvrq5GbGwswsPDceTIEURHRyMuLg6V\nlZVm4/bu3Yu0tDSsWbMGOTk5qKiowMKFCwEAJSUliIuLQ2xsLI4fP46ZM2fi5Zdf5mlqiYhkYrH4\n5+XlQa1WIyoqCq6uroiIiICXlxdycnLMxmVlZSEiIgI+Pj7w9PREQkICsrKyUFdXh6ysLAwePBij\nRo2CSqWCTqfDZ599BrWaq05ERHKwWH31ej18fX3Ntvn4+KCwsNBsW2FhIR544AGzMVVVVSgpKcGp\nU6fg7e2Nl156CYMGDcKECRNQV1cHNzc3O+0GERGJYbHPv6qqqtHpczUaTaMlG4PBAI1GY7rdEGMw\nGFBRUYH9+/cjOTkZS5cuxcaNGzFjxgzs2rULXbp0sSpRW5aIpCwrSV2S4pKWOHK+VnKRa5+d9flS\nGke/ThaLv1arbZSE0WiEu7u72TaNRoPq6mrTbYPBAADo1KkT3NzcMHToUISEhAAAJk2ahLS0NBw/\nfhwjRoywKtFTp05B7PGSp06dMv3cQ0JsV5Gxt8eTZVKeL2d9ruXaZ2d9vpydlBrkCBaLf+/evZGR\nkWG2Ta/XQ6fTmW3z9fU1WwrS6/Xo3LkzunXrBh8fH1y8eNFsfH19vajT3Pbt2xdXD/9o9fiGmAYV\nh3fbHHvJhtOx3xqvOOfFh5g9X8USYuWiPy06xCxvvbidNos9n2d7LLWaigMnRI23x+vU0h8Qi8U/\nODgYNTU1WLt2LSZOnIisrCyUlZWZPsU3GDt2LN5++22MGjUK9957L5YvXw6dTge1Wo1x48ZhwoQJ\n2LdvH4YOHYp169ahuroagwYNsnonNBoNrlo9+mZMgwoJsbaQGq80Up4vZ32u5dpnZ32+nF1r1yBL\nLH7h6+bmhtTUVGRnZyMoKAgZGRlISUmBu7s7pk+fjlWrbpxLPTQ0FC+88AJmzpyJ4cOHw9PTE/Pm\nzQMA9OnTBykpKVi6dCkCAwOxefNmrFq1Cp06dWppaiIichCrTuzm5+eH9evXN9q+evVqs9sxMTGI\niYlp8jFCQkIa/W+BiIjkwUZ7IiIFYvEnIlIgFn8iIgVi8SciUiAWfyIiBWLxJyJSIBZ/IiIF4gXc\niZqRm5uLJUuWAABmz56NwYMHy5xR2/Lu5suixv9tfHcHZUK24Cd/oiYIgoDk5GSUl5ejvLwcycnJ\nos5FRdTWsfgTNcFoNKK0tNR0u7S0lKdCpnaFyz5E//H0F5+YfhYEAfBwB6795zrUHu54NvszqFQq\n05jsZ2e2dopEdsNP/kRNUKlUcB0aALhrAHcNXIcGmBV+ImfHT/5EzXDp1R0uz4+VOw0ih+AnfyIi\nBWLxJyJSIBZ/IiIFYvEnIlIgFn8iIgVi8SciUiAWfyIiBWLxJyJSIBZ/BcnNzUVkZCQiIyORm5sr\ndzpEJCMWf4XgWSqJ6FY8vUM79ln6SNPPtbUCSkvrTbdLS0uRtnokXF1vnq9myvNftWp+RCQffvJX\niA4dAHf3m7fd3W9sIyJlYvFXCJVKhccHqaHVAlot8PggNc9SSaRgVhX/goICREREwN/fH+PGjcP3\n33/f5Lj09HQMGTIEAQEBmDt3LqqqqhqN+e677+Dn54fKykppmZNoPXqoEBnhgsgIF/TowcJPpGQW\ni391dTViY2MRHh6OI0eOIDo6GnFxcY2K9969e5GWloY1a9YgJycHFRUVWLhwodmYiooKvPHGG/yi\nkYhIZhaLf15eHtRqNaKiouDq6oqIiAh4eXkhJyfHbFxWVhYiIiLg4+MDT09PJCQkICsrC3V1daYx\nSUlJGD16tP33goiIRLH4lZ9er4evr6/ZNh8fHxQWFpptKywsRFhYmNmYqqoqlJSUoHv37ti6dSuu\nXLmCv/71r1i9erXoRG25fqqUa65KvV6rM17vVc6c5Xyt5JpXrn121udLaRz9fFks/lVVVdBqtWbb\nNBpNo8QMBgM0Go3pdkOMwWDA5cuXsWzZMvzrX/9CbW2tTYmeOnUK99sQ06CHhNiuImNvj3cWcuYs\nZW658pY6r1z7bL/n616Z5nVOUmqQI1gs/lqttlGhNxqNcL+1bxA3/iBUV1ebbhsMBlP8a6+9htmz\nZ8Pb2xvFxcU2Jdq3b19cPfyj6JgGFYd32xx7yYaDYW+Nl0v+SXHj7ZbzefEhZnOLfIvYL++jts+r\nPy16OvN4cTttFns+z/ZYCXZcLJdlXmdVceCEqPH2eL5a+gNisfj37t0bGRkZZtv0ej10Op3ZNl9f\nX7OlIL1ej86dOwMAfvjhB/z4449ISkpCff2NA42GDRuGVatWYeDAgVbthEajwVWrRprHNKiQEGsL\nqfFykDNnKXPLlbec7xElPl/OrrVrkCUWv/ANDg5GTU0N1q5di9raWmRmZqKsrAwhISFm48aOHYsN\nGzbgzJkzuHbtGpYvXw6dTofu3bvjxIkTOHr0KI4ePYqtW7cCAHJycqwu/EREZF8Wi7+bmxtSU1OR\nnZ2NoKAgZGRkICUlBe7u7pg+fTpWrVoFAAgNDcULL7yAmTNnYvjw4fD09MS8efMcvgNERCSeVQf4\n+/n5Yf369Y223961ExMTg5iYmBYfq0ePHvjpp59EpEhERPbG0zsQESkQiz8RkQKx+BMRKRCLPxGR\nArH4ExEpEIs/EZECsfgTESkQiz8RkQKx+BMRKRCLPxGRArH4ExEpEIs/EZECsfgTESkQiz8RkQKx\n+BMRKRCLPxGRArH4ExEpEIs/EZECsfjLIDc3F5GRkYiMjERubm6rxRIRNWDxb2WCICA5ORnl5eUo\nLy9HcnIyBEFweCwR0a2suoA7SZP7D53p5+paAaWlRtPt0tJS7FupQ0dXlWnb4Bnbm3wco9GI0tJS\ns1ij0QitVuuArImoPWPxb2VuHYAu7ipUVN34xN7FXQW3Fl6F7Z/+2fRzTW3jT/lfrhkPt1v+cOim\nfWm/ZImo3eKyTytTqVQYE+gKDw3goQHGBLpCpVJZDgTg2gHwdL9529P9xjYiIrFYOmTwUHcXvDZO\n/FKNSqVC6GMdsPvQdQBA6GMdrP7DQUR0KxZ/J9O7hxoze7jJnQYROTmrln0KCgoQEREBf39/jBs3\nDt9//32T49LT0zFkyBAEBARg7ty5qKqqMt23ceNGjBw5EgEBAXj22Wdx9OhR++wBERGJZrH4V1dX\nIzY2FuHh4Thy5Aiio6MRFxeHyspKs3F79+5FWloa1qxZg5ycHFRUVGDhwoUAgLy8PCxevBjLli3D\n0aNHMXnyZMTGxuL33393zF4REVGLLBb/vLw8qNVqREVFwdXVFREREfDy8kJOTo7ZuKysLERERMDH\nxweenp5ISEhAVlYW6urq8Msvv+C///u/8fDDD0OtVmP8+PFwcXHB2bNnHbZjRETUPItr/nq9Hr6+\nvmbbfHx8UFhYaLatsLAQYWFhZmOqqqpQUlKCZ555xmzssWPHUFlZ2ehxW2I0Gi0PskOMPWLlnFvO\nfZbCGfNW4ntECjnfX87I0c+XxeJfVVXV6CAijUbTKDGDwQCNRmO63RBjMBjMxp09exazZs3CrFmz\ncOedd1qd6KlTp3C/1aNvxjToISG2q8jY2+OVECuVM+YtdV7nf53vlWle5ySlBjmCxeKv1WobFXqj\n0Qh3d3ezbRqNBtXV1abbDUW/U6dOpm3ffvstZs+ejalTp2LGjBmiEu3bty+uHv5RdEyDisO7bY69\nZMMpdG6NP37M9tiff7A9Nv+k7bGSnBcfYjZ3sYRYKc6La0Iwm1d/WvR05vHidtos9nye7bES7LhY\nLsu8zqriwAlR4+3xfLX0B8Ri8e/duzcyMjLMtun1euh0OrNtvr6+ZktBer0enTt3Rrdu3QAAX3zx\nBd5//33Mnz+/Uaw1NBoNrtoQ06BCQqwtpMQ7Y6xUzpi3Et8jUsj5/moLWrsGWWLxC9/g4GDU1NRg\n7dq1qK2tRWZmJsrKyhASEmI2buzYsdiwYQPOnDmDa9euYfny5dDpdFCr1fjuu+/wzjvv4B//+IdN\nhZ+IiOzL4id/Nzc3pKamIikpCYsXL0bPnj2RkpICd3d3TJ8+HQMHDkRsbCxCQ0NRXFyMmTNn4sqV\nKxg2bBjmzZsHAEhNTUVtbS1eeOEFs8detmwZhg4d6pg9IyKiZll1hK+fnx/Wr1/faPvq1avNbsfE\nxCAmJqbRuE8//dTG9IioPdr4RZnomOee9XJAJsrFE7sRESkQiz8RkQKx+BMRKRCLPxGRArH4ExEp\nEIs/EZECsfiTVXJzcxEZGYnIyEjk5tpwvgsialNY/MkiQRCQnJyM8vJylJeXIzk5GYLQ+GLyROQ8\nWPzJIqPRiNLSUtPt0tJSnp6XyMnxGr7UpCX/GmX6+Xpt40/5KzaOQwfXmxePnx21q1XysmT05g9F\njd8xPtFBmTiHyC/yRY3f9OwjDsqEWhuLP1nk0gHQdAKM/7lyp6bTjW1Ezuj0yhJR4/1e9HZQJvLi\nsg9ZpFKp0G+wGh21QEct0G+wGiqVynIgEbVZ/PxGVvG+X42wKH5WIGov+NtMRKRALP5ERArE4k9E\npEAs/kRECsTiT0SkQOz2ISKy0i+LCkWNv2dubwdlIh0/+RMRKRCLPxGRArH4ExEpEIs/EZECsfgT\nESkQiz8RkQJZ1epZUFCAt956C2fPnkXPnj3xzjvvwN/fv9G49PR0pKWlobKyEqGhoZg/fz7c3d0B\nANu3b8eSJUvw22+/YdCgQXj//ffh5eVl370hImqnSlfsFDW+W/yfWrzf4if/6upqxMbGIjw8HEeO\nHEF0dDTi4uJQWVlpNm7v3r1IS0vDmjVrkJOTg4qKCixcuBAAcPr0abz99ttYvHgx8vLy4OXlhddf\nf13UjhARkf1YLP55eXlQq9WIioqCq6srIiIi4OXlhZycHLNxWVlZiIiIgI+PDzw9PZGQkICsrCzU\n1dVh27ZtePLJJ/Hoo49Co9Fg7ty5OHDgAMrKyhy2Y0RE1DyLxV+v18PX19dsm4+PDwoLzY90Kyws\nxAMPPGA2pqqqCiUlJY3uu+OOO9ClSxfo9Xqp+RMRkQ0srvlXVVVBq9WabdNoNI0u4G0wGKDRaEy3\nG2IMBkOj+xruNxgMVidqywXDpVxkXOoFyuWa2xlj5Zyb++wcsfaIl2Petvx8qQRBaHx17lv885//\nxMGDB7F69WrTtlmzZsHPzw8vvviiaduYMWMQFxeH0aNHAwAqKysREBCAnJwcJCUlISAgADNmzDCN\nHzRoED7++GMMHDjQ4k4cO3bM4hgiImosMDCwye0WP/n37t0bGRkZZtv0ej10Op3ZNl9fX7OlIL1e\nj86dO6Nbt27w9fU1W+IpLy9HRUVFo+UksckTEZFtLK75BwcHo6amBmvXrkVtbS0yMzNRVlaGkJAQ\ns3Fjx47Fhg0bcObMGVy7dg3Lly+HTqeDWq2GTqfDV199haNHj6K6uhqLFy/G0KFDcccddzhsx4iI\nqHkWl32AG62aSUlJ+Omnn9CzZ08kJSXB398f06dPx8CBAxEbGwsAWLNmDdLT03HlyhUMGzYM7733\nnmntf8eOHVi2bBl+/fVXDBw4EAsWLMBdd93l2L0jIqImWVX8iYiofeHpHYiIFIjFn4hIgVj8iYgU\niMWfiEiBWPyJiBTI6Yv/pUuXUFRUJHcaFtXV1SEnJwc5OTn4+eefRcefO3cOmzdvxu7du60af+LE\nCdTU1Iiep61yltf5dsy79ThjzmLZc/+cutXzww8/RHFxMS5evIjNmzdj165dptNLWPLll1+iT58+\n6Nmzp+h5lyxZgn379kEQBDz00EN45JFHMGXKlBZjXn75ZRw+fBgeHh64fPky7rzzTjzyyCN45JFH\n8PLLL7cY++9//xvvvfce+vfvjx9++AH/93//h/z8fNTV1eHRRx9tMiYwMBDV1dXw8fFBnz598PDD\nD6NPnz7o06cPPDw8LO5jXV0dRowYgT179sDV1dXi+FuVlZXh8OHDTb4WO3bswKBBg0Qd4yHldQZs\ne70aSHmfyJW3lJyl5u2Mz7VcOQPAL7/8gnvuucfq8WFhYRgxYgTi4uLMDpL98ssv8ec//1nU3E79\nyf/kyZNYsWIFunTpAhcXF2RmZlodu3//fkyZMgVjxoxBVFQUPvzwQ5w4ccKq2AMHDiArKwtdunTB\nyJEjcerUKYsx3377LbZu3Yo9e/bg8OHDWLRoEQYOHIhz585ZjP3444+Rnp6O9PR0dOhw44wcHTt2\nNF0voSnHjh3Dl19+iYSEBHh7e2PRokWIiYnBY489hrCwMMyaNQurVq1qNt7FxQUdO3ZEXV2dxfxu\nl5qaigsXLjR5X3FxMVJTU0U9npTXGbDt9Wog5X0iV95ScpaatzM+13LlDAD/8z//Y/VYANi8eTPO\nnTuHsLAwrFixArt378Znn32GN998U9TjAE5e/F1dXVFdXQ2VSgUAqK2ttTr2559/xjfffINt27Zh\n6tSpKC9yTVnCAAASjUlEQVQvR2pqKtavX28xtuEvrlqtRlhYmFUF0tvb2xTn6emJ4OBgvPDCC1i6\ndKnF2GvXrqF///4AYNrXXr164ezZsy3G3XfffXjqqadQVFSExMREnDhxArt370Z4eDgOHDiAgwcP\nthgfExNjdkI/a+Xk5CAyMrLJ+8LDw7F3715RjyfldQZse70aSHmfyJW3lJyl5u2Mz7VcOQPAQw89\nZNUHwAazZ89GUVERBg8ejIKCAsyZMwerV6/GBx98YPVjNHDq4j9t2jRMmzYNv/32G7Kzs0XFGo1G\n04scFhaGq1evYtmyZcjKyrIYO3ToUFRVVaFXr17YvHkzCgoKLMY8//zz+Pzzz0Xl2MDX1xeHDh0C\nADSs0nXo0AHXr1+3Kn7//v2YPHky3Nzc0KNHD8TFxVl1eo2PP/4Yq1atQmxsLLKzs1FcXGzVfGVl\nZc1eotPLy0v0RXykvM6Aba9XAynvE7nylpKz1Lyd8bmWK2cAOHToEKKiojB79mx89tlnFs9gfPz4\ncWzZsgXLly/HypUrsW3bNnTv3h2XL1+2OmcTwclduHBB+Pjjj4UlS5YIly9ftjpu586dwuTJk4Ut\nW7YIW7ZsEcaMGSMIgiBERUVZ/RhXrlwRFi5cKOzZs8fi2CeeeELo16+fMGfOHCE7O1soKiqyep4D\nBw4IgwcPFnbv3i089thjgiAIwp49ewSdTmdVfFhYmHDixAmzbbW1tcKQIUNajMvLyxPS0tKEOXPm\nCCNHjhT8/PyEQYMGCdOmTWsx7vHHHxdKSkqavK+kpEQICgqyKu9b2fo630rM69VA6vtEjrzt8d6W\nmrczPtdy5CwIgnD+/HkhPz9fyMzMFN59990Wx44ZM0Y4d+6c2bbff//dpt8ppyr+169fF/bt2yfs\n27fP5l+kWxUVFQkrV64UFi9eLJw7d06ora0VlixZ4pB5Dx48KHzyySdCQkKCEBYWZnUhbZCVlSUE\nBQUJfn5+Qnh4uODv7y9kZ2dbFZudnS2MGDFC2Ldvn1k+gwYNErUPV69eFfLy8oRPP/20xXEvvvii\n8Pe//73J+xYvXiy8+OKLouaVm7Xvk7bEGXMWBOfMW0rOCxYsEF566SVhzJgxwvXr14Xt27e3OH7L\nli1CaGiosGfPHuH69euCIAjCjz/+KAQHB4vO26m6fZrrmOnXrx/i4+NFPZaYb/ildOo059q1a8jP\nz8ePP/6IqVOnWhVjMBhw+PBh/P777+jXr5/V10MAbnQMffTRR6ivr8ddd92FoqIixMbG2px/S06e\nPIlJkyZh7NixePrpp+Ht7Y2SkhJkZ2dj27Zt+Ne//oW+ffs2ipPaJWTvLiNAWieIXJwxZ8A585aa\n86RJk7Bu3TpER0dj7dq1mDZtGj799NMWYz7//HMsXboUNTU1uPvuu3H58mVERUXhjTfeEJW7xYu5\ntCXffvstdu7cCW9vb1y9ehX5+fnIz88X9e18g4Zv+KOjozFy5MgW++ftOW8DDw8PPP7443j88cet\njtFqtRg2bJhN84WHh2P06NE4fPgwfvvtNzzwwAPo16+fTY9lSb9+/ZCSkoL58+cjMzMTKpUKgiCg\nZ8+eSElJabLwAze6hLp27drkfcXFxThx4gQSExObnVdqfFPEvE+aU1JSAm9vb9FxtsbbI2db5pUa\nK/dzLUfOtnxR/Ze//AXPPfccjh07hp9//hn33nsvHnvsMVHzAk72ha+UjpnbifmG357zykmj0WDo\n0KEYP368wwp/gyeeeAK7du3Czp07sW7dOuzcuRO7du3C4MGDm42R2iVk7y4jQFonCHDjS/OwsDDk\n5OSIntvWeKk52zqv1Fg5n2u5crb1i2oXFxcEBQVh3LhxCAoKMv3xEMOpir+UjpnbifmG357zKkFZ\nWRl27NgB4EZLakBAAHr16gXgxvLLb7/91myclC4he3cZAdI6Qb755hv88ccf8PHxwbp16/DLL7+I\nmtvWeCk5S81bSqxcz7VcOTfEL1iwAE8//TTOnDnT4rE7dif6WwIZSemYaY413/A7Yt727IMPPhBW\nrlzZ5H2ffPKJsGDBgibvk9ol5Iguowa2dILU19cLgiAIEyZMMP0shtR4W3KWOq/UnAWh9Z9rOXKe\nNm2asGjRIiE7O1vQ6/U2zSmVU635L1y4EPn5+SgoKMDSpUtRVFSELl26oG/fvkhLS2sxtq6uDt9+\n+y0A4MEHH8S9994L4MYyzquvvuqweZUoJycHGRkZTd4XHh6OSZMmNbn2HhAQgIyMDMyZM6fRfevW\nrcPAgQNbnFdqfEuseZ/c7tb/itvy33Kp8bbkLHVeqTkDrf9cy5HzokWLcPr0aezYsQOLFi2CVqtF\nly5d0KdPH5uO1rWFUxX/wYMHm60Z39oxY8krr7xic8eOlHmVyNbll9jYWEyaNAnl5eXNdgm1REq8\nIzqFiJpzxx13IDg4GKtWrcLXX38NtVqNrVu3Wn1wmD04VfG/nZiOGXt27NjSqaMkrq6uKC0tRbdu\n3RrdV1paajo/0e1s7RKyR7wjOoUaCBK7qW2Nl9plJCVvKbFS8na2nKurq3H9+nW4ublh7NixrVr8\nnWrNX4pRo0YJ1dXVcqehCPY4yEuv1wvHjh2zeT1UbPyoUaOEX3/9tcn7fv31V2HkyJE25SEIgrBx\n40abY22NX7FihdCvXz+zA/taY16psVLzdracv/76ayEmJkbYunWrkJmZKYwePdqmHGzhVAd5SbF+\n/XpUV1e3+YNG2gNnPMhr4MCBOHr0aLOPHRgYaPG8K23FN998g++++w6HDx+Gt7c35s+fL+q0wXJx\nxrztkfOlS5ewdetWGAwGjBs3TtTBm1IopviHhITgypUrCAsLw5NPPon+/fujR48ecqfVbh08eBDz\n58/HhQsXzJZf3n777WZ7/RcsWICuXbsiLi6u0X3/+Mc/UF5e3uLSi5T44OBgZGVlNbtUNWbMGNPJ\n9cS4dOkS6uvrcd9994mOtTVeEASoVCpMnDgRn3/+uc1fYrY2KXlLOa++lFhrcrb2pGvdu3cXPb8U\nTr3mLwY7dlpXw0Fe58+fR3l5Oe6880706tULZ86cwXvvvddkR4OtXUL2iHdEp5DUi7nYGi+1e0XK\nKQukxErJe//+/fjoo4/g6ekJT09P9O/fH6NHjzadCt1RsdbkHBoaatX+tHYDiWKKPzt25NGrVy90\n794dO3bsQGJiIn744Qf4+/s3OVbOg7ykdho15eTJk6bztjRcYERM8Zcabysppyyw16klxGo4r75a\nrcbu3buxe/dupKam4oknnsDEiRMdFmuNW48a3r9/PzZv3oz4+Hj06NEDxcXFWLlyJZ555hnJ84il\nmOJ/O3bsON7Zs2exfv16bNu2DQaDAfX19UhNTUVISEiT423tErJHvNROo+bykXIxF6nxtq7o3n7K\ngp07d7ZKbANb8m44r37DvP/+97/x8ccfY9KkSRYLuJRYa3K+tQsoLS0N69atM333dP/99+PBBx/E\n5MmTERERYdVc9qLY4k+Os2XLFmzcuBHHjx/HQw89hPj4eIwZMwY6nQ4PP/xws3FyH+TV3FKVrRrO\n2/LHH3/YdDEXqfG2FhMppyyQeroDwLa8p06dimnTppliL126BLVaDbXa8hlspMSKzbmsrAwajcZs\nm0ajsenUI1Ip5gtfaj1+fn7o2rUrPvroI7OzkIaEhCArK6vZjhtbu4TsEe+og7wuXryI7du3o6am\nBhMmTDAdWd5a8VJcvXoVq1atQmBgIEJDQ1st1lbFxcXYtm0bjEYjxo4di549e2LFihV45ZVXHBor\nRmxsLFxdXfHaa6+he/fuuHTpEv73f/8XNTU1LV5T2xFY/MnuUlJSsGnTJpSWlmLYsGF49tlnMXz4\ncAwdOrTF4g/Y1iVkj3hbO4WkdnI4qhPEUpeQlHnbaveKHMR2CpWVleGvf/0rDh06ZFrOCwoKwt//\n/vdmv69yFBZ/cghBELB//35s2rQJ+/btQ9euXXHlyhVs2LChxaWfBk11CW3YsMHq856Ijf/Tn/6E\njIyMJn8By8rKMGnSJOzatavRfX5+fpI6OaTGN8WaLiEp8zoiZ0C+LiMpsa+//jq+++470Z1CJSUl\npqOCpRyFLUlrHU1GylVSUiKsWLFCGDFihNCnTx9h1qxZVsVVV1cLmzdvFiZMmCD4+fkJEydOFDWv\nmPjAwMAWHysgIKDJ7b/88ovp38aNG4W//OUvwsGDB4ULFy4IBw8eFCZNmiRs2rSp2ceVGt+UhuvH\nTp48WRAEQZg6dapd53VEzoIgCOPHjzfl/dVXXwmvvvpqm4+dMmWKUFdXJwiCYIqNj48XPv/8c9OY\n8PBw08/JyclWP7ajsfhTq6mvrxf27dsnxMXFtTjuzJkzwrvvvisEBQUJ/fr1E/r27SscOHDA6nls\nibfH6aBHjRollJWVmW379ddfhVGjRlmVt9T4BlOmTBGMRqMQHR0tCMLNPwKOmNdeOQuCYLqedUxM\njCAIgjBnzpw2HzthwgShpqbGdDs2Nlaoq6sz+6ARGBhoOlX0gAEDrH5sR2O3D7UalUqFYcOGNXsp\nSlu7hOwRb4+DvKR2ctirE0Rsl5CUee3ZvSJXl5GUWGs6hR599FFMmjQJvr6+qKmpwd/+9rcmH+vd\nd9+1el574Jo/tRm2dgnZI15qpxEgvZPDnp0gYrqEpMzriO4VubqMbI211ClUUVGB9evXo6ioCFu2\nbMGYMWOafJwFCxaIylcqFn9qM6R0CdkjXmqnkdRODjHx9uy4kZK32Fi5uozaSodSbGxsq7d0NofF\nn9oUQWKXkNR4QHqnkdRODmviHdFxIyVva2Pl6jJqKx1KdXV1+PLLL5Gfn4/Kykqz+1p72Ydf+FKb\nZWuXkD3ire0UktrJYWu81I4bKXlLiZWry6itdCi9+eabQnBwsJCQkCAkJiaa/WttLP7U5lnbJWSP\neLGdQlI7OezRCWJLx42Uee3VvSJXl5GcHUoDBw4ULl++LHoeR2C3D7V5lrqE7BFva6eQ1E4Oe3SC\n2NJxI2Vee3WvyNVlJGeH0h133IE777xT9DyOYP2Zi4jascTERBQWFuKTTz4xnZK4uWv63mrx4sUY\nNmyY6ayO169fb/Kfo+KBG1chS0xMRHFxMerr61FUVIQ33nijxfZUKfPaI2db85Y79nZTpkyBu7s7\n5s6di7Nnz+LVV19tcXxCQgLef/99/PHHH6Lnsjd+4UsE6Z1CgPRODlvjpXYZSclbSmxrdhlJibVn\np9BPP/2E+Ph4FBcXw8XFxey+/Px8q+axFxZ/ov8QJHYKSe3kkBpva7eOlHnt0b3SGl1GUmLt2Sk0\nduxYPPTQQ9DpdNBqtWb3BQUFWU7ajrjmT/Qft343UFpaik2bNuGLL75AREQEnnrqKSxbtqzF+KSk\nJOzZswdBQUGNfrGtISb+2WefxRdffAEAWLFiBeLj421uLZWSt9hYKXnLFWvPK3EVFRVh8+bNjT71\ny0K2r5qJnICYTiGpnRxi4u15vhgpeYuNlavLqC10KAmCIERHRwuFhYU2z29P/ORP1AIxnUZSOznE\nxNvzfDFS8hYbK1eXUVvoUAKA4OBgxMbGYsKECbj77rvN7mvutA+O4pKUlJTUqjMStVN33nknPv/8\ncwwYMKBRgbB3/PDhw3Ht2jVUVFTg9OnT+K//+i/U19c3+vfUU085NG+xsVLyliv2VsePH8fevXvR\nv39/eHh4oLi4GO+++y68vb2h0+ks7v9rr72G+vp6nDhxAnl5eaZ/hw4dsvoaAvbCL3yJ7ERqJ4et\n8VK7jKTkLSVWri4juTqU2hoWfyI7kdrJYWu81I4bKXlLiZWry0juDqW2gmv+RHYitZPD1nipXUZS\n8pYS25pdRlJi7dlZ1Zaw+BPZSb9+/XDx4kX4+Pi0avzOnTuxdevWFs/Z74h5pcZKybs1Yy9cuABB\nEKBSqfDpp58iPj5e9JxtEYs/kZ1I7eSwNV5ql5GUvKXEtmaXkZTYtnolLqm45k9kJ81d/UmlUmHP\nnj0Oi8/OzsahQ4cwZ84cq85HZK95pcZKybs1Y9vqlbikYvEncnJt6XwxYsjVZSRXh1Jbw+JP5OTa\n0vlixJCry0iuDqW2hmv+RE6uTZ0vRgS5uozk6lBqa1j8iZyc1C4jucjVZSRXh1Jbw+JP5OTa0vli\nxJCry0iuDqW2hmv+RE5OapeRXOTqMpKrQ6mtYfEnIrKSs3ZWNYXFn4jISs7aWdUUFn8iIisNGDAA\nR48edbrOqqao5U6AiMhZNHQKtQfs9iEispKzdlY1hcs+RERWctbOqqaw+BMRKRDX/ImIFIjFn4hI\ngVj8iYgUiMWfiEiBWPyJiBSIxZ+ISIFY/ImIFOj/AaB4uxoTJ1cGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10ecf0438>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot = sns.barplot(data=importances)\n",
    "for i in plot.get_xticklabels():\n",
    "    i.set_rotation(90)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "From the above chart, the three most important features for this application are: $n_e$, $\\mathrm{mfd}_{e \\rightarrow ag}$, and $\\mathsf{ACC}$."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}