W6D1-Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demo W6D1: Data Analysis Tools I - Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"pandas-chaos.jpg\" width=\"400\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1.1 Series\n",
    "\n",
    "**Creating series from dictionaries.** First, let's important the `pandas` module, define a dictionary, and see how we can turn it into a series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # IF THE CELL ABOVE IMPORTING PANDAS THROWS AN ERROR\n",
    "# # TRY UNCOMMENTING THE LAST TWO LINES IN THIS CELL, THEN RUN IT\n",
    "# # THEN RERUN THE CELL IMPORTING PANDAS\n",
    "# import sys\n",
    "# ! conda install --yes --prefix {sys.prefix} -c conda-forge pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "violent_rates_dict = {\"Belfast East\": 13.9,\n",
    "                      \"Belfast North\": 24.2,\n",
    "                      \"Belfast South\": 20.6,\n",
    "                      \"Belfast West\": 25.4\n",
    "                     }\n",
    "\n",
    "vrni_series = pd.Series(violent_rates_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the dictionary. It contains the by now familiary keys and values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "violent_rates_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's look at the same data as a `pandas` series. This is much easier to read, since it comes in a tabular format with one column for indices and one column for values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vrni_series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Accessing data in a series.** To draw specific content from a `pandas` series, we can use the string indices like in a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vrni_series['Belfast East']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But, we can also use numeric indices like in lists since series are ordered. Using a single numeric index, returns the value stored in the series at the specified position."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vrni_series[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we instead specify a range of indices, i.e. slice the data, we get a subset of the series including the character indices, their associated values, and an indicator of the series type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vrni_series[0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question.** What is the type of the value returned if we specify one numeric index in the series? What is the type if we slice the list?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with lists, we can also slice the series at select indices instead of using a range of indices. To do so, we need to index the series using `[[`; where the innermost `[` define a list of indices and the outermost `[` index the series. The cell below returns the first and last entry in the series, first by specifying the numeric index then by specifying the character index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(vrni_series[[0,-1]])\n",
    "print(vrni_series[['Belfast East','Belfast West']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Creating series from other data structures.** Now, let's see how we can create series from other data structures than dictionaries, starting with lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "constituency_names = [\"Belfast East\", \n",
    "                      \"Belfast North\", \n",
    "                      \"Belfast South\", \n",
    "                      \"Belfast West\"]\n",
    "\n",
    "violent_rates_list = [13.9,\n",
    "                      24.2,\n",
    "                      20.6,\n",
    "                      25.4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a series from two lists containing values and character indices, we can pass the values as the first argument to the `Series` command and specify the argument `index` as equal to the list of indices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vrni_series_from_lists = pd.Series(violent_rates_list, index = constituency_names)\n",
    "\n",
    "vrni_series_from_lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to ensure that we have the same data in the two series we have created so far, we can ask Python to compare them, which will compare the values at each index position to each other. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vrni_series_from_lists == vrni_series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, we can check to see if _all_ values in the series are equal at each index position."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all (vrni_series_from_lists == vrni_series)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we forget to specify the series indices when we create it, we can specify them after the fact by assigning the list of indices to the `index` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vrni_series_from_lists = pd.Series(violent_rates_list)\n",
    "\n",
    "vrni_series_from_lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vrni_series_from_lists.index = constituency_names\n",
    "\n",
    "vrni_series_from_lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Numerical operations with series.** `Pandas` series can contain any type of data and have different commands applicable to each type of data. For numeric data you might be interested in descriptive statistics which we will cover in the next session. For all types of data, you can count the number of data points in your series using the `count` command. Note that this counts only non-missing data points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vrni_series_from_lists.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(vrni_series_from_lists)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we are appending an entry to our `Series` with a missing value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "vrni_series_with_missing = vrni_series_from_lists.append(pd.Series(np.NaN))\n",
    "\n",
    "vrni_series_with_missing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vrni_series_with_missing.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(vrni_series_with_missing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1.2 DataFrames\n",
    "\n",
    "**Creating DataFrames from dictionaries.** In most applied cases, you will be interested to have more than one value associated with each observation. To accommodate this, `pandas` introduces `DataFrame` containers which we can also construct from dictionaries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, we first define two additional lists containing values for our constituencies, then we construct a dictionary from the four lists and keys we assign to them. Finally, we turn this complex dictionary containing four keys and values consisting of lists into a neat `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "theft_rates = [7.6, 11.2, 14.7, 10.9]\n",
    "veh_crime_rates = [1.8, 3.4, 4.5, 5.6]\n",
    "\n",
    "complex_dictionary = {\"Name\": constituency_names,\n",
    "                      \"Violence Rate\": violent_rates_list,\n",
    "                      \"Theft Rate\": theft_rates,\n",
    "                      \"Veh Crime Rate\": veh_crime_rates}\n",
    "\n",
    "df_from_dict = pd.DataFrame(complex_dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does the dictionary look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "complex_dictionary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And what does the same data look like in a `DataFrame` container?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_from_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Creating DataFrames from other data structures.** First, a `DataFrame` out of lists of tuples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_of_tuples = [(\"Belfast East\", 13.9, 7.6, 1.8),\n",
    "                  (\"Belfast North\", 24.2, 11.2, 3.4),\n",
    "                  (\"Belfast South\", 20.6, 14.7, 4.5),\n",
    "                  (\"Belfast West\", 25.4, 10.9, 5.6)]\n",
    "\n",
    "df_from_list_of_tuples = pd.DataFrame(list_of_tuples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This also creates a neat `DataFrame` but does not give us the column headings or variable names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_from_list_of_tuples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second, a `DataFrame` out of a list of dictionaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_of_dicts = [{\"Name\": \"Belfast East\",\n",
    "                  \"Violence Rate\": 13.9,\n",
    "                  \"Theft Rate\": 7.6,\n",
    "                  \"Veh Crime Rate\": 1.8},\n",
    "                 {\"Name\": \"Belfast North\", \n",
    "                  \"Violence Rate\": 24.2,\n",
    "                  \"Theft Rate\": 11.2, \n",
    "                  \"Veh Crime Rate\": 3.4},\n",
    "                 {\"Name\": \"Belfast South\", \n",
    "                  \"Violence Rate\": 20.6, \n",
    "                  \"Theft Rate\": 14.7,\n",
    "                  \"Veh Crime Rate\": 4.5},\n",
    "                 {\"Name\": \"Belfast West\", \n",
    "                  \"Violence Rate\": 25.4, \n",
    "                  \"Theft Rate\": 10.9, \n",
    "                  \"Veh Crime Rate\": 5.6}]\n",
    "\n",
    "df_from_list_of_dicts = pd.DataFrame(list_of_dicts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another neat `DataFrame` with column headings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df_from_list_of_dicts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Accessing data in a DataFrame.** Like a `pandas` `Series`, we can access the data in a `DataFrame` using string and numeric indices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To access the `Name` column, we can index it by it's string name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_from_dict['Name']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or access it from the `DataFrame` attribute `Name`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_from_dict.Name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question.** What happens if we try to access the column \"Veh Crime Rate\" as an attribute of our `DataFrame`?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we don't know the name of the column, we want to index, we can look up all names using the `columns` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_from_dict.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to looking at the column indices, we can also set them by assigning a list of strings with the length of the columns to the column attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_from_dict.columns = ['Name', 'ViolenceRate', 'TheftRate', 'VehCrimeRate']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using numeric indices, allows us to access the rows in the `DataFrame` and functions analogous to indexing lists, i.e. `[StartIndex:EndIndex:StepLength]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_from_dict[0::2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also access the same rows by passing a list of booleans (True/False) to the indexing brackets. NB that there are two sets of `[`, one for the indexing the other to define the list of booleans."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_from_dict[[True, False, True, False]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with `Series`, we can add a character `index` to our `DataFrame` that we can then use to index the values in specific cells. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "constituency_initials = [\"BE\", \"BN\", \"BS\", \"BW\"]\n",
    "\n",
    "df_from_dict.index = constituency_initials\n",
    "\n",
    "df_from_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_from_dict[\"VehCrimeRate\"][\"BE\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beyond accessing rows through their position, we can also subset the `DataFrame` according to specific conditions. So, if we only wanted to look at constituencies that have a vehicle crime rate above 4, we would do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_from_dict[df_from_dict[\"VehCrimeRate\"] > 4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to subsetting by inclusion, i.e. saying we want to look at specific columns we can also subset by exclusion using the `drop` command and specifying the `axis` argument as `1` to subset columns and `0` to subset rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_from_dict.drop([\"ViolenceRate\"], axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_from_dict.drop([\"BE\"], axis = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question.** Looking at the cell above and knowing how indexing rows works, can you tell how subsetting here works? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Manipulating existing DataFrames.** So far, we have only asked Python to show us different slices of our data but we can also manipulate the underlying data such as subset it permanently or add rows or columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cell below creates a subset of our original data only containing the constituencies that have a vehicle crime rate above 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_from_dict_highVehCrimeRate = df_from_dict[df_from_dict[\"VehCrimeRate\"] > 4]\n",
    "\n",
    "df_from_dict_highVehCrimeRate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To add a row to our exisiting `DateFrame`, we first need to define the new row, turn it into a `DataFrame` object and then use the append command. NB we are setting the `ignore_index` argument to `True` here which removes our string row indices and also provides us with consistent row indices. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_row = [{\"Name\": \"East Antrim\",\n",
    "            \"ViolenceRate\": 9.7,\n",
    "            \"TheftRate\": 4.5,\n",
    "            \"VehCrimeRate\": 0.9}]\n",
    "\n",
    "new_row_df = pd.DataFrame(new_row)\n",
    "\n",
    "df_from_dict_longer = df_from_dict.append(new_row_df, ignore_index = True)\n",
    "\n",
    "df_from_dict_longer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we did not set the `ignore_index` argument to `True`, we would either end up with inconsistent row indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_from_dict_inconsistent = df_from_dict.append(new_row_df)\n",
    "\n",
    "df_from_dict_inconsistent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we would end up with duplicate indices, which can lead to unexpected indexing behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_from_list_of_dicts.columns = ['Name', 'ViolenceRate', 'TheftRate', 'VehCrimeRate']\n",
    "\n",
    "df_from_list_of_dicts_duplicate_indices = df_from_list_of_dicts.append(new_row_df)\n",
    "\n",
    "df_from_list_of_dicts_duplicate_indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_from_list_of_dicts_duplicate_indices[\"Name\"][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.1.3 Pandas for easy import and manipulation of tabular data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we briefly touched upon in previous sessions, `Pandas` also has built-in functionality to import all kinds of tabular data which we can then manipulate with our knowledge about how to work with `DataFrames`. So rather than reading last weeks causes of death CSV line by line, we can do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "us_deaths_2016 = pd.read_csv('nchs_death_causes.csv')\n",
    "\n",
    "us_deaths_2016"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, with the death data in a `DataFrame`, we can easily subset the data. For example, if we want to only look at deaths attributable to accidents or Alzheimer's disease, we could use the `DataFrame` method `isin` test whether the cause for a given row is one of either."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accidents_alzheimer_2016 = us_deaths_2016[us_deaths_2016[\"Cause Name\"].isin([\"Unintentional injuries\", \"Alzheimer's disease\"])]\n",
    "\n",
    "accidents_alzheimer_2016"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And plot it just as easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(accidents_alzheimer_2016['Age-adjusted Death Rate'][accidents_alzheimer_2016['Cause Name'] == 'Unintentional injuries'],\n",
    "         accidents_alzheimer_2016['Age-adjusted Death Rate'][accidents_alzheimer_2016['Cause Name'] == \"Alzheimer's disease\"], \n",
    "         'o')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(accidents_alzheimer_2016['Age-adjusted Death Rate'][accidents_alzheimer_2016['Cause Name'] == \"Alzheimer's disease\"])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}