03_wrangling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Workbook : Data Wrangling\n",
    "\n",
    "To get a feel for data wrangling and to help get you started thinking about your second assignment. We'll work through an example in this workbook to:\n",
    "* read a DataFrame into Python\n",
    "* wrangle the data\n",
    "* clean the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part I: Setup\n",
    "\n",
    "Data wrangling often requires additional functionality outside what's included in Python by default. For this, we'll import other functions from helpful packages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Import the following packages using their common shortened name found in parentheses:**\n",
    "\n",
    "* `numpy` (`np`)\n",
    "* `pandas` (`pd`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Run the following cell code to make things throughout the rest of this workbook a little prettier.** (Note: You don't have to edit code here, but are free to and see what changes to be sure you understand each line.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure libraries\n",
    "\n",
    "# Don't display too many rows/cols of DataFrames\n",
    "pd.options.display.max_rows = 7\n",
    "pd.options.display.max_columns = 8\n",
    "\n",
    "# Round decimals when displaying DataFrames\n",
    "pd.set_option('precision', 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Read the CSV file at the URL 'https://raw.githubusercontent.com/fivethirtyeight/data/master/steak-survey/steak-risk-survey.csv' into Python in and assign it to the variable `survey`**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert survey.shape == (551, 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These data contain survey responses from Americans who responded to a SurveyMonkey Audience poll. These data were used in the [FiveThirtyEight](https://fivethirtyeight.com) article: *[How Americans Like Their Steak](https://fivethirtyeight.com/features/how-americans-like-their-steak/)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part II: Wrangling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Write a line of code to look at the first few rows of the DataFrame**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "survey.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice about the first row of the dataframe? Notice that it's not actually an observation from a respondent? **Remove this row from the dataset. Assign this back to the variable `survey`. Print the first few rows again to make sure you've accomplished this.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "## YOUR CODE HERE "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert survey.shape == (550, 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that there are a lot of different questions that were asked of respondents (columns) and 550 people who responded to the survey (rows). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Print a list of all the column names in this DataFrame.**\n",
    "\n",
    "We'll only end up working with a subset of these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a sense of what information is included in the dataset. In the coming weeks, we'll answer the following questions:\n",
    "1. Who cheats more on their significant other - males or females?\n",
    "2. Are cigarette smokers less likely to skydive?\n",
    "3. Do people in New England gamble more than other parts of the country?\n",
    "\n",
    "To answer these we'll only need data from *some* of the columns in the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's drop the columns we don't need. **Drop the first two columns from the dataset. This should still be assigned to the variable `survey`.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert survey.shape == (550,13)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've got the columns we want, let's clean up those column names. **Rename the columns in `survey` so the appropriate columns have the following names:**\n",
    "\n",
    "* smoking\n",
    "* alcohol\n",
    "* gamble\n",
    "* skydiving\n",
    "* speeding\n",
    "* cheated\n",
    "* steak\n",
    "* steak_preference\n",
    "* gender\n",
    "* age\n",
    "* income \n",
    "* education\n",
    "* region\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert list(survey) == ['smoking',\n",
    "                        'alcohol',\n",
    "                        'gambling',\n",
    "                        'skydiving',\n",
    "                        'speeding',\n",
    "                        'cheated',\n",
    "                        'steak',\n",
    "                        'steak_preference',\n",
    "                        'gender',\n",
    "                        'age',\n",
    "                        'income',\n",
    "                        'education',\n",
    "                        'region']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're in pretty good shape now. **Print the first few rows of the `survey` DataFrame to see what the data look like at this point.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "survey.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part III: Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Now that we've go the data we need, let's tet a sense for how much missing data there is in this dataset by determining how many null-containing rows there are in `survey`. Assign this value to the variable `null_rows`.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert null_rows == 217"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good to know that lots of people didn't answer every question. We'll keep that in mind as we work with this dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simply dropping missing observations is typically not good practice; however, in this case we'll drop observations that have missing data across the entire row, as these are individuals who didn't participate in the survey at all. **Remove rows where ALL the columns have missing data for that participant.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert survey.shape == (541, 13)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Print the first few rows to remind yourself what the data look like at this point.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the first row no longer has all missing data here. We've got a dataset we can work with now!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Great work on this workbook! We'll continue to work with this dataset in section to answer our questions of interest. You can work on A2 or help a classmate work through this - we always understand things best once we've had to explain them to someone else.**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}