3_Pandas_Intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas Primer  \n",
    "\n",
    "[Pandas](https://pandas.pydata.org/) is a popular and useful data analysis library. \"Pandas\" stands for \"panel data analysis\" or sometimes \"Python data analysis\".   \n",
    "\n",
    "Pandas is the go to package for working with panel data, CSVs, data stored in text files, Excel sheets, or even SQL databases. If you're coming from Excel, you'll find the Pandas approach fairly intuitive.   \n",
    "\n",
    "In this primer, we'll cover the basics.\n",
    "\n",
    "Imports:\n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll see Pandas usually abbreviated as ```pd``` and Numpy usually abbreviated as ```np```.\n",
    "\n",
    "In Pandas, data is stored as either a \"series\" (one-dimensional) or a \"dataframe\" (two-dimensional).  \n",
    "\n",
    "You'll usually import data into a dataframe, but to begin we'll create this basic dataframe from scratch using ```pd.dataframe()```. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({\"Person\":[\"John\", \"Myla\", \"Lewis\", \"John\", \"Myla\"],\n",
    "                   \"Age\": [24., np.nan, 21., 33, 26],\n",
    "                   \"Single\": [False, True, True, True, False]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python \n",
    "df\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View just the column headers:  \n",
    "\n",
    "```python\n",
    "df.columns\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select individual columns:  \n",
    "\n",
    "```python\n",
    "df['Age']\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select multiple columns by nesting a list:  \n",
    "\n",
    "```python\n",
    "df[['Person','Age']]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select rows by index position using ```iloc```:  \n",
    "\n",
    "```python\n",
    "df.iloc[1:3]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select by values using ```loc```:  \n",
    "\n",
    "```python\n",
    "df.loc[df['Single'] == True]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select based on multiple conditions:\n",
    "\n",
    "```python\n",
    "df.loc[(df['Age'] > 20) & (df['Age'] < 30)]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a new dataframe based on a selection:\n",
    "\n",
    "```python\n",
    "newDF = df.loc[(df['Age'] > 20) & (df['Age'] < 30)]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A note on \"copying\"... The above method is really a view rather than a copy. You can sometimes unintentionally modify the original doing it this way, which may give you unexpected results down the line if you plan on using the original again. What?? Yeah.  \n",
    "\n",
    "You may get some oblique warnings if you do it the above way...  \n",
    "\n",
    "The prefered way to split off a selection of a dataframe is using the ```.copy()``` method:  \n",
    "\n",
    "```python\n",
    "newDF2 = df.loc[(df['Age'] > 20) & (df['Age'] < 30)].copy()\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here is the difference. Try this simple modification:  \n",
    "\n",
    "```python\n",
    "newDF['Age'] = newDF['Age'] + 1\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uh. What?  \n",
    "\n",
    "Okay, try again using ```newDF2```: \n",
    "\n",
    "```python\n",
    "newDF2['Age'] = newDF['Age'] + 1\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a **TON** of built in dataframe methods that help you with munging, manipulating, statistics, and a more. See the docs: [https://pandas.pydata.org/pandas-docs/stable/reference/frame.html](https://pandas.pydata.org/pandas-docs/stable/reference/frame.html).  \n",
    "\n",
    "Some simple descriptive statistics...  \n",
    "\n",
    "```python\n",
    "\n",
    ".min()\n",
    ".max()\n",
    ".mean()\n",
    ".sum()\n",
    ".cumsum()\n",
    ".mode()\n",
    ".median()\n",
    ".count()\n",
    ".std()\n",
    ".unique()\n",
    ".describe()\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dealing with missing data:  \n",
    "\n",
    "Lot's of [approaches](https://pandas.pydata.org/pandas-docs/stable/reference/frame.html#missing-data-handling), but commonly:  \n",
    "\n",
    "```.isna()``` \n",
    "```.fillna()```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note! Was this change permanent?  \n",
    "\n",
    "Two ways to \"save\" a change:   \n",
    "\n",
    "```python\n",
    "df = df.fillna(20.0)\n",
    "```\n",
    "OR  \n",
    "\n",
    "```python\n",
    "df.fillna(20.0, inplace=True)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating new columns:  \n",
    "```python\n",
    "df['chainsaw'] = 'vrooooooom'\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Math problem:\n",
    "```python\n",
    "df['result'] = df['Age'] ** 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "String methods:  \n",
    "\n",
    "Concat new string to a column value to form a new column:\n",
    "```python\n",
    "df['email'] = df['Person'] + '@cuboulder.rules'\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split a string column using ```.str.split()```:  \n",
    "\n",
    "Note the [documentation](https://pandas.pydata.org/docs/reference/api/pandas.Series.str.split.html). Use the ```expand``` parameter to split into columns.  \n",
    "\n",
    "Split email on ```@```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What did ```.str.split()``` return?  \n",
    "\n",
    "\n",
    "How do we select an element by it's index position?  \n",
    "\n",
    "Make a ```domain``` column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "df\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delete a column using ```.drop()```\n",
    "```python\n",
    "df.drop(columns='chainsaw', inplace=True)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delete multiple columns.  \n",
    "Feed the columns parameter a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying a custom function to a column.  \n",
    "\n",
    "Sometimes, you just have to do something your way! Imagine you have a \"formula\" you want to apply across all the rows in a column, excel style. You may be tempted to iterate through the rows of a column (and you can using ```.iterrows```), but this is not the most efficient way. \n",
    "\n",
    "You can use ```.apply()``` to apply a custom function to all rows in a column. \n",
    "\n",
    "Let's start with a quick side trip on functions:\n",
    "\n",
    "```python\n",
    "def mingle(age, single):\n",
    "    if age < 30:\n",
    "        if single == True:\n",
    "            result = 'READY TO MINGLE'\n",
    "        else: \n",
    "            result = \"Home by 8pm\"\n",
    "    else:\n",
    "        result = \"Home by 8pm\"\n",
    "    \n",
    "    return result\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does this function work? Let's try!  \n",
    "\n",
    "```python\n",
    "mingle(25,True)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, now we'll use [```df.apply()```](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html), a [```lambda```](https://www.w3schools.com/python/python_lambda.asp) function to run this on our dataframe with the ```Age``` and ```Single``` columns as input.  \n",
    "\n",
    "(Don't get too hung up on the details right now :-) ). \n",
    "\n",
    "```python\n",
    "df['status'] = df.apply(lambda row: mingle(row['Age'], row['Single']), axis = 1)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reshaping data  \n",
    "\n",
    "First, what is the shape?  \n",
    "\n",
    "```python\n",
    "df.shape\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Transpose using ```.T```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pivoting and Pivot Tables "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({\"A\": [\"foo\", \"foo\", \"foo\", \"foo\", \"foo\",\n",
    "                        \"bar\", \"bar\", \"bar\", \"bar\"],\n",
    "                    \"B\": [\"one\", \"one\", \"one\", \"two\", \"two\",\n",
    "                        \"one\", \"one\", \"two\", \"two\"],\n",
    "                    \"C\": [\"small\", \"large\", \"large\", \"small\",\n",
    "                        \"small\", \"large\", \"small\", \"small\", \"large\"],\n",
    "                    \"D\": [1, 2, 2, 3, 3, 4, 5, 6, 7]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can create Excel-style pivot tables--useful for aggregating:\n",
    "\n",
    "```python\n",
    "pd.pivot_table(df,values='D', index='A', aggfunc='sum')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rename a column using ```.rename()```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Merge the aggregated data back to our original dataframe:  \n",
    "\n",
    "```python\n",
    "df = pd.merge(df, aggDF, on='A')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, what's the proportion of of individual ```D``` to ```sumD```? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar: ```.groupby()```  \n",
    "\n",
    "```python\n",
    "df.groupby(['A','C']).sum('D')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's all for this lesson... questions?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "datacamp",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}