04-Pandas-Data-Frame.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas for Data Analysis\n",
    "\n",
    "Pandas package implements functionalities like data frames in R. There are many similarities but also differences. We will go over some differences in the context of working with the basketball data.\n",
    "\n",
    "Obtaining data from NBA can be done using the function developed previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "def get_nba_data(endpt, params, return_url=False):\n",
    "\n",
    "    ## endpt: https://github.com/seemethere/nba_py/wiki/stats.nba.com-Endpoint-Documentation\n",
    "    ## params: dictionary of parameters: i.e., {'LeagueID':'00'}\n",
    "    \n",
    "    from pandas import DataFrame\n",
    "    from urllib.parse import urlencode\n",
    "    import json\n",
    "    \n",
    "    useragent = \"\\\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9\\\"\"\n",
    "    dataurl = \"\\\"\" + \"http://stats.nba.com/stats/\" + endpt + \"?\" + urlencode(params) + \"\\\"\"\n",
    "    \n",
    "    # for debugging: just return the url\n",
    "    if return_url:\n",
    "        return(dataurl)\n",
    "    \n",
    "    jsonstr = !wget -q -O - --user-agent={useragent} {dataurl}\n",
    "    \n",
    "    data = json.loads(jsonstr[0])\n",
    "    \n",
    "    h = data['resultSets'][0]['headers']\n",
    "    d = data['resultSets'][0]['rowSet']\n",
    "    \n",
    "    return(DataFrame(d, columns=h))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the function, we can get data about teams and players"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## get all teams\n",
    "params = {'LeagueID':'00'}\n",
    "teams = get_nba_data('commonTeamYears', params)\n",
    "\n",
    "## get all players\n",
    "params = {'LeagueID':'00', 'Season': '2016-17', 'IsOnlyCurrentSeason': '0'}\n",
    "players = get_nba_data('commonallplayers', params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Programming style\n",
    "\n",
    "Programming language is really like a language, and you will get better with practice. It is good to think about good programming style and better way to do the same thing. By better, I mean more readable, concise, efficient (computationally), etc.\n",
    "\n",
    "For example, there are guides and articles such as these:\n",
    "- http://docs.python-guide.org/en/latest/writing/style/#short-ways-to-manipulate-lists\n",
    "- https://google.github.io/styleguide/pyguide.html?showone=List_Comprehensions#List_Comprehensions\n",
    "- https://google.github.io/styleguide/pyguide.html?showone=Naming#Naming\n",
    "- https://www.python.org/dev/peps/pep-0008/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import this"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above easter egg is from Zen of Python: https://www.python.org/dev/peps/pep-0020/. \n",
    "\n",
    "- https://www.quora.com/What-do-different-aphorisms-in-The-Zen-of-Python-mean \n",
    "- 20th aphorism?:https://www.reddit.com/r/Python/comments/3cjhlo/this_disobeys_the_zen_of_python/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas\n",
    "\n",
    "Pandas has an extensive set of functions. Refer to [Chapter 3 in PDSH](https://jakevdp.github.io/PythonDataScienceHandbook/03.00-introduction-to-pandas.html) and the [official website](https://pandas.pydata.org). Latest stable release documentation is here: [http://pandas.pydata.org/pandas-docs/stable/api.html](http://pandas.pydata.org/pandas-docs/stable/api.html).\n",
    "\n",
    "## Pandas Series \n",
    "\n",
    "The section on `Series` is here: http://pandas.pydata.org/pandas-docs/stable/api.html#series. These are available by placing a dot after the object.\n",
    "\n",
    "### Data frames are made of Series\n",
    "Pandas data frames are different objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"data frame object :\", type(teams))\n",
    "print(\"data row object   :\", type(teams.iloc[0]))\n",
    "print(\"data column object:\", type(teams.ABBREVIATION))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that rows as well as columns of pandas data frame are `Series` objects. (In R, rows would be a smaller data frame.)\n",
    "\n",
    "There are categories of functions that are applicable to certain object types:\n",
    "\n",
    "- Pandas general functions: http://pandas.pydata.org/pandas-docs/stable/api.html#general-functions   \n",
    "    e.g., [`pandas.melt()`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.melt.html#pandas-melt) take `DataFrame` as input. \n",
    "- Series methods: http://pandas.pydata.org/pandas-docs/stable/api.html#series\n",
    "- DataFrame methods: http://pandas.pydata.org/pandas-docs/stable/api.html#dataframe\n",
    "\n",
    "### Pandas (often) shows you views\n",
    "\n",
    "Recall that python objects are often _views_ of the same instance in memory space. Following says these are the same objects in memory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "temp = teams\n",
    "print(id(temp) == id(teams))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, if you change one, you see the change in the other:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s1 = pd.Series([0.25, 0.5, 0.75, 1.0], index=['a', 'b', 'c', 'd'])\n",
    "s2 = s1\n",
    "print(\"id of s1:\", id(s1))\n",
    "print(\"id of s2:\", id(s2))\n",
    "print(\"s1 is s2:\", s1 is s2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s1[0] = 10000\n",
    "\n",
    "print(\"s1 changed:\", s1[0])\n",
    "print(\"s2 also   :\", s2[0])\n",
    "#print(\"s1 is s2:\", s1[0] is s2[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Needs to be copied in order to make an independent variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "abbr = teams.ABBREVIATION.copy()\n",
    "abbr is teams.ABBREVIATION"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing\n",
    "\n",
    "There are many different ways to index `Series` and `DataFrames` in pandas: https://pandas.pydata.org/pandas-docs/stable/indexing.html#different-choices-for-indexing.\n",
    "\n",
    "- `.loc` is primarily for using labels and booleans: e.g., column and row indices, comparison operators, etc\n",
    "- `.iloc` is primarily for using integer positions: i.e., like you would matrices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "abbr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dict(abbr.head().items())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Series as lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(abbr.head().items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "abbr.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many more useful functions and properties. Refer to [Chapter 3 in PDSH](https://jakevdp.github.io/PythonDataScienceHandbook/03.00-introduction-to-pandas.html). Latest stable release documentation is here: [http://pandas.pydata.org/pandas-docs/stable/api.html](http://pandas.pydata.org/pandas-docs/stable/api.html).\n",
    "\n",
    "The section on `Series` is here: http://pandas.pydata.org/pandas-docs/stable/api.html#series. These are available by placing a dot after the object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "abbr.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A convenient method function is [`str`](http://pandas.pydata.org/pandas-docs/stable/api.html#string-handling). This allows functions to be applied to each value as strings separately. For example, we can search for patterns. For example, we can search for teams that end with letter `S`: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "abbr.str.contains('S$')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Exercise__: how would you use this to pick out team names that end with S? Can you use the resulting boolean `Series`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## abbr.loc[abbr.str.contains('S$')] ## what is the problem?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Exercise__: what is `dir()` function?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## dir(abbr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Frames\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting columns\n",
    "\n",
    "Following ways to call columns are equivalent. The *dot notation* is easier to read."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "temp = teams.copy()\n",
    "\n",
    "print(temp['MIN_YEAR'].head())\n",
    "print(temp.MIN_YEAR.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting columns\n",
    "\n",
    "Note that you cannot set a new column with a dot notation. Consider the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "temp['new_column_1'] = temp.MAX_YEAR\n",
    "temp.new_column_2 = temp.MAX_YEAR\n",
    "temp.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, you can set an existing column with dot notation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "temp.LEAGUE_ID = 'ZZ'\n",
    "temp.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Frame, Series, dtype\n",
    "\n",
    "This is different than R data frame in that columns in R data frames have their data types: e.g., `factor`, `integer`, `numeric`, etc. Pandas data frame columns are *all* `Series` with different dtypes. With column types not specified, everything is of dtype `object`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(teams.ABBREVIATION.dtype)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "teams.ABBREVIATION = teams.ABBREVIATION.astype('category')\n",
    "teams.TEAM_ID      = teams.TEAM_ID.astype('category')\n",
    "teams.MIN_YEAR     = teams.MIN_YEAR.astype('int')\n",
    "teams.MAX_YEAR     = teams.MAX_YEAR.astype('int')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `object` is a general term"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(type(teams.iloc[0]))\n",
    "print(teams.iloc[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Condition based slicing\n",
    "\n",
    "Subset just the current teams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "teams = teams[teams.MAX_YEAR == 2017]\n",
    "teams['TEAM_AGE'] = teams.MAX_YEAR - teams.MIN_YEAR\n",
    "\n",
    "teams_clean = teams.copy() ## make a copy for later\n",
    "teams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the following indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('*** indexing with .iloc:\\n', teams.iloc[1])\n",
    "print('\\n*** indexing with .loc :\\n', teams.loc[14])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Subset just the players in current teams:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "players = players[players.TEAM_ID.isin(teams.TEAM_ID)]\n",
    "players.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List players groupped by teams:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "players.groupby('TEAM_CODE')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above is called an iterable. You can iterate on the object to see the _views_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for t, p in players.groupby('TEAM_NAME'):\n",
    "    print(\"***\", t)\n",
    "    print('; '.join(p.DISPLAY_LAST_COMMA_FIRST.values), '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Merging data frames\n",
    "\n",
    "First we can create a table of unique rows with full team names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "team_names = players[['TEAM_ABBREVIATION', 'TEAM_CODE']].drop_duplicates()#.set_index('TEAM_ABBREVIATION')\n",
    "team_names.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have team codes (names) as a new column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "teams_clean.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "teams = pd.merge(teams_clean, team_names, left_on='ABBREVIATION', right_on='TEAM_ABBREVIATION')\n",
    "teams.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can apply `str` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "teams.TEAM_CODE = teams.TEAM_CODE.str.capitalize() # returns values so needs to be reassigned\n",
    "teams.sort_values('ABBREVIATION', inplace=True)    # modifies object\n",
    "teams.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "players.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interaction with Widgets\n",
    "\n",
    "One of the advantages of Jupyter notebooks is that it is browser-based. Browsers are highly interactive, and we can also interact with the data by using interactive widgets IPython provides.\n",
    "\n",
    "We will digress a little bit, and talk about widgets. Widgets take user input by waiting for some action. We can create a simple slider to select some number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import interact, FloatSlider, Dropdown, Button\n",
    "\n",
    "def selected_val(x):\n",
    "    print('Selected value is', x)\n",
    "\n",
    "xslider = FloatSlider(min=0.0, max=10.0, step=0.05)\n",
    "interact(selected_val, x=xslider);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(x, y):\n",
    "    print(x, y)\n",
    "    \n",
    "drop1 = {'Galileo': 10, 'Brahe': 11, 'Hubble': 12}\n",
    "drop2 = {'Apple': 345, 'Orange': 234, 'Banana': 123}\n",
    "\n",
    "interact(f, x=drop1, y=drop2);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "menu = {\n",
    "    'juice':['apple', 'peach', 'grape'],\n",
    "    'tea':['ginger', 'green', 'earl grey'],\n",
    "}\n",
    "\n",
    "selected = 'tea'\n",
    "\n",
    "flavor = Dropdown(options=menu[selected], value=menu[selected][0])\n",
    "drink = Dropdown(options=menu.keys(), value=selected)\n",
    "order = Button(description='Order!', icon='check')\n",
    "\n",
    "def update_drink(change):\n",
    "    flavor.options = menu[change['new']]\n",
    "    flavor.selected = menu[change['new']][0]\n",
    "    \n",
    "def make_order(change):\n",
    "    print(flavor.value, drink.value)\n",
    "    \n",
    "drink.observe(update_drink, names='value')\n",
    "order.on_click(make_order)\n",
    " \n",
    "display(flavor, drink, order)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Exercise__: Can you add a widget for selecting the size? Size is independent of flavors; however, it should be included when the order is made. Allow for sizes small, regular, and large."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}