03-Binning-and-aggregation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Binning and Aggregation\n",
    "\n",
    "We have discussed **data**, **marks**, **encodings**, and **encoding types**.\n",
    "The next essential piece of Altair's API is its approach to binning and aggregating data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import altair as alt\n",
    "\n",
    "# Altair plots render by default in JupyterLab and nteract\n",
    "\n",
    "# Uncomment/run this line to enable Altair in the classic notebook (not in JupyterLab)\n",
    "# alt.renderers.enable('notebook')\n",
    "\n",
    "# Uncomment/run this line to enable Altair in Colab\n",
    "# alt.renderers.enable('colab')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from vega_datasets import data\n",
    "cars = data.cars()\n",
    "\n",
    "cars.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Group-By in Pandas\n",
    "\n",
    "One key operation in data exploration is the *group-by*, discussed in detail in [Chaper 4](https://jakevdp.github.io/PythonDataScienceHandbook/03.08-aggregation-and-grouping.html) of the *Python Data Science Handbook*.\n",
    "In short, the group-by *splits* the data according to some condition, *applies* some aggregation within those groups, and then *combines* the data back together:\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/eitanlees/altair-tutorial/master/images/split-apply-combine.png\">\n",
    "\n",
    "[Figure source](https://jakevdp.github.io/PythonDataScienceHandbook/03.08-aggregation-and-grouping.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the cars data, you might split by Origin, compute the mean of the miles per gallon, and then combine the results.\n",
    "In Pandas, the operation looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cars.groupby('Origin')['Miles_per_Gallon'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Altair, this sort of split-apply-combine can be performed by passing an aggregation operator within a string to any encoding. For example, we can display a plot representing the above aggregation as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(cars).mark_bar().encode(\n",
    "    y='Origin',\n",
    "    x='mean(Miles_per_Gallon)'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the grouping is done implicitly within the encodings: here we group only by Origin, then compute the mean over each group."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-dimensional Binnings: Histograms\n",
    "\n",
    "One of the most common uses of binning is the creation of histograms. For example, here is a histogram of miles per gallon:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(cars).mark_bar().encode(\n",
    "    alt.X('Miles_per_Gallon', bin=True),\n",
    "    alt.Y('count()'),\n",
    "    alt.Color('Origin')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One interesting thing that Altair's declarative approach allows us to start assigning these values to different encodings, to see other views of the exact same data.\n",
    "\n",
    "So, for example, if we assign the binned miles per gallon to the color, we get this view of the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(cars).mark_bar().encode(\n",
    "    alt.Color('Miles_per_Gallon', bin=True),\n",
    "    alt.X('count()'),\n",
    "    alt.Y('Origin')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us a better appreciation of the proportion of MPG *within* each country.\n",
    "\n",
    "If we wish, we can normalize the counts on the x-axis to compare proportions directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(cars).mark_bar().encode(\n",
    "    alt.Color('Miles_per_Gallon', bin=True),\n",
    "    alt.X('count()', stack='normalize'),\n",
    "    alt.Y('Origin')\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that well over half of US cars were in the \"low mileage\" category.\n",
    "\n",
    "Changing the encoding again, let's map the color to the count instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(cars).mark_bar().encode(\n",
    "    alt.X('Miles_per_Gallon', bin=True),\n",
    "    alt.Color('count()'),\n",
    "    alt.Y('Origin')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we see the same dataset as a heat map!\n",
    "\n",
    "This is one of the beautiful things about Altair: it shows you through its API grammar the relationships between different chart types: for example, a 2D heatmap encodes the same data as a stacked histogram!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other aggregates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aggregates can also be used with data that is only implicitly binned.\n",
    "For example, look at this plot of MPG over time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(cars).mark_point().encode(\n",
    "    x='Year:T',\n",
    "    color='Origin',\n",
    "    y='Miles_per_Gallon'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fact that the points overlap so much makes it difficult to see important parts of the data; we can make it clearer by plotting the mean in each group (here, the mean of each Year/Country combination):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(cars).mark_line().encode(\n",
    "    x='Year:T',\n",
    "    color='Origin',\n",
    "    y='mean(Miles_per_Gallon)'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ``mean`` aggregate only tells part of the story, though: Altair also provides built-in tools to compute the lower and upper bounds of confidence intervals on the mean.\n",
    "\n",
    "We can use ``mark_area()`` here, and specify the lower and upper bounds of the area using ``y`` and ``y2``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(cars).mark_area(opacity=0.3).encode(\n",
    "    x='Year:T',\n",
    "    color='Origin',\n",
    "    y='ci0(Miles_per_Gallon)',\n",
    "    y2='ci1(Miles_per_Gallon)'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Time Binnings with *timeUnit*\n",
    "\n",
    "One special kind of binning is the grouping of temporal values by aspects of the date: for example, month of year, or day of months.\n",
    "To explore this, let's look at a simple dataset consisting of average temperatures in Seattle:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "temps = data.seattle_temps()\n",
    "temps.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we try to plot this data with Altair, we will get a ``MaxRowsError``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(temps).mark_line().encode(\n",
    "    x='date:T',\n",
    "    y='temp:Q'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(temps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aside: How Altair Encodes Data\n",
    "\n",
    "We chose to raise a MaxRowsError for datasets larger than 5000 rows because of our observation of students using Altair, because unless you think about how your data is being represented, it's quite easy to end up with **very** large notebooks inwhich performance will suffer.\n",
    "\n",
    "When you pass a pandas dataframe to an Altair chart, the result is that the data is converted to JSON and stored in the chart specification. This specification is then embedded in the output of your notebook, and if you make a few dozen charts this way with a large enough dataset, it can significantly slow down your machine.\n",
    "\n",
    "So how to get around the error? A few ways:\n",
    "\n",
    "1) Use a smaller dataset. For example, we could use Pandas to aggregate the temperatures by day:\n",
    "   ```python\n",
    "   import pandas as pd\n",
    "   temps = temps.groupby(pd.DatetimeIndex(temps.date).date).mean().reset_index()\n",
    "   ```\n",
    "\n",
    "2) Store the data to file, and reference the data by filename instead. This lets us plot the full dataset without having to embed it in the notebook.\n",
    "\n",
    "3) Disable the MaxRowsError using\n",
    "   ```python\n",
    "   alt.data_transformers.enable('default', max_rows=None)\n",
    "   ```\n",
    "   But note this can lead to *very* large notebooks if you're not careful.\n",
    "\n",
    "4) Automatically load the data from disk using\n",
    "   ```python\n",
    "   alt.data_transformers.enable('json')\n",
    "   ```\n",
    "   which will automatically save the data to file and provide the url of that file to the chart each time a chart is created.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll do the latter here, which is the most convenient and leads to the best performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.data_transformers.enable('json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(temps).mark_line().to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(temps).mark_line().encode(\n",
    "    x='date:T',\n",
    "    y='temp:Q'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This data is a little bit crowded; suppose we would like to bin this data by month. We'll do this using the ``month`` aggregate on the date:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(temps).mark_point().encode(\n",
    "    x='month(date)',\n",
    "    y='temp:Q'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This might be clearer if we now aggregate the temperatures:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(temps).mark_bar().encode(\n",
    "    x='month(date):O',\n",
    "    y='mean(temp):Q'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also split dates two different ways to produce interesting views of the data; for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(temps).mark_rect().encode(\n",
    "    x='date(date):O',\n",
    "    y='month(date):O',\n",
    "    color='mean(temp):Q'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can look at the hourly average temperature as a function of month:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alt.Chart(temps).mark_rect().encode(\n",
    "    x='hours(date):O',\n",
    "    y='month(date):O',\n",
    "    color='mean(temp):Q'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This kind of transform can be quite useful when working with temporal data.\n",
    "\n",
    "More information on ``timeUnit`` is available here: https://altair-viz.github.io/user_guide/transform.html#timeunit-transform"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}