0_potholes.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CSS Labs: Algorithms & Society with Chicago Potholes\n",
    "In this lab, we'll explore the effects of algorithms on society by using data from [Chicago's Open Data portal](https://data.cityofchicago.org).\n",
    "\n",
    "Recently, we have discussed how big data can hurt or help communities.  It is crucial to fully understand the output of an algorithm and whether or not it is harming a community through bias.  A large sample size can lead to a lower sampling variability and therfore a much more precise analysis.  This precision can sometimes leave out other portions of the population and therefore not be trully applicable in the real world when applied.\n",
    "\n",
    "\n",
    "#### Imagine we work for the Chicago Department of Transportation (CDoT)\n",
    "We will build an algorithm to solve a problem the city faces every day.  We will then see what issues that algorithm might create in the real world.  With the help of data visualization tools, we will be able to look at how algorithms can be biased towards certain groups when applied.\n",
    "\n",
    "For the rest of this lab, we will pretend that we are CDoT employees."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 1: Background\n",
    "### 1.1 Learning Objectives\n",
    "1. Become familiar with Chicago open city data.\n",
    "1. Understand how algorithms can be used to inform policy and real-world decisions.\n",
    "1. Understand how algorithms can sometimes have adverse, unintended concequences when used for decisionmaking. \n",
    "\n",
    "### 1.2 Data\n",
    "- Data for this lab come from [Chicago's Open Data portal](https://data.cityofchicago.org). They're all real data.\n",
    "- For simplicity, we merged multiple data sets before this lab that have information about the city's population (size, age, health, race, class, education, income), infrasctructure (housing, streets, traffic), and services (311, repairs, complaints about police). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 2: Setup\n",
    "### 2.1 Import Python Modules\n",
    "Run the following cell to import the necessary python modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import datetime\n",
    "from tqdm import tqdm \n",
    "from scipy.stats import linregress, nbinom\n",
    "import numpy as np\n",
    "from statsmodels.nonparametric.smoothers_lowess import lowess\n",
    "import matplotlib.pyplot as plt\n",
    "#plt.style.use('ggplot')\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Data\n",
    "Run the following cell to import the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "calls = pd.read_csv('data/clean/311_complaints_by_community_area.tsv', sep='\\t')\n",
    "stats = pd.read_csv('data/clean/community_area_stats.tsv', sep='\\t')\n",
    "ph_reports = pd.read_csv('data/clean/potholes.tsv.gz', sep='\\t')\n",
    "\n",
    "#data preparation\n",
    "ph_reports['creation_date'] = pd.to_datetime(ph_reports.creation_date, format=\"%m/%d/%Y\")\n",
    "ph_reports['completion_date'] = pd.to_datetime(ph_reports.completion_date, format=\"%m/%d/%Y\")\n",
    "ph_reports['days_open'] = (ph_reports.completion_date - ph_reports.creation_date).apply(lambda x: x.days)\n",
    "ph_reports.sort_values(by='creation_date', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### About pothole data\n",
    "- Part of CDoT's job is to fill potholes in city streets. To do this, they rely on city residents reporting potholes to the 311 system, which keeps track of all the reported pothole locations as well as the status of repairs.\n",
    "- The pothole data lists every report of a pothole to the city between January 1, 2011, and December 31, 2018.\n",
    "- Below, we take a peak at the data to get a feel for what it looks like. \n",
    "- Notice that the number of pothole reports the city gets changes over time. This is true for two major reasons:\n",
    "    - Winter weather damages streets. When the streets thaw out each spring, that damage turns into potholes. Therefore, we see more reports in the spring than other times of year.\n",
    "    - Some winters are more mild or harsh than other winters. 2014 was one of the [worst winters on record](https://en.wikipedia.org/wiki/Early_2014_North_American_cold_wave), and we see many more pothole reports that year.\n",
    "- Notice also that we have the average amount of traffic on the streets with reported potholes. \n",
    "    - This comes from a different city data source, which measures the number of cars per day driving on busy streets.\n",
    "    - About half of pothole reports are from streets with no traffic data. These streets are generally small and have low traffic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ph_reports.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ph_reports.resample('w', on='creation_date').request_number.count().plot(figsize=(12,10))\n",
    "plt.title('Pothole reports per week')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-info\">\n",
    "\n",
    "#### Short answer 1:\n",
    "The city receives thousands of pothole reports every week! While they do eventually fix all potholes, with so many reports it can be hard for us to know where to start. \n",
    "- How would you determine the order of fixing? Please describe **one** approach that you can think of and explain your **reasoning** behind it. (2-3 sentences)\n",
    "\n",
    "</div >"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your answers here:**\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 3: Algorithm to the Rescue\n",
    "\n",
    "- Algorithms, \"expert systems,\" and machine learning can give us a prioritized list of where to work first. \n",
    "\n",
    "#### Approaches\n",
    "- As with any problem, there are multiple ways we can try to solve this and get the city a list of which potholes to repair first. \n",
    "    - **Random**: We could just pick random potholes to fix each day. This gives every pothole an equal chance of being fixed each day.\n",
    "    - **First come, first serve**: We could fix the potholes in the order they get reported to us.\n",
    "    - **Busy streets first**: We could fix potholes on the busiest streets first, so that the repairs we make help the largest number of drivers.\n",
    "    - **Most complaints first**: Multiple people can (and do) call to complain about a single pothole. In order to minimize complaints, we could fix the potholes with the most complaints first. \n",
    "    - **Other**: We could think of other ways as well. The real algorithm the city uses is more complicated than these ones."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-info\">\n",
    "    \n",
    "#### Short answer 2:\n",
    "- Which of these do you think we **should** use? In a few sentences, say why.\n",
    "- Which of these do you think is the **worst** idea? In a few sentences, say why. \n",
    "<div/>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your answers here:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Helper functions\n",
    "- Don't worry how exactly the code in the cell works. Just run it and scroll down."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "rules = ['random', \n",
    "         'real', 'high_traffic_first', 'first_come_first_serve', \n",
    "         'high_complaint_first']\n",
    "\n",
    "# duplicate reports are closed on the same day for the same address\n",
    "g = ph_reports.groupby(['completion_date', 'address'])\n",
    "\n",
    "# the number of potholes found when the city went to repair\n",
    "a = g[['n_potholes_on_block']].mean()\n",
    "\n",
    "# a way to count the number of rows / reports for each pothole group\n",
    "b = g[['community_area_number']].count()\n",
    "b.columns = ['n_reports']\n",
    "\n",
    "# put together the data above and rename the columns\n",
    "c = a.merge(b, left_index=True, right_index=True).reset_index().reset_index()\n",
    "c.columns = ['pothole_id', 'completion_date', 'address', 'n_potholes_on_block',\n",
    "             'n_reports']\n",
    "\n",
    "# merge our findings back into the reports data\n",
    "ph_reports = ph_reports.merge(c).sort_values(by='creation_date')\n",
    "\n",
    "# remove the duplicate reports\n",
    "potholes = ph_reports.drop_duplicates(subset=['pothole_id'], keep='first')\n",
    "\n",
    "# remove reports that are not on city property\n",
    "potholes = potholes[potholes.outcome != 'bad_request']\n",
    "\n",
    "def run_simulation(rule='random', per_day='real'):\n",
    "    start = datetime.datetime(year=2011, month=1, day=1)\n",
    "    end = datetime.datetime(year=2019, month=1, day=1)\n",
    "\n",
    "    #the number of blocks fixed per day\n",
    "    if per_day == 'mean':\n",
    "        spd = int(potholes.groupby('completion_date').request_number.count().mean())\n",
    "    if per_day == 'median':\n",
    "        spd = int(potholes.groupby('completion_date').request_number.count().median())\n",
    "    elif per_day == 'real':\n",
    "        spd = potholes.groupby('completion_date')[['request_number']].count().reset_index().sort_values(by='completion_date')\n",
    "\n",
    "    reports = ph_reports[['creation_date', 'pothole_id', 'n_reports']]\n",
    "    result = potholes.copy()\n",
    "    result = result[['creation_date', 'completion_date', 'n_potholes_on_block', 'community_area_number', 'ssa',\n",
    "           'avg_traffic', 'days_open', 'pothole_id']]\n",
    "    result = result[(result['creation_date'] <= end) & (result['creation_date'] >= start)]\n",
    "    \n",
    "    #if using the real results, skip the simulation\n",
    "    if rule != 'real':\n",
    "        result['completion_date'] = pd.NaT\n",
    "\n",
    "        # walk through time one day at a time\n",
    "        for today in tqdm(spd.completion_date):\n",
    "            #get all complaints up to today\n",
    "            complaints = reports[reports.creation_date <= today]\n",
    "            #count complaints per pothole\n",
    "            complaints = complaints.groupby('pothole_id').n_reports.count().reset_index()\n",
    "\n",
    "            #get all potholes that aren't yet resolved\n",
    "            unresolved = result[result.completion_date.isna()]\n",
    "            unresolved = unresolved.merge(complaints, how='inner', on='pothole_id')\n",
    "\n",
    "            #figure out how many potholes we can fix today\n",
    "            if per_day == 'real':\n",
    "                n_done = spd[spd.completion_date == today].request_number.values[0]\n",
    "            else:\n",
    "                n_done = spd\n",
    "            \n",
    "            #decide what to fix based on our rule\n",
    "            if rule == 'random':\n",
    "                fix = unresolved.pothole_id.sample(n=min(n_done, unresolved.shape[0]))\n",
    "            elif rule == 'first_come_first_serve':\n",
    "                fix = unresolved.sort_values(by='creation_date').pothole_id.head(n=n_done)\n",
    "            elif rule == 'high_traffic_first':\n",
    "                fix = unresolved.sort_values(by=['avg_traffic', 'creation_date'], \n",
    "                                             ascending=[False, True]).pothole_id.head(n=n_done)\n",
    "            elif rule == 'high_complaint_first':\n",
    "                fix = unresolved.sort_values(by=['n_reports', 'creation_date'], \n",
    "                                             ascending=[False, True]).pothole_id.head(n=n_done)\n",
    "            else:\n",
    "                print('Invalid rule. Check your spelling?')\n",
    "                return\n",
    "            \n",
    "            # fix the selected potholes today\n",
    "            result.loc[result.pothole_id.isin(fix), 'completion_date'] = today\n",
    "            \n",
    "        result['days_open'] = (result.completion_date - result.creation_date).apply(lambda x: x.days)\n",
    "    \n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simulation\n",
    "- In this next cell, we're going to simulate what would have happened if the city used each of our algorithms. The simulation doesn't look like much while it is running; we just see a progress bar. In the simulation code, however, your computer is doing a lot. \n",
    "- Simulation steps:\n",
    "    1. Begin January 1st, 2011.\n",
    "    2. Get the list of complaints to the city about potholes made on that day.\n",
    "    3. See how many potholes the city was able to fix that day (this changes each day depending on many things, like weekends or bad weather). Call this number N.\n",
    "    4. Use the algorithm to rank which potholes are highest priority.\n",
    "    5. Fix the top N potholes with the highest priority.\n",
    "    6. Leave the remaining potholes to be fixed in the future.\n",
    "    7. Fast-forward to the next day.\n",
    "    8. Repeat steps 2-7 until December 31, 2018.\n",
    "- **Note**: the simulation is a little slow. It may take ~10 minutes to test all algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = dict()\n",
    "\n",
    "for r in rules:\n",
    "    print(r)\n",
    "    results[r] = run_simulation(rule=r)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How long does each algorithm take to fix potholes?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Below, we see some summary statistics for how long it took to fix potholes with each algorithm.\n",
    "- Look at the row for the real algorithm. It shows a few things:\n",
    "    - The minimum number of days to fix a pothole is 0: some were fixed on the same day they were reported!\n",
    "    - 25% of potholes were fixed within 1 day, 50% were fixed within 6 days, and 75% were done within 22 days of the initial complaint. \n",
    "    - But the maximum number of days is 1194: that's over 3 years waiting! \n",
    "- Compare this to the first come first serve algorithm.\n",
    "    - The maximum wait time is only 94 days, but 50% potholes take more than 18 days."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "long = []\n",
    "for r in results.keys():\n",
    "    tmp = results[r][['pothole_id', 'days_open']].copy()\n",
    "    tmp['simulation'] = r\n",
    "    long.append(tmp)\n",
    "long = pd.concat(long)\n",
    "\n",
    "long.groupby(by='simulation').days_open.describe().round(1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Below, we can see histograms of the data. Note that the scale of x- and y-axes are different."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "long = []\n",
    "for r in results.keys():\n",
    "    tmp = results[r][['pothole_id', 'days_open']].copy()\n",
    "    tmp['simulation'] = r\n",
    "    long.append(tmp)\n",
    "long = pd.concat(long)\n",
    "\n",
    "ax = long[['simulation', 'days_open']].hist(by='simulation', bins=50, figsize=[14,10])\n",
    "for a1 in ax:\n",
    "    for a in a1:\n",
    "        a.set_xlabel(\"Days waiting for repair\")\n",
    "        a.set_ylabel(\"Number of Potholes\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-info\">\n",
    "\n",
    "#### Short answer 3:\n",
    "This table contains lots of information such as the minimum, maximum, and median wait time. \n",
    "- Which statistics presented in the table are important to consider? \n",
    "- Based on the statistics, which algorithm do you think the city should use, and why?\n",
    "    \n",
    "There is no right or wrong answer.\n",
    "Explain your reasoning in a 2-3 sentences.\n",
    "    \n",
    "</div >"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your answers here:**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 4: Unintended Consequences\n",
    "- We saw in the previous section that some potholes get fixed faster than others in each algorithm. The algorithms decided what order to fix potholes in based on seemingly neutral things, like what order they were reported or how busy the streets are.\n",
    "- One of the big challenges in algorithm design, however, is that algorithms with seemingly neutral inputs can sometimes still result in unfair or biaded outcomes.\n",
    "- In this section, we will look at whether our algorithms treat different Chicago neighborhoods fairly.\n",
    "\n",
    "#### Helper functions\n",
    "Don't worry about how this code works, just run it and scroll down."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "subset = ['pct_white', 'per_capita_income_', 'community_area_number',\n",
    "          'pct_rental_housing', 'total_population', 'median_age',\n",
    "          'percent_households_below_poverty', 'police_complaints_per_thousand']\n",
    "\n",
    "together = potholes[['creation_date', 'pothole_id', 'community_area_number', 'avg_traffic']]\n",
    "for r in results.keys():\n",
    "    tmp = results[r][['pothole_id', 'days_open']]\n",
    "    tmp.columns = ['pothole_id', r]\n",
    "    together = together.merge(tmp, on=['pothole_id'])\n",
    "together = together.merge(stats[subset], on='community_area_number', how='left')\n",
    "\n",
    "long2 = []\n",
    "for r in results.keys():\n",
    "    tmp = results[r][['pothole_id', 'days_open', 'community_area_number']].copy()\n",
    "    tmp['simulation'] = r\n",
    "    long2.append(tmp)\n",
    "long2 = pd.concat(long2)\n",
    "long2 = long2.groupby(by=['simulation', 'community_area_number']).median().reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Does everyone wait the same amount of time for repairs?\n",
    "- In the histgram below, we look at how long each community area (neighborhood) has to wait for potholes to be fixed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = long2[['simulation', 'days_open']].hist(by='simulation', bins =[0,1, 2, 3, 4, 5, 6,7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20,21,22,23, 24,25, 26, 27,28, 29, 30], figsize=(14,10), sharey=True,sharex=True)\n",
    "for a1 in ax:\n",
    "    for a in a1:\n",
    "        a.set_xlabel(\"Average (median) days waiting for repair\")\n",
    "        a.set_ylabel(\"Number of Community Areas\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-info\">\n",
    "    \n",
    "#### Short answer 4:\n",
    "One way to measure fairness is to see if every community has to wait about the same amount of time for repairs.\n",
    "- Based on the plot, which algorithm do you think is the most fair? Why?\n",
    "- Which algorithm seems to be the least fair? Why? \n",
    "    \n",
    "Explain in 1-2 sentences for each question.\n",
    "</div>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your answers here:**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Which communities are getting slower service?\n",
    "- Now that we know not all communities have to wait equally for repairs when using our algorithms, let's look at which ones get faster and slower service. \n",
    "- In the example below, we'll look at neighborhood income to see if **high and low income neighborhoods** are treated the same by each algorithm. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "outcome = 'per_capita_income_'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp = together[together[outcome].notna()].copy()\n",
    "tmp = tmp.groupby(outcome).median().reset_index().sort_values(by=outcome)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(15,15))\n",
    "for r in rules:\n",
    "    plt.scatter(x=tmp[outcome], y=tmp[r].values, alpha=.7)\n",
    "    y2 = lowess(endog=tmp[r], exog=tmp[outcome], is_sorted=True)[:,1]\n",
    "    plt.plot(tmp[outcome], y2, label=r, linewidth=4)\n",
    "\n",
    "ax.set_xlabel('Community area '+outcome)\n",
    "ax.set_ylabel('Median number of days')\n",
    "ax.set_title('Average time to fix potholes')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-info\">\n",
    "\n",
    "#### Short answer 5:\n",
    "\n",
    "- Which algorithms result in **equal** treatment across neighborhoods with different income? How do you know? \n",
    "- Which algorithms result in **different** treatment across neighborhoods with different income? How do you know? \n",
    "    \n",
    "Explain in 2-3 sentences for each question.\n",
    "</div>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your answers here:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Extreme wait times\n",
    "- Often in government, business, and other areas, we have specific goals we want to meet. For example, maybe we want to fix potholes within 7 days of hearing about them. We can measure our progress toward that goal as the fraction of potholes fixed within one week. Or maybe we decide waiting 2 months for a pothole repair is unacceptable and we want to keep track of where and how often that happens. \n",
    "- In the cell below, we look at exactly that by plotting the fraction of potholes in each neighborhood that go beyond our limit. \n",
    "- Right now, the number of days is set to 70. **Try it**: Change the `max_days` to other values like 30, 60, 100, re-run the cell, and see what happens. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_days = 70\n",
    "\n",
    "slow_repairs = together.copy()\n",
    "for r in rules:\n",
    "    slow_repairs[r] = (slow_repairs[r] > max_days).astype(int)\n",
    "tmp = slow_repairs[slow_repairs[outcome].notna()].copy()\n",
    "tmp = tmp.groupby(outcome).mean().reset_index().sort_values(by=outcome)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(15,15))\n",
    "for r in rules:\n",
    "    plt.scatter(x=tmp[outcome], y=tmp[r].values, alpha=.7)\n",
    "    y2 = lowess(endog=tmp[r], exog=tmp[outcome], is_sorted=True)[:,1] \n",
    "    plt.plot(tmp[outcome], y2, label=r, linewidth=4)\n",
    "\n",
    "ax.set_xlabel('Community area '+outcome)\n",
    "ax.set_ylabel('Fraction of potholes that exceed our limit')\n",
    "ax.set_title('Potholes that take more than '+str(max_days)+' days to fix')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## This code shows the same data a different way, by looking at the average income\n",
    "## for potholes that take a long time and comparing across neighborhoods\n",
    "\n",
    "result_t = []\n",
    "for r in rules:\n",
    "    tmp = dict()\n",
    "    tmp['algorithm'] = r\n",
    "    tmp['mean'] = slow_repairs[slow_repairs[r] > 0][outcome].mean()\n",
    "    tmp['sem'] = slow_repairs[slow_repairs[r] > 0][outcome].sem() * 1.96\n",
    "    result_t.append(tmp)\n",
    "result_t = pd.DataFrame(result_t).sort_values(by='mean').reset_index(drop=True).reset_index()\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(12,4))\n",
    "result_t.plot.scatter(x='mean', y='index', xerr='sem', ax=ax)\n",
    "plt.yticks(result_t.index, result_t.algorithm)\n",
    "ax.set_xlabel('Community area '+ outcome)\n",
    "ax.set_ylabel(None)\n",
    "ax.set_title('Average ' + outcome + ' for potholes that take >'+ str(max_days)+' days to fill')\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-info\">\n",
    "    \n",
    "#### Short answer 6:\n",
    "- Which algorithms seem to produce the most uneven wait times for different neighborhoods? How do you know? (1-2 sentences)\n",
    "- Is the difference in wait times more extreme if we define a \"long wait\" as something short, like 7 days, or something long, like 90 days? Or is it about the same? (2-3 sentences)\n",
    "</div>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your answers here:**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seciton 5: Try it Yourself\n",
    "- We can look at more than just income to see if neighborhoods are treated fairly. \n",
    "- In the code below, change `'per_capita_income_'` to a different variable to investigate something else.\n",
    "\n",
    "- Here are the options:\n",
    "    - `'pct_white'` Percent of residents who are White\n",
    "    - `'pct_rental_housing'` Percent of housing that is rented (not owner-occupied)\n",
    "    - `'total_population'` Number of people in the community area\n",
    "    - `'median_age'` Median age of residents\n",
    "    - `'percent_households_below_poverty'` Percent of households below the poverty line\n",
    "    - `'police_complaints_per_thousand'` Number of complaints about the Chicago Police Department, per thousand residents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "outcome = '???'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp = together[together[outcome].notna()].copy()\n",
    "tmp = tmp.groupby(outcome).median().reset_index().sort_values(by=outcome)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(15,15))\n",
    "for r in rules:\n",
    "    plt.scatter(x=tmp[outcome], y=tmp[r].values, alpha=.7)\n",
    "    y2 = lowess(endog=tmp[r], exog=tmp[outcome], is_sorted=True)[:,1]\n",
    "    plt.plot(tmp[outcome], y2, label=r, linewidth=4)\n",
    "\n",
    "ax.set_xlabel('Community area '+outcome)\n",
    "ax.set_ylabel('Median number of days')\n",
    "ax.set_title('Average time to fix potholes')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_days = 7\n",
    "\n",
    "\n",
    "slow_repairs = together.copy()\n",
    "for r in rules:\n",
    "    slow_repairs[r] = (slow_repairs[r] > max_days).astype(int)\n",
    "tmp = slow_repairs[slow_repairs[outcome].notna() & slow_repairs[r].notna()].copy()\n",
    "tmp = tmp.groupby(outcome).mean().reset_index().sort_values(by=outcome)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(15,15))\n",
    "for r in rules:\n",
    "    plt.scatter(x=tmp[outcome], y=tmp[r].values, alpha=.7)\n",
    "    y2 = lowess(endog=tmp[r], exog=tmp[outcome], is_sorted=True)[:,1] \n",
    "    plt.plot(tmp[outcome], y2, label=r, linewidth=4)\n",
    "\n",
    "ax.set_xlabel('Community area '+outcome)\n",
    "ax.set_ylabel('Fraction of potholes')\n",
    "ax.set_title('Potholes that take more than '+str(max_days)+' days to fix')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-info\">\n",
    "    \n",
    "#### Short Answer 7:\n",
    "- Play with the settings for the two charts above. \n",
    "\n",
    "- In the first chart, please try all the options for variables presented above other than `'per_capita_income_'`. What patterns do you see with each of the variable? Write a short paragraph. \n",
    "\n",
    "- In the second chart, in addition to specifying the variable you are looking at, you need to pick some value for the extreme wait time. What value do you choose? What patterns do you see with each of the variable? Write a short paragraph.\n",
    "\n",
    "- Does what you found influence your ideas about which algorithm the city should use? Write a few sentences saying how and why, or why not. \n",
    "    \n",
    "</div>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your answers here:**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What we learned:\n",
    "In this notebook, we have learned that different approaches give wait times of different min, max, median, 25% percentile and 75% percentile values (which is explored in Short Answer 3), and the wait time (extreme or not)  for different communities (which can be examined through various dimensions, as explored in Short Answers 4-7) is also different. \n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-warning\">\n",
    "\n",
    "\n",
    "### Reflection: Part 1\n",
    "\n",
    "Please rate each of the algorithm in terms of fairness and efficiency (you can use three levels in your rating: low, medium, high) and fill out the following table.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "| algorithms | income  | pct_white    | pct_rental   | population  | age | poverty | police |\n",
    "|---:|:-------------|:-----------|:------|:------|:------|:------|:------|\n",
    "| random - fairness |   fill cell1  |  fill cell2     |      |      |      |     |     |\n",
    "| random - efficiency                   |     |       |      |      |      |     |     |\n",
    "| real - fairness                       |     |       |      |      |      |     |     |\n",
    "| real - efficiency                     |     |       |      |      |      |     |     |\n",
    "| high traffic first - fairness         |     |       |      |      |      |     |     |\n",
    "| high traffic first - efficiency       |     |       |      |      |      |     |     |\n",
    "| first come first serve - fairness     |     |       |      |      |      |     |     |\n",
    "| first come first serve - efficiency   |     |       |      |      |      |     |     |\n",
    "| high complaint first - fairness       |     |       |      |      |      |     |     |\n",
    "| high complaint first - efficiency     |     |       |      |      |      |     |     |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-warning\">\n",
    "    \n",
    "### Reflection: Part 2\n",
    "\n",
    "\n",
    "- Considering all the analyses we have done and taking into consideration fairness and efficiency, which approach do you think is the best overall? Explain your reasoning in 5-6 sentences.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your answers here:**"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review\n",
    "- In this lab, we examined a **real case of algorithms** being used in society. We put ourselves in the shoes of the Chicago Department of Transportation and built several algorithms to help us repair city infrastructure. \n",
    "- These algorithms were all built with **neutral information** about the problem at hand: reports about potholes and road use.\n",
    "- We saw that some of our algorithms had **unintended concequences**. Not all communities got the same level of service when we simulated using these algorithms. This means we should worry about **fairness and equity**: is everyone being treated equally as a result of our algorithms? \n",
    "- After we saw that not every community was treated the same, we looked for **systematic bias** in the algorithms' results. We checked whether some algorithms treated rich and poor neighborhoods differently. \n",
    "- **The point to remember** is not which specific algorithm is best, or which way or measuring fairness is best. Instead, the major lesson from this lab is that algorithms sometimes have unintended concequences when they get used for making real world decisions. We told the algorithms nothing about the neighborhoods where potholes were located, but those neighborhoods still weren't all treated the same way by the algorithms. Thus, any time someone proposes to use an algorithm for making decisions like this, they should first simulate it and measure what effects it would have. Would the results be fair? Would they be fast enough to meet goals? These are important questions anyone can ask about the **outcomes** of algorithms, even if they don't understand how the algorithms work on the inside. \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}