leah-strava-heat-map.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "hidePrompt": false
   },
   "source": [
    "## Leah's Strava Heatmap Experiments 2016-2021\n",
    "\n",
    "Leah started running consistently in the Fall 2016 right around Thanksgiving."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "hideCode": false,
    "hidePrompt": false
   },
   "outputs": [],
   "source": [
    "# to convert -- CLI\n",
    "# jupyter nbconvert --to hide_code_html strava.ipynb\n",
    "import os\n",
    "from datetime import datetime\n",
    "from glob import glob\n",
    "\n",
    "import pandas as pd\n",
    "import geopandas as gpd\n",
    "import shapely\n",
    "from shapely.geometry import LineString\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import folium\n",
    "from folium.plugins import HeatMap\n",
    "\n",
    "from stravalib.client import Client\n",
    "from stravalib.util import limiter\n",
    "\n",
    "from process_strava import authenticate, swap\n",
    "from process_strava import get_activities, get_stream_data \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "client = Client(rate_limiter=limiter.DefaultRateLimiter())\n",
    "client = authenticate(\"strava-secrets.txt\", client)\n",
    "\n",
    "athlete_info = client.get_athlete()\n",
    "athlete_info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Athlete id=10295934 firstname='Leah' lastname='Wasser'>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "athlete_info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hideCode": false,
    "hidePrompt": false
   },
   "outputs": [],
   "source": [
    "# Get all activities which we use to get activity ideas for gps data below\n",
    "activities = client.get_activities()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "hidePrompt": false
   },
   "source": [
    "Below, I loop through each ativity and add it to a new list object which \n",
    "can then be turned into a dataframe for easy parsing below.\n",
    "\n",
    "Running this cell takes time as it's accessing a lot of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "daily_data_df_orig = get_activities(client)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary Data\n",
    "\n",
    "Below are several tables showing the data in aggregated form.\n",
    "Summarizing by \n",
    "\n",
    "1. Daily Actiities, \n",
    "2. Month, \n",
    "3. Cumulative by year\n",
    "\n",
    "This is the data used to create the plots below!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "hidePrompt": false
   },
   "source": [
    "# Spatial Data - The GPS Points Representing Runs on a Map \n",
    "\n",
    "This post helped me access this data with `stravalib`. Again stravalib is a Python \n",
    "package that makes it easier to access the strava API and associated data requests.\n",
    "\n",
    "https://medium.com/analytics-vidhya/accessing-user-data-via-the-strava-api-using-stravalib-d5bee7fdde17  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hideCode": false,
    "hidePrompt": false
   },
   "outputs": [],
   "source": [
    "# activity_number = 0\n",
    "# types = ['time', 'distance', 'latlng', 'altitude',\n",
    "#          'velocity_smooth', 'moving', 'grade_smooth']\n",
    "# #activity_data=client.get_activity_streams(df['id'][activity_number], types=types)\n",
    "\n",
    "# act = 4506113699\n",
    "# activity_data = client.get_activity_streams(act, types=types)\n",
    "# activity_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "hidePrompt": false
   },
   "source": [
    "## Grab spatial run data\n",
    "\n",
    "The effort below is rate-limited. I figured this out somewhere in another notebook.\n",
    "The better option is to save each year as a csv file and open the data to process it.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download Spatial Data By Year\n",
    "\n",
    "This operation hits significant rate limits so it's best to \n",
    "do it year by year. \n",
    "\n",
    "Strava limits used to be 600 every 15 mins. It may be less now but \n",
    "it is definitely less with the free account"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "types = ['time', 'distance', \n",
    "         'latlng', 'altitude',\n",
    "         'velocity_smooth', \n",
    "         'moving', 'grade_smooth']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting streaming data takes time. so i save each year and combine below to make sure i'm not redownloading data I already have. I could do this programatically in a more clean way..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data_2016_gdf = get_stream_data(2016, daily_data_df_orig, types)\n",
    "# data_2017_gdf = get_stream_data(2017, daily_data_df_orig, types)\n",
    "# data_2018_gdf = get_stream_data(2018, daily_data_df_orig, types)\n",
    "# data_2019_gdf = get_stream_data(2019, daily_data_df_orig, types)\n",
    "# data_2020_gdf = get_stream_data(client,\n",
    "#                                 2020,\n",
    "#                                 daily_data_df_orig,\n",
    "#                                 types)\n",
    "year_to_get = 2021\n",
    "# Get 2021 data\n",
    "data_2021_gdf = get_stream_data(client,\n",
    "                                year_to_get,\n",
    "                                daily_data_df_orig,\n",
    "                                types)\n",
    "\n",
    "# open 2021\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save all data as is as a pickle for now\n",
    "#import pickle\n",
    "\n",
    "# Create a list of all gdf objects\n",
    "# TODO - change this to open years 2016-2020 when 2020 is done and to\n",
    "# # then add 2021\n",
    "# all_years = [data_2020_gdf,\n",
    "#              data_2019_gdf,\n",
    "#              data_2018_gdf,\n",
    "#              data_2017_gdf,\n",
    "#              data_2016_gdf]\n",
    "\n",
    "# with open('strava_data', 'wb') as all_years_file:\n",
    "#     pickle.dump(all_years, all_years_file\n",
    "\n",
    "\n",
    "# Add 2021 gdf to data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The 2021 data will need to be added each time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open each of the spatial datasets and combine to create a single gdf\n",
    "all_shpfiles = sorted(glob(os.path.join(\"data\", \"*.shp\")))\n",
    "\n",
    "# Open each shapefile & concat for plotting\n",
    "all_years = []\n",
    "for afile in all_shpfiles[0:1]:\n",
    "    all_years.append(gpd.read_file(afile))\n",
    "\n",
    "all_years_gdf = pd.concat(all_years,\n",
    "                          ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_years_gdf.tail(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2021 data plot\n",
    "f, ax = plt.subplots(figsize=(12, 6))\n",
    "data_2021_gdf.plot(ax=ax)\n",
    "ax.set(title=\"Ugly plot of 2021 spatial data\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f, ax = plt.subplots(figsize=(12, 6))\n",
    "all_years_gdf.plot(ax=ax)\n",
    "ax.set(title=\"Ugly plot of all run spatial data\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hideCode": false,
    "hidePrompt": false
   },
   "outputs": [],
   "source": [
    "# data_2020_gdf = gpd.read_file(all_shpfiles[4])\n",
    "# # Plot of 2020 data only \n",
    "# f, ax = plt.subplots()\n",
    "# data_2020_gdf.plot(ax=ax)\n",
    "# ax.set(title=\"Leah's 2020 Adventures\")\n",
    "# plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": false,
    "hidePrompt": false
   },
   "source": [
    "## Interactive Map of All Activities\n",
    "\n",
    "Below is an interactive plot of my data from 2020!\n",
    "\n",
    "TODO's:\n",
    "\n",
    "1. Add other years of data\n",
    "2. Add label popups maybe?? tbd as lots of overlapping lines here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simplify 202 data\n",
    "# data_simp = data_2021_gdf.copy()\n",
    "# data_simp[\"geometry\"] = data_simp.simplify(tolerance=.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot 2020 data only\n",
    "# m = folium.Map([39.95, -105.2],\n",
    "#                zoom_start=11)\n",
    "\n",
    "# for index, row in data_simp.iterrows():\n",
    "#     folium.PolyLine(\n",
    "#         row.xy,\n",
    "#         line_weight=3,\n",
    "#         color=\"purple\",\n",
    "#         opacity=.5\n",
    "#     ).add_child(folium.Popup(str(row.activity_id))).add_to(m)\n",
    "\n",
    "# m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2021 data\n",
    "all_years_gdf.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, i'm grabbing only the 2021 data and creating a heat map.\n",
    "there are bunch of nuances to consider here\n",
    "\n",
    "1. error in gps - you can see each run points may be close but slightly \"off\" yielding a less accurate and less clean heat map\n",
    "2. selecting the gradient of colors. \n",
    "\n",
    "lots to play with here to make the final map a bit cleaner. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_points(gdf):\n",
    "    # swap x and y for folium because it wants lat first not lon\n",
    "    gdf[\"geometry\"] = gdf.geometry.map(swap)\n",
    "\n",
    "    # Get points\n",
    "\n",
    "    gdf['points'] = gdf.apply(\n",
    "        lambda x: [y for y in x['geometry'].coords], axis=1)\n",
    "\n",
    "\n",
    "    line_list = []\n",
    "    for aline in gdf.points:\n",
    "        line_list.append([list(tpl) for tpl in aline])\n",
    "\n",
    "    # now flatten it - this should be all runs in 2021\n",
    "    return [item for sublist in line_list for item in sublist]\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "points_2021 = clean_points(data_2021_gdf)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Heatmap for 2021 Running Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate a heat map for 2021 data\n",
    "m = folium.Map([39.96, -105.27], zoom_start=14)\n",
    "HeatMap(points_2021,\n",
    "        name=\"runs\",\n",
    "        radius=2,\n",
    "        blur=3,\n",
    "        gradient={0.1: 'thistle',\n",
    "                  0.3: 'purple',\n",
    "                  0.7: 'orange',\n",
    "                  1: 'indigo'}).add_to(m)\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_points = clean_points(all_years_gdf)\n",
    "len(all_points)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Because this takes a while to process, it could make sense to process and export as pickle\n",
    "# all_years_gdf_points = all_years_gdf.copy()\n",
    "\n",
    "# # swap x and y for folium because it wants lat first not lon\n",
    "# all_years_gdf_points[\"geometry\"] = all_years_gdf_points.geometry.map(swap)\n",
    "\n",
    "# # Get points\n",
    "\n",
    "# all_years_gdf_points['points'] = all_years_gdf_points.apply(\n",
    "#     lambda x: [y for y in x['geometry'].coords], axis=1)\n",
    "\n",
    "\n",
    "# line_list = []\n",
    "# for aline in all_years_gdf_points.points:\n",
    "#     line_list.append([list(tpl) for tpl in aline])\n",
    "\n",
    "# # now flatten it - this should be all runs in 2021\n",
    "# flat_list = [item for sublist in line_list for item in sublist]\n",
    "#all_points[4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save all points as a pickle - maybe process by year and save it out?\n",
    "#all_points[0:400]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate a heat map of all data across time - missing data i believe.\n",
    "m = folium.Map([39.96, -105.27], zoom_start=14)\n",
    "HeatMap(all_points, \n",
    "        name=\"runs\",\n",
    "        radius=4, blur=5,\n",
    "        gradient = {0.1: 'thistle', \n",
    "                   0.3: 'lime',\n",
    "                   0.7: 'orange', \n",
    "                   1: 'indigo'}).add_to(m)\n",
    "\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simplify all data\n",
    "all_years_simp = all_years_gdf.copy()\n",
    "all_years_simp[\"geometry\"] = all_years_simp.simplify(tolerance=.01)\n",
    "\n",
    "all_years_simp.crs = \"EPSG:4326\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # This is some experiment with a chloropleth ??\n",
    "# m = folium.Map([39.95, -105.2],\n",
    "#                zoom_start=11) \n",
    "# folium.Choropleth(\n",
    "#     all_years_simp,\n",
    "#     line_weight=3,\n",
    "#     line_color='blue'\n",
    "# ).add_to(m)\n",
    " \n",
    "# m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Patience - this is parsing a LOT of activities\n",
    "# # TODO - this doesn't run right now. \n",
    "# m = folium.Map([39.95, -105.2],\n",
    "#                zoom_start=11)\n",
    "\n",
    "# for aline in all_years_simp.xy:\n",
    "#     folium.PolyLine(\n",
    "#         aline,\n",
    "#         line_weight=3,\n",
    "#         color=\"purple\",\n",
    "#         opacity=.5\n",
    "#     ).add_to(m)\n",
    "\n",
    "# m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hideCode": false,
    "hidePrompt": false
   },
   "outputs": [],
   "source": [
    "# # this is parsing a LOT of activities \n",
    "# # TODO also doesn't run\n",
    "# map_data.crs = \"EPSG:4326\"\n",
    "\n",
    "# m = folium.Map([39.95, -105.2],\n",
    "#                zoom_start=11)\n",
    "\n",
    "# for aline in map_data.xy:\n",
    "#     folium.PolyLine(\n",
    "#         aline,\n",
    "#         line_weight=3,\n",
    "#         color=\"purple\",\n",
    "#         opacity=.5\n",
    "#     ).add_to(m)\n",
    "\n",
    "# m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "# Convert to html with code hidden!\n",
    "!jupyter nbconvert --to html --TemplateExporter.exclude_input=True  leah-summary-strava-heat-map.ipynb\n",
    "\n",
    "# https://github.com/jupyter/nbconvert/issues/944 <- export issues of plotly\n",
    "# Fix == setting the save state setting -https://github.com/jupyter-widgets/ipywidgets/issues/1632#issuecomment-510138573"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Hide code",
  "hide_code_all_hidden": false,
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "288px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}