weather_forecast.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating an interactive climate map in Plotly (& Geopandas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  \n",
    "- **DATA**:\n",
    "\n",
    "     - The data for this particular lesson is **synthetic** / **fake!** It was originally scraped from [Berkeley Earth](http://berkeleyearth.lbl.gov/country-list/), but then regenerated based a simple polynomial trend and monthly seasonality modeled from the original data source & injected with random noise. Hence, the data in this repo is somewhat realistic but not authentic.\n",
    "\n",
    "\n",
    "- **INSTALLATION**:\n",
    "\n",
    "    - **Make sure you already have plotly and geopandas installed**! (e.g. `pip` or `conda`), e.g.:\n",
    "\n",
    "        - `pip install plotly`\n",
    "    \n",
    "        - `conda install -c conda-forge geopandas --yes`\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read in historical temperature data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Read in the data with pandas\n",
    "import pandas as pd\n",
    "\n",
    "df =  pd.read_csv('./data/TG_STAID002759.txt',header=14, index_col=1, parse_dates=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from format import format_data_frame\n",
    "df = format_data_frame(df, set_index_time_step=False)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Group / aggregate the temperature anomaly data by country, year\n",
    "- For simplicity, we're only interested in yearly averages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['country'] = 'Germany'\n",
    "\n",
    "df['year'] = df.index.year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df = df.groupby(['country', 'year']).mean()\n",
    "df\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#save the group result as a dataframe and reset the index so that it becomes easier later to subselect a year\n",
    "df = df.reset_index()\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read in the geographic data (geometric shapes of all countries in the world) \n",
    "- Hint: Use GeoPandas\n",
    "    - What is a **Shape file (.shp)?**\n",
    "        - https://en.wikipedia.org/wiki/Shapefile#Shapefile_shape_format_(.shp)\n",
    "- Publicly available GIS data downloaded from *Natural Earth*: https://www.naturalearthdata.com/downloads/110m-cultural-vectors/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Read in the shapefile with geopandas\n",
    "import geopandas as gpd\n",
    "gdf = gpd.read_file('./data/ne_110m_admin_0_countries.shp')\n",
    "# type(gdf['geometry'])\n",
    "gdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Select the ADMIN and geometry column\n",
    "gdf = gdf[['ADMIN', 'geometry']]\n",
    "gdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Time for Visualization with Ploty\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot data on a map for a single year (we can make it interactive later)\n",
    "- Import the **\"Plotly Express\"** module (usually imported as ``px``), which contains high-level functions that can create entire figures at once. Plotly Express is a built-in part of the plotly library, and is the recommended starting point for creating most common figures. \n",
    "    - \"Plotly Express is a terse, consistent, high-level API for creating figures.\" This means that it's a lot easier to use (less code) than the alternative option, which is to manually create, manipulate, configure, and render the underlying \"Graph Objects\" yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import plotly.express as px"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generate a GeoJSON string for a single year.\n",
    "- The Plotly library (as well as many other JavaScript-based mapping libraries) requires the data to be in GeoJSON format.\n",
    "    - **HINT**: How can you convert a GeoDataFrame into a GeoJSON? Think of how you might do this in regular pandas.\n",
    "- Let's use the year 2013 as an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df['temp_c'] = df['TG'].round(2)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_2013 = df[df['year']==2013]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "# geojson"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert the GeoJSON string to an actual python dictionary\n",
    "- Strangely enough, the GeoJSON object that is generated from a GeoDataFrame is represented as a string.\n",
    "- Some plotting libraries can parse the string directly, while others (e.g. Plotly) require the GeoJSON data in python to be an actual dictionary.\n",
    "- We can use the built-in ``json`` library to convert this string into a dictionary, so that Plotly can understand it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "geojson = json.loads(geojson)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# geojson['features'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP 6: Generate an interactive choropleth map of the data for a single year\n",
    "- Use the convenient ``px.choropleth()`` method to generate the interactive figure, which you can even see directly in your Jupyter Notebook!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = px.choropleth_mapbox(\n",
    "    mapbox_style='white-bg',      # see other styles: https://plotly.com/python/mapbox-layers/\n",
    "    data_frame = df,                  # dataframe that contains all years you can use gdf or df\n",
    "    geojson=geojson,                   # we can still use the JSON data from 2000, assuming the countries are the same over time\n",
    "    featureidkey='properties.ADMIN',      # name of JSON key within the \"properties\" value that contains country names\n",
    "    locations='country',                    # name of the dataframe column that contains country names\n",
    "    color='temp_c',                       # name of the dataframe column that contains numerical data you want to display (Like hue)\n",
    "    center={\"lat\": 51.1657, \"lon\": 10.4515},\n",
    "    zoom=1,\n",
    "    color_continuous_scale=\"thermal\",    # see other options: https://plotly.com/python/builtin-colorscales/\n",
    "    range_color=(-3, 3),\n",
    "    #opacity =0.5\n",
    ")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP 7: Export the figure to an HTML file, so you can open it in your web browser!\n",
    "- The reason the figure is interactive (e.g. you can zoom around, it has hover effects, etc.) is because there's a of front-end (i.e. client-side) JavaScript code that Plotly creates for you automatically.\n",
    "- Use the ``.write_html()`` method to export the file and open it up in a web browser.\n",
    "    - Bonus: Open the ``.html`` file in a text editor and see if you can understand any of it :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig.write_html(\"2013_map.html\", include_plotlyjs='cdn')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#bash command to open the file (should use your web browser by default)\n",
    "\n",
    "#google-chrome 2013_map.html # for linux\n",
    "# !open 2013_map.html # for mac\n",
    "#!start 2013_map.html # for windows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP 8: Add an interactive slider / animation to toggle between multiple years in a single file.\n",
    "- For this, we can use the convenient ``px.choropleth_mapbox()`` method that already has some built-in, front-end animation features right out of the box.\n",
    "- This time, pass the **entire DataFrame** (not just a single year slice) as an argument into the function.\n",
    "    - **WARNING**: While the HTML script seems to have fairly good performance right out-of-the-box, the resulting HTML file is pretty massive (~50 MB on hard disk), so you might want to delete the file afterwards!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig2 = px.choropleth_mapbox(\n",
    "    mapbox_style='white-bg',      # see other styles: https://plotly.com/python/mapbox-layers/\n",
    "    data_frame = df,                  # dataframe that contains all years you can use gdf or df\n",
    "    geojson=geojson,                   # we can still use the JSON data from 2000, assuming the countries are the same over time\n",
    "    featureidkey='properties.ADMIN',      # name of JSON key within the \"properties\" value that contains country names\n",
    "    locations='country',                    # name of the dataframe column that contains country names\n",
    "    color='temp_c',                       # name of the dataframe column that contains numerical data you want to display\n",
    "    center={\"lat\": 51.1657, \"lon\": 10.4515},\n",
    "    zoom=2,\n",
    "    animation_frame='year',             # name of dataframe column that you want to make frames of\n",
    "    animation_group='country',   \n",
    "    color_continuous_scale=\"thermal\",    # see other options: https://plotly.com/python/builtin-colorscales/\n",
    "    range_color=(-3, 3),\n",
    "    color_continuous_midpoint=0,\n",
    "    opacity =0.5)\n",
    "\n",
    "fig2.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig2.write_html(\"all_years_interactive.html\", include_plotlyjs='cdn')\n",
    "#this could take up to minute to generate -- file is very LARGE!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!open all_years_interactive.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig2.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next Steps / Bonus / Further Exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you got everything working (congratulations!) and enjoyed using a high-level, interactive, JavaScript-based visualization library like Plotly, then you should have a look at [Dash](https://plotly.com/dash/).\n",
    "- What is **Dash**?\n",
    "    - As of 2020, Dash is a super popular framework for building entire data visualization applications, not just \"static\" HTML files.\n",
    "    - You can think of it as an extension to Plotly. It's actually built on top of 3 libraries:\n",
    "        - *Plotly* itself (Dash was actually made by the same developers);\n",
    "        - *Flask* (a web server framework in python); and\n",
    "        - *React.js* (a powerful JavaScript library for building really nice user interfaces).\n",
    "    - It's basically a visualization library that can be used to make entire, sophisticated dashboards that are powered by a back-end web server. Here are some [example of machine-learning visualization apps](https://dash-gallery.plotly.host/Portal/?search=[Machine%20Learning]) built using Dash."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to get started with Dash?\n",
    "- It's a python library, so a simple ``pip install dash`` should work fine.\n",
    "- I'd love to eventually include a simple step-by-step example as part of this tutorial, but that's still a work in progress..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the meanwhile, here are places to go for code snippets and inspiration:\n",
    "- [Dash Sample App](https://github.com/plotly/dash-sample-apps/tree/master/apps/dash-opioid-epidemic) *(from the Plotly developers)*. Check out all the other [amazing stuff you can build with Plotly and Dash](https://dash-gallery.plotly.host/Portal/)!\n",
    "- The [great code snippets / examples from the Plotly website](https://plotly.com/python/maps/)\n",
    "- The [Capital Bike Dashboard Project](https://github.com/GesaJo/Capital-Bike-Dashboard) by [Gesa Johannsen](https://github.com/GesaJo), fellow Data Science Instructor @ Spiced Academy.\n",
    "- *Towards Data Science Article* by [Jun Ye](https://github.com/Perishleaf): [Build an Interactive Climate Choropleth Map with Plotly and Dash](https://towardsdatascience.com/build-an-interactive-choropleth-map-with-plotly-and-dash-1de0de00dce0)\n",
    "    - and the source code: https://github.com/Perishleaf/data-visualisation-scripts/tree/master/dash_project_medium"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**:\n",
    "\n",
    "Keep in mind that these are massive, powerful libraries that by no means are you expected to learn as part of this course! This tutorial / exercise is more meant for you to get a taste of what's possible in terms of interactive web-based visualization in Python. Learning these libraries is, quite frankly, a matter of just following a couple tutorials, trying it out yourself for a project that interests you, and sometimes digging deep into documentation and/or StackOverflow. This could be a worthwhile time investment as part of a final project!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.4 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4 (default, Aug 13 2019, 15:17:50) \n[Clang 4.0.1 (tags/RELEASE_401/final)]"
  },
  "vscode": {
   "interpreter": {
    "hash": "e6c8ece07e9164d2a31687d29b107d2624b25817155fdef291395eab91afaa05"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}