BNHM-integrateData-2020.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below to setup the notebook (_shift+return in cell to run_ or Press Run button in the menu)\n",
    "Do you see a number in the left margin of the cell below? If so, click on _Kernel->Restart and Clear Output_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install --no-cache-dir shapely\n",
    "!pip install -U folium\n",
    "\n",
    "%matplotlib inline\n",
    "import os\n",
    "import time\n",
    "import folium\n",
    "from datetime import datetime\n",
    "from shapely.geometry import Point, mapping\n",
    "from shapely.geometry.polygon import Polygon\n",
    "import matplotlib as mpl\n",
    "from matplotlib.collections import PatchCollection\n",
    "import matplotlib.pyplot as plot\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import requests\n",
    "from datascience import *\n",
    "from shapely import geometry as sg, wkt\n",
    "from scripts.espm_module import *\n",
    "import json\n",
    "import random\n",
    "from IPython.core.display import display, HTML\n",
    "import ipywidgets as widgets\n",
    "plot.style.use('seaborn')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ESPM / IB 105 Natural History Museums and Data Science\n",
    "\n",
    "The goal of this notebook is to access and integrate diverse data sets to visualize correlations and discover patterns to address questions of species’ responses to environmental change. We will use programmatic tools to show how to use Berkeley resources such as the biodiversity data from biocollections and online databases, field stations, climate models, and other environmental data.\n",
    "\n",
    "Before we begin analyzing and visualizing biodiversity data, this introductory notebook will help familiarize you with the basics of programming in Python.\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "0 - [Jupyter Notebooks](#jupyter)\n",
    "    \n",
    "1 - [Python Basics](#basics)\n",
    "\n",
    "2 - [GBIF API](#gbif)\n",
    "\n",
    "3 - [Comparing California Oak Species](#oak)\n",
    "\n",
    "4 - [Cal-Adapt API](#adapt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 0: Our Computing Environment, Jupyter notebooks  <a id='jupyter'></a>\n",
    "This webpage is called a Jupyter notebook. A notebook is a place to write programs and view their results. \n",
    "\n",
    "### Text cells\n",
    "In a notebook, each rectangle containing text or code is called a *cell*.\n",
    "\n",
    "Text cells (like this one) can be edited by double-clicking on them. They're written in a simple format called [Markdown](http://daringfireball.net/projects/markdown/syntax) to add formatting and section headings.  You don't need to learn Markdown, but you might want to.\n",
    "\n",
    "After you edit a text cell, click the \"run cell\" button at the top that looks like ▶| to confirm any changes. (Try not to delete the instructions of the lab.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code cells\n",
    "Other cells contain code in the Python 3 language. Running a code cell will execute all of the code it contains.\n",
    "\n",
    "To run the code in a code cell, first click on that cell to activate it.  It'll be highlighted with a little green or blue rectangle.  Next, either press ▶| or hold down the `shift` key and press `return` or `enter`.\n",
    "\n",
    "Try running this cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Hello, World!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And this one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"\\N{WAVING HAND SIGN}, \\N{EARTH GLOBE ASIA-AUSTRALIA}!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Python basics <a id='basics'></a>\n",
    "Before getting into the more high level analyses we will do on the GBIF and Cal-Adapt data, we need to cover a few of the foundational elements of programming in Python.\n",
    "\n",
    "#### A. Expressions\n",
    "The departure point for all programming is the concept of the __expression__. An expression is a combination of variables, operators, and other Python elements that the language interprets and acts upon. Expressions act as a set of instructions to be fed through the interpreter, with the goal of generating specific outcomes. See below for some examples of basic expressions. Keep in mind that most of these just map to your understanding of mathematical expressions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'you' + ' and me'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will notice that only the last line in a cell gets printed out. If you want to see the values of previous expressions, you need to call the `print` function on that expression. Functions use parenthesis around their parameters, just like in math!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(2 + 2)\n",
    "\n",
    "print('you' + ' and I')\n",
    "\n",
    "print(12 ** 2)\n",
    "\n",
    "print(6 + 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### B. Variables\n",
    "In the example below, `a` and `b` are Python objects known as __variables__. We are giving an object (in this case, an `integer` and a `float`, two Python data types) a name that we can store for later use. To use that value, we can simply type the name that we stored the value as. Variables are stored within the notebook's environment, meaning stored variable values carry over from cell to cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = 4\n",
    "b = 10/5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that when you create a variable, unlike what you previously saw with the expressions, it does not print anything out.\n",
    "\n",
    "We can continue to perform mathematical operations on these variables, which are now placeholders for what we've assigned:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('a + b')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### C. Lists\n",
    "The following few cells will introduce the concept of __lists__.\n",
    "\n",
    "A list is an ordered collection of objects. They allow us to store and access groups of variables and other objects for easy access and analysis. Check out this [documentation](https://www.tutorialspoint.com/python/python_lists.htm) for an in-depth look at the capabilities of lists.\n",
    "\n",
    "To initialize a list, you use brackets. Putting objects separated by commas in between the brackets will add them to the list. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# an empty list\n",
    "lst = []\n",
    "print(lst)\n",
    "\n",
    "# reassigning our empty list to a new list\n",
    "lst = [1, 3, 6, 'lists', 'are' 'fun', 4]\n",
    "print(lst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To access a value in the list, put the index of the item you wish to access in brackets following the variable that stores the list. Lists in Python are zero-indexed, so the indicies for `lst` are 0, 1, 2, 3, 4, 5, and 6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Elements are selected like this:\n",
    "example = lst[2]\n",
    "\n",
    "# The above line selects the 3rd element of lst (list indices are 0-offset) and sets it to a variable named example.\n",
    "print(example)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### D. Dictonaries\n",
    "\n",
    "Dictionaries are `key`-`value` pairs. Just like a word dictinary, you have a key that will index a specific definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_dict = {'python': 'a large heavy-bodied nonvenomous constrictor snake occurring throughout the Old World tropics.'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get a `value` back out by indexing the `key`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_dict['python']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But like real dictionaries, there can be more than one definition. You can keep a `list`, or even another dictionary within a specific `key`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_dict = {'python': ['a large heavy-bodied nonvenomous constrictor snake occurring throughout the Old World tropics.',\n",
    "                      'a high-level general-purpose programming language.']}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can index the `list` after the `key`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_dict['python'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_dict['python'][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Part 2: GBIF API<a id='gbif'></a>\n",
    "\n",
    "The Global Biodiversity Information Facility has created an API that we can use to get data about different species at the [GBIF Web API](http://www.gbif.org/developer/summary).\n",
    "\n",
    "You can think of a Web API call as a fancy URL, what do you think the end of this URL means:\n",
    "\n",
    "http://api.gbif.org/v1/occurrence/search?year=1800,1899\n",
    "\n",
    "If you're guessing that it limits the search to the years 1800-1899, you're right! Go ahead and click the URL above. You should see something like this:\n",
    "\n",
    "```\n",
    "{\"offset\":0,\"limit\":20,\"endOfRecords\":false,\"count\":5711947,\"results\":[{\"key\":14339704,\"datasetKey\":\"857aa892-f762-11e1-a439-00145eb45e9a\",\"publishingOrgKey\":\"6bcc0290-6e76-11db-bcd5-b8a03c50a862\",\"publishingCountry\":\"FR\",\"protocol\":\"BIOCASE\",\"lastCrawled\":\"2013-09-07T07:06:34.000+0000\",\"crawlId\":1,\"extensions\":{},\"basisOfRecord\":\"OBSERVATION\",\"taxonKey\":2809968,\"kingdomKey\":6,\"phylumKey\":7707728,\"classKey\":196,\"orderKey\":1169,\"familyKey\":7689,\"genusKey\":2849312,\"speciesKey\":2809968,\"scientificName\":\"Orchis militaris L.\",\"kingdom\":\"Plantae\",\"phylum\":\"Tracheophyta\",\"order\":\"Asparagales\",\"family\":\"Orchidaceae\",\"genus\":\"Orchis\",\"species\":\"Orchis \n",
    "```\n",
    "\n",
    "It might look like a mess, but it's not! This is actually very structured data, and can easily be put into a table like format, though often programmers don't do this because it's just as easy to keep it as is.\n",
    "\n",
    "You might be able to pick out the curly braces `{` and think this it's a dictionary. You'd be right, except in this format we call it [JSON](https://en.wikipedia.org/wiki/JSON).\n",
    "\n",
    "---\n",
    "\n",
    "## *Argia arioides*\n",
    "\n",
    "![argia arioides](https://upload.wikimedia.org/wikipedia/commons/thumb/b/bd/Argia_agrioides-Male-1.jpg/220px-Argia_agrioides-Male-1.jpg)\n",
    "\n",
    "When performing data analysis, it is always important to define a question that you seek the answer to. *The goal of finding the answer to this question will ultimately drive the queries and analysis styles you choose to use/write.*\n",
    "\n",
    "For this example, we are going to ask: **where have [*Argia agrioides*](https://www.google.com/search?q=Argia+agrioides&rlz=1C1CHBF_enUS734US734&source=lnms&tbm=isch&sa=X&ved=0ahUKEwji9t29kNTWAhVBymMKHWJ-ANcQ_AUICygC&biw=1536&bih=694) (the California Dancer dragonfly) been documented? Are there records at any of our field stations?**\n",
    "\n",
    "The code to ask the API has already been written for us! This is often the case with programming, someone has already written the code, so we don't have to. We'll just set up the `GBIFRequest` object and assign that to the variable `req`, short for \"request\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "req = GBIFRequest()  # creating a request to the API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, so how do we make searches? We can use a Python `dictionary` to create our query parameters. We'll ask for the `scientificName` of the California Dancer (*Argia arioides*):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'scientificName': 'Argia agrioides'}  # setting our parameters (the specific species we want)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the parameters, we can feed this to our `req` variable to get back all the pages of data. We'll then make sure that each record has a `decimalLatitude`, otherwise we'll thow it out for now. Lastly, we'll print out the first five records:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'scientificName': 'Argia agrioides'}  # setting our parameters (the specific species we want)\n",
    "pages = req.get_pages(params)  # using those parameters to complete the request\n",
    "records = [rec for page in pages for rec in page['results'] if rec.get('decimalLatitude')]  # sift out valid records\n",
    "records[:5]  # print first 5 records"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataFrames\n",
    "\n",
    "JSON is great, but it might be conceptually easier to make this a table. We'll use the popular [`pandas`](http://pandas.pydata.org/) Python library. In `pandas`, a DataFrame is a table that has several convenient features. For example, we can access the columns of the table like we would `dict`ionaries, and we can also treat the columns and rows themselves as Python `list`s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df = pd.DataFrame(records)   # converts the JSON above to a dataframe\n",
    "records_df.head()  # prints the first five rows of the dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since each column (or row) above can be thought of as a `list`, that means we can use list functions to interact with them! One such function is the `len` function to get the number of elements in a `list`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(records_df.columns), len(records_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many columns are there? (hint: there are 118) How many records/rows? (hint: it's the other value)! That's a lot of information. What variables do we have in the columns?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use two methods from `pandas` to do a lot more. The `value_counts()` method will tabulate the frequency of the row value in a column, and the `plot.barh()` will plot us a horizontal bar chart:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df['country'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df['country'].value_counts().plot.barh()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df['county'].value_counts().plot.barh()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df['basisOfRecord'].value_counts().plot.barh()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df['institutionCode'].value_counts().plot.barh()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `groupby()` method allows us to count based one column based on another, and then color the bar chart differently depending on a variable of our choice:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df.groupby([\"collectionCode\", \"basisOfRecord\"])['basisOfRecord'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df.groupby([\"collectionCode\", \"basisOfRecord\"])['basisOfRecord'].count().unstack().plot.barh(stacked=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can use `plot.hist()` to make a histogram:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df['elevation'].plot.hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "**Exercise**: Edit the code below to search for a different species you're interested in, then adapt some of the graphing cells above to explore your data!\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_req = GBIFRequest()  # creating a request to the API\n",
    "my_params = {'scientificName': 'Taricha rivularis'}  # setting our parameters (the specific species we want)\n",
    "my_pages = my_req.get_pages(my_params)  # using those parameters to complete the request\n",
    "my_records = [rec for page in my_pages for rec in page['results'] if rec.get('decimalLatitude')]  # sift out valid records\n",
    "my_records_df = pd.DataFrame(my_records)  # make a dataframe\n",
    "my_records_df.tail()  # print last 5 rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_records_df['year'].plot.hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### Mapping\n",
    "\n",
    "Now we can map our latitude and longitude points. We'll want to color code by `collectionCode` so we can see which collection made the observation or has the specimen. We'll use a function that does this automatically, but it will randomly generate a color, so if you get a lot of a similar color maybe run the cell a couple times!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "color_dict, html_key = assign_colors(records_df, \"collectionCode\")\n",
    "display(HTML(html_key))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can map the points!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "mapa = folium.Map(location=[37.359276, -122.179626], zoom_start=5) # Folium is a useful library for generating\n",
    "                                                                   # Google maps-like map visualizations.\n",
    "for r in records_df.iterrows():\n",
    "    lat = r[1]['decimalLatitude']\n",
    "    long = r[1]['decimalLongitude']\n",
    "    folium.CircleMarker((lat, long), color=color_dict[r[1]['collectionCode']], popup=r[1]['basisOfRecord']).add_to(mapa)\n",
    "mapa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "To get the boundries for all the reserves, we will need to send a request to get GeoJSON, which is a format for encoding a variety of geographic data structures. With this code, we can request GeoJSON for all reserves and plot ocurrences of the species. First we'll assign the API URL that has the data to a new variable `url`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://ecoengine.berkeley.edu/api/layers/reserves/features/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we make the requests just like we did earlier through the GBIF:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reserves = requests.get(url, params={'page_size': 30}).json()\n",
    "reserves"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you look closely, this is just bounding boxes for the latitude and longitude of the reserves.\n",
    "\n",
    "There are some reserves that the EcoEngine didn't catch, we'll add the information for \"Blodgett\", \"Hopland\", and \"Sagehen\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "station_urls = {\n",
    "    'Blodgett': 'https://raw.githubusercontent.com/BNHM/spatial-layers/master/wkt/BlodgettForestResearchStation.wkt',\n",
    "    'Hopland': 'https://raw.githubusercontent.com/BNHM/spatial-layers/master/wkt/HoplandResearchAndExtensionCenter.wkt',\n",
    "    'Sagehen': 'https://raw.githubusercontent.com/BNHM/spatial-layers/master/wkt/SagehenCreekFieldStation.wkt'\n",
    "}\n",
    "reserves['features'] += [{'type': \"Feature\", 'properties': {\"name\": name}, 'geometry': mapping(wkt.loads(requests.get(url).text))}\n",
    "                for name, url in station_urls.items()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see all the station names by indexing the `name` value for the reserves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[r['properties']['name'] for r in reserves['features']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can send this `geojson` directly to our mapping library `folium`. You'll have to zoom in, but you should see blue outlines areas, there are the reserves!:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mapb = folium.Map(location=[37.359276, -122.179626], zoom_start=5) \n",
    "\n",
    "wkt = folium.features.GeoJson(reserves)\n",
    "mapb.add_child(wkt)\n",
    "for r in records_df.iterrows():\n",
    "    lat = r[1]['decimalLatitude']\n",
    "    long = r[1]['decimalLongitude']\n",
    "    folium.CircleMarker((lat, long), color=color_dict[r[1]['collectionCode']], popup=r[1]['basisOfRecord']).add_to(mapb)\n",
    "mapb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "**Exercise**: Edit the code below where you see `...` to map your selection for `scientificName` from `my_records_df` (PS. no need to add single quotes)\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "color_dict, html_key = assign_colors(my_records_df, \"collectionCode\")\n",
    "display(HTML(html_key))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mapc = folium.Map([37.359276, -122.179626], zoom_start=5)\n",
    "\n",
    "points = folium.features.GeoJson(reserves)\n",
    "mapc.add_child(points)\n",
    "for r in my_records_df.iterrows():\n",
    "    lat = r[1]['decimalLatitude']\n",
    "    long = r[1]['decimalLongitude']\n",
    "    folium.CircleMarker((lat, long), color=color_dict[r[1]['collectionCode']]).add_to(mapc)\n",
    "mapc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "We can also find out which stations have how many *Argia argrioides*. First we'll have to add a column to our `DataFrame` that makes points out of the latitude and longitude coordinates:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_point(row):\n",
    "    return Point(row['decimalLongitude'], row['decimalLatitude'])\n",
    "\n",
    "records_df[\"point\"] = records_df.apply(lambda row: make_point (row),axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can write a little function to check whether that point is in one of the stations, and if it is, we'll add that station in a new column called `station`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def in_station(reserves, row):\n",
    "    \n",
    "    reserve_polygons = []\n",
    "\n",
    "    for r in reserves['features']:\n",
    "        name = r['properties']['name']\n",
    "        poly = sg.shape(r['geometry'])\n",
    "        reserve_polygons.append({\"id\": name,\n",
    "                                 \"geometry\": poly})\n",
    "    \n",
    "    sid = False\n",
    "    for r in reserve_polygons:\n",
    "        if r['geometry'].contains(row['point']):\n",
    "            sid = r['id']\n",
    "            sid = r['id']\n",
    "    if sid:\n",
    "        return sid\n",
    "    else:\n",
    "        return False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now apply this function to the `DataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df[\"station\"] = records_df.apply(lambda row: in_station(reserves, row),axis=1)\n",
    "in_stations_df = records_df[records_df[\"station\"] != False]\n",
    "in_stations_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see if this corresponds to what we observed on the map:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "in_stations_df.groupby([\"species\", \"station\"])['station'].count().unstack().plot.barh(stacked=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Part 3: Comparing California Oak species:<a id='oak'></a>\n",
    "\n",
    "\n",
    "| ![quercus douglassi](https://upload.wikimedia.org/wikipedia/commons/thumb/6/6d/Large_Blue_Oak.jpg/220px-Large_Blue_Oak.jpg)  | ![quercus lobata](https://upload.wikimedia.org/wikipedia/commons/thumb/8/86/Valley_Oak_Mount_Diablo.jpg/220px-Valley_Oak_Mount_Diablo.jpg) | ![quercus durata](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b8/Quercusduratadurata.jpg/220px-Quercusduratadurata.jpg) | ![quercus agrifolia](https://upload.wikimedia.org/wikipedia/commons/thumb/d/d1/Quercus_agrifolia_foliage.jpg/220px-Quercus_agrifolia_foliage.jpg) |\n",
    "|:---:|:---:|:---:|:---:|\n",
    "| *Quercus douglassi* | *Quercus lobata* | *Quercus durata* | *Quercus agrifolia*|\n",
    "\n",
    "\n",
    "Let's search for these different species of oak using our `GBIF` API and collect the observations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "species_records = []\n",
    "\n",
    "species = [\"Quercus douglassi\", \"Quercus lobata\", \"Quercus durata\", \"Quercus agrifolia\"]\n",
    "\n",
    "for s in species:\n",
    "    req = GBIFRequest()  # creating a request to the API\n",
    "    params = {'scientificName': s}  # setting our parameters (the specific species we want)\n",
    "    pages = req.get_pages(params)  # using those parameters to complete the request\n",
    "    records = [rec for page in pages for rec in page['results'] if rec.get('decimalLatitude')]  # sift out valid records\n",
    "    species_records.extend(records)\n",
    "    time.sleep(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can convert this JSON to a `DataFrame` again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df = pd.DataFrame(species_records)\n",
    "records_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many records do we have now?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(records_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how they're distributed among our different species queries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df['species'].value_counts().plot.barh()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can group this again by `collectionCode`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df.groupby([\"species\", \"collectionCode\"])['collectionCode'].count().unstack().plot.barh(stacked=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also map these like we did with the *Argia arioides* above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "color_dict, html_key = assign_colors(records_df, \"species\")\n",
    "display(HTML(html_key))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mapc = folium.Map([37.359276, -122.179626], zoom_start=5)\n",
    "\n",
    "points = folium.features.GeoJson(reserves)\n",
    "mapc.add_child(points)\n",
    "for r in records_df.iterrows():\n",
    "    lat = r[1]['decimalLatitude']\n",
    "    long = r[1]['decimalLongitude']\n",
    "    folium.CircleMarker((lat, long), color=color_dict[r[1]['species']]).add_to(mapc)\n",
    "mapc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the same code we wrote earlier to see which reserves have which species of oak:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df[\"point\"] = records_df.apply(lambda row: make_point (row),axis=1)\n",
    "records_df[\"station\"] = records_df.apply(lambda row: in_station(reserves, row),axis=1)\n",
    "in_stations_df = records_df[records_df[\"station\"] != False]\n",
    "in_stations_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can make a bar graph like we did before, grouping by `species` and stacking the bar based on `station`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "in_stations_df.groupby([\"species\", \"station\"])['station'].count().unstack().plot.barh(stacked=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ---\n",
    "\n",
    "# Part 4: Cal-Adapt API<a id='adapt'></a>\n",
    "\n",
    "Let's get back the data from *Argia agrioides* with the GBIF API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "req = GBIFRequest()  # creating a request to the API\n",
    "params = {'scientificName': 'Argia agrioides'}  # setting our parameters (the specific species we want)\n",
    "pages = req.get_pages(params)  # using those parameters to complete the request\n",
    "records = [rec for page in pages for rec in page['results'] if rec.get('decimalLatitude')]  # sift out valid records\n",
    "records[:5]  # print first 5 records"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll make a `DataFrame` again for later use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df = pd.DataFrame(records)\n",
    "records_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will use the [Cal-Adapt](http://api.cal-adapt.org/api/) web API to work with time series raster data. It will request an entire time series for any geometry and return a Pandas `DataFrame` object for each record in all of our *Argia agrioides* records:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "req = CalAdaptRequest()\n",
    "records_g = [dict(rec, geometry=sg.Point(rec['decimalLongitude'], rec['decimalLatitude']))\n",
    "             for rec in records]\n",
    "ca_df = req.concat_features(records_g, 'gbifID')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the first five rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ca_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(ca_df.columns), len(ca_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks like the time series data we want for each record (the unique ID numbers as the columns). Each record has the projected temperature in Fahrenheit for 273 years (every row!). We can plot predictions for few random records:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a line plot using the first 9 columns of dataframe\n",
    "ca_df.iloc[:,:9].plot()\n",
    "\n",
    "# Use matplotlib to title your plot.\n",
    "plot.title('Argia agrioides - %s' % req.slug)\n",
    "\n",
    "# Use matplotlib to add labels to the x and y axes of your plot.\n",
    "plot.xlabel('Year', fontsize=18)\n",
    "plot.ylabel('Degrees (Fahrenheit)', fontsize=16)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like temperature is increasing across the board wherever these observations are occuring. We can calculate the average temperature for each year across observations in California:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmax_means = ca_df.mean(axis=1)\n",
    "tmax_means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's happening to the average temperature that *Argia agrioides* is going to experience in the coming years across California?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmax_means.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is there a temperature at which the *Argia agrioides* cannot survive? Is there one in which they particularly thrive?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "What if we look specifically at the field stations and reserves? We can grab our same code that checked whether a record was within a station, and then map those `gbifID`s back to this temperature dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records_df[\"point\"] = records_df.apply(lambda row: make_point (row),axis=1)\n",
    "records_df[\"station\"] = records_df.apply(lambda row: in_station(reserves, row),axis=1)\n",
    "in_stations_df = records_df[records_df[\"station\"] != False]\n",
    "in_stations_df[['gbifID', 'station']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall the column headers of our `ca_df` are the `gbifID`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ca_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we subset the temperature dataset for only the observations that occurr within the bounds of a reserve or field station:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "station_obs = [str(id) for id in list(in_stations_df['gbifID'])]\n",
    "ca_df[station_obs]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's graph these observations from Santa Cruz Island against the average temperature across California where this species was observed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot.plot(tmax_means)\n",
    "plot.plot(ca_df[station_obs])\n",
    "\n",
    "# Use matplotlib to title your plot.\n",
    "plot.title('Argia agrioides and temperatures in Santa Cruz Island')\n",
    "\n",
    "# Use matplotlib to add labels to the x and y axes of your plot.\n",
    "plot.xlabel('Year', fontsize=18)\n",
    "plot.ylabel('Degrees (Fahrenheit)', fontsize=16)\n",
    "plot.legend([\"CA Average\", \"Santa Cruz Island\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does this tell you about Santa Cruz Island? As time goes on and the temperature increases, might Santa Cruz Island serve as a refuge for *Argia agrioides*?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}