Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Api data vis nov23 #45

Merged
merged 2 commits into from
Dec 4, 2023
Merged

Api data vis nov23 #45

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b6bcb20
Small updates in API presentation
luisesanmartin Nov 12, 2023
407188e
API materials update
luisesanmartin Nov 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified 1-foundations/4-api-and-dataviz/Introduction to APIs.pdf
View file
Open in desktop
Binary file not shown.
Binary file modified 1-foundations/4-api-and-dataviz/Introduction to APIs.pptx
View file
Open in desktop
Binary file not shown.
288 changes: 288 additions & 0 deletions 1-foundations/4-api-and-dataviz/foundations-s4-api-bonus.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,288 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"https://colab.research.google.com/github/worldbank/dec-python-course/blob/main/1-foundations/4-api-and-dataviz/foundations-s4-api-bonus.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import requests"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Argument-based parameters\n",
"\n",
"- The example of geoBoundaries we previously saw takes _URL-based_ API query parameters\n",
"- Many APIs take _argument-based_ query parameters\n",
"- This will be the case every time you have to use query parameters separated by an ampersand symbol (`&`) after a question mark (`?`) in the API endpoint\n",
"- Take this generic example:\n",
"\n",
"`https://api.org/endpoint/?parameter1=value1&parameter2=value2&parameter3=value3`\n",
"\n",
"- Theoretically, it's _possible_ to modify an API URL call using concatenated strings in Python to build argument-based queries"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"endpoint = 'https://api.org/endpoint/'\n",
"p1 = 'parameter1'\n",
"v1 = 'value1'\n",
"url = endpoint+'?'+p1+'='+v1\n",
"print(url)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Then we could use `requests` as usual to obtain a response from this API\n",
"\n",
"```{python}\n",
"requests.get(url)\n",
"```\n",
"\n",
"- However, the convention in Python is to **use the argument `params` of `requests.get()`** to pass argument-based query parameters\n",
"\n",
"```{python}\n",
"parameters = {'parameter1': 'value1'}\n",
"requests.get(endpoint, params=parameters)\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Applied example: the WBG API\n",
"\n",
"To illustrate an example of argument-based parameters, we'll use again the endpoint of the total population API to fetch country population data. Note that this time the parameters `date` and `year` are passed in the dictionary named `parameters` and not in the URL string as we did before."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def fetch_population_by_year(year):\n",
" \n",
" endpoint = 'https://api.worldbank.org/v2/country/all/indicator/SP.POP.TOTL'\n",
" parameters = {'date': year, 'format':'json'}\n",
" # note: the API documentation specifies that format=json\n",
" # is a required parameter in order to return the results as JSON\n",
" response = requests.get(endpoint, params=parameters)\n",
" \n",
" if response.status_code == 200:\n",
" \n",
" data = response.json()\n",
" return data\n",
" \n",
" else:\n",
" \n",
" print('Request failed!')\n",
" return None"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pop_2015 = fetch_population_by_year(2015)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pop_2015"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A few notes:\n",
"- Remember that this is the same data you obtain when accessing https://api.worldbank.org/v2/country/all/indicator/SP.POP.TOTL?date=2015&format=json on a web browser\n",
"- The first element of this list contains metadata about the data returned by the API"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pop_2015[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Note the detail of the information in the keys `page`, `pages`, `per_page`, and `total` in the first element of `pop_2015`"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print('Total obs: {}'.format(pop_2015[0]['total']))\n",
"print('Obs per page: {}'.format(pop_2015[0]['per_page']))\n",
"print('Total pages: {}'.format(pop_2015[0]['pages']))\n",
"print('Current page: {}'.format(pop_2015[0]['page']))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- The result is only one page with 50 observations, out of a total of 266\n",
"- This means that results are incomplete! More API calls are needed to complete the total 266 observations\n",
"- A further inspection of the API documentation shows that the parameters `page=page_number` or `per_page=obs_per_page` can be used to retrieve complete results.\n",
"\n",
"**Important:** When fetching data using APIs, always inspect the result to look for possible limitations in the results' default format. You might be inadvertently missing observations in your API calls!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Bonus exercise\n",
"\n",
"Modify the function `fetch_population_by_year()` below to retrieve the complete results for a given year (not only page 1) from the population endpoint.\n",
"\n",
"Suggested steps:\n",
"\n",
"1. Run `fetch_population_by_year()` for any year and inspect the resulting list. You will note that the first element of the list is a dictionary with a key `total` that indicates the total number of observations in the query result\n",
"1. In your function, save that number into a variable\n",
"1. Send a new API request adding the parameter `per_page` in the parameters dictionary. Its value should be the total number of observations\n",
"1. The JSON object in the response content will be a new list with the same format: the first element of the list will contain metadata about the API result and the data will be in the second element\n",
"1. Return the second element of your list"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def fetch_population_by_year(year):\n",
" \n",
" endpoint = 'https://api.worldbank.org/v2/country/all/indicator/SP.POP.TOTL'\n",
" parameters = {'date': year, 'format':'json'}\n",
" response = requests.get(endpoint, params=parameters)\n",
" \n",
" if response.status_code == 200:\n",
" \n",
" data = response.json()\n",
" \n",
" # === ADD YOUR SOLUTION HERE ===\n",
" # Note that this time we're not giving you the steps\n",
" # to follow and variables to create. Figuring that\n",
" # out is also part of the exercise :)\n",
" \n",
" return data\n",
" \n",
" else:\n",
" \n",
" print('Request failed!')\n",
" return None"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Coding API clients - More takeaways\n",
"\n",
"- Passing API call parameters as arguments is a more elegant way of coding API clients. Argument-based parameters also work better when introducing non-conventional characters as arguments. Think, for example: how would you include a space character in a URL? argument-based parameters handle those cases seamlessly\n",
"- When authentication is needed, API keys are usually passed as argument-based parameters. If the API has a dedicated client library, they will ask for the key after importing the library (as `user_agent` in `Nominatim()`)\n",
"- Many APIs will combine the use of URL-based and argument-based parameters to pass information in API queries. A good API client will take that into account to build the correct query URL and parameters argument. Take the following example:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def fetch_population_by_year_country(year, country):\n",
" \n",
" endpoint = 'https://api.worldbank.org/v2/country/' + country + '/indicator/SP.POP.TOTL'\n",
" parameters = {'date': year, 'format':'json'}\n",
" # note: the API documentation specifies that format=json\n",
" # is a required parameter in order to return the results as JSON\n",
" response = requests.get(endpoint, params=parameters)\n",
" \n",
" if response.status_code == 200:\n",
" \n",
" data = response.json()\n",
" return data\n",
" \n",
" else:\n",
" \n",
" print('Request failed!')\n",
" return None"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"brazil_data = fetch_population_by_year_country(2015, 'BRA')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"brazil_data"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Loading