Analysis and Statistics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysing Data + Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What we will be doing and how"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lesson, we will be analysing a dataset from the World Health Organisation. We will first load the data that we will be using, and then looking through it to understand it. When this is done, we will need to clean it on order to get the parts that are interesting to us, and then we will find some correlations among variable to see how they relate to each other. More specifically, we will be looking at the correlation between GDP and life expectancy.\n",
    "\n",
    "\n",
    "### How does this notebook work?\n",
    "\n",
    "Run the cells with code written on them. To do this, you can select them and press Shift + Enter or press the \"Play\" button on the left side of the cell (if you do not see this, hover with the mouse on the cell and it should appear). Remember that all cells need to be run in order, even those in which you did not need to write any code. If you think you might have skipped one, on the left side of the cell you will find the line number, which corresponds to the order in which you have ran the cells. Keep in mind that if you run a cell and it gets numbered as 4, if you run the same cell immediately after it will be renumbered to 5.\n",
    "\n",
    "Looking at the code you will notice some parts are incomplete and have '\\_\\_' written instead. This means that you need to complete that part of the code. In some other parts, you will need to write your own code. This will be specified.\n",
    "\n",
    "You will also find hyperlinks to documentation on different functions that we will use. It is recommended to look at them to familiarise yourself with what you are doing and how they work.\n",
    "\n",
    "There are also questions to be completed in text cells. Click twice on them to start editing them or select them and press Enter, and press Shift+Enter when you are finished to go back to reading mode. The questions can be answered in one or two sentences in general."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#First, we need to import the tools we will be using. Links to documentation of the different\n",
    "#tools we will be using can be found on the explanation of each point.\n",
    "import os\n",
    "import pandas\n",
    "import matplotlib.pyplot\n",
    "import numpy\n",
    "import seaborn\n",
    "import math"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you have links to the libraries we have imported:\n",
    "\n",
    "[OS library](https://docs.python.org/3/library/os.html): Library with different useful functions to interact with the Operating System. We will be using this for loading the datasets.\n",
    "\n",
    "[Pandas](https://pandas.pydata.org/): Pandas is a tool for data analysis and manipulation.\n",
    "\n",
    "[Seaborn](https://seaborn.pydata.org/): Seaborn is a library specialised in statistical data visualisation.\n",
    "\n",
    "[Numpy](https://numpy.org/): Numpy is the Python library for mathematics. We will use it for performing operations on our data.\n",
    "\n",
    "[Matplotlib](https://matplotlib.org/): Matplotlib is another library specilised in visualisation for Python.\n",
    "\n",
    "[Math module](https://docs.python.org/3/library/math.html): This module provides mathematical functions defined by the C standard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Activity 1: Getting to know our data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we need to load the data we will be using.\n",
    "\n",
    "For this we have to get the path to the data we are using, and then read the file. In this case, it is in the datasets\n",
    "folder inside our working directory called 'datasets', and the file name is 'who_countries.csv'. This dataset contains health information on several countries of the world from 2000 to 2015.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Get the path to the file we will be using. In this case, the dataset is under the datasets folder\n",
    "path = os.path.join(os.getcwd(), 'datasets', 'who_countries.csv')\n",
    "#Load the data into the countries_info variable. This results in a DataFrame object.\n",
    "countries_info = pandas.read_csv(path, delimiter = ',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we we will look at the dataset that we have loaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "countries_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us a rough idea of the data we are working with, but we can get more information on what is part of the dataset using other functions, as shown below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Number of rows and columns in the format (rows, columns)\n",
    "print(\"Number of rows and columns:\",countries_info.shape, \"\\n\") \n",
    "#Name of the columns\n",
    "print(\"Name of the columns: \",countries_info.columns, \"\\n\")\n",
    "#Check the type of each of the columns of the data we are using\n",
    "print(\"Type of each of the columns:\\n\",countries_info.dtypes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When analysing data, we are interested in different statistics. The [describe()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.describe.html) function in Python gives us many of these statistics for our dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Use the DataFrame function describe() to see a summary of the dataset\n",
    "__.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "***Looking at the columns, you will notice that the \"Country\" column does not appear. Looking at the types of each of the columns, can you infer why this is the case?***\n",
    "\n",
    "***This is a summary of statistics over the whole dataset, which takes countries around the world over 15 years. What has been the maximum life expectancy so far in the world?***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we know that our dataset has 2938 rows, and that it should have entries for each country between 2000 and 2015, meaning 16 entries each. 2938 is not a multiple of 16, which means that some countries are missing entries. We will have to keep this in mind for our analysis, as the amount of data we have for each year might differ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at the summary statistics again, especially the mean. This mean is calculated taking values of different countries in different years, so it can not really tell us much (for example, we can not know the average life expectancy in the world because it combines different years). This means that we will need to take a specific year to look at the countries' GDP and life expectancy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Activity 2: Tidying up the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part we will look only at the columns we are interested in. This means that we will have to take just those, and rename them when necessary to get our data to fit conventions. For this, we will use the [loc[]](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.loc.html) function, to which we give the names of the columns we want to select as input. The columns we are interested in are the following: Country, Year, GDP, Life expectancy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "selection_info = countries_info.loc[:,['Country', 'Year', 'GDP', 'Life expectancy']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The naming of the columns is not as conventions dictate. In regards to column naming, the conventions ask for names to be like_this (i.e. lowercase letters and underscore instead of space). None of the columns we have extracted comply with this, so we will now rename them using the [rename()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.rename.html) function. The first one is an example of how to do it, do the same to rename the rest of the columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#In the following code, the name before the : is the current column name, the one after is what\n",
    "#you want to change it to.\n",
    "selection_info.rename({'Country':'country'},axis = 'columns', inplace=True)\n",
    "#Copy the above line and change the names for the rest of the columns that need so."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have this selection and the names , we can look at one specific country, Australia in this case.\n",
    "For this, we will use the [loc[]](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.loc.html) function again, but this time we will use it so it only takes entries with the value 'Australia' in the column 'Country'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info_au = selection_info.loc[selection_info['country'] == 'Australia']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "info_au"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Looking at Australia's information, have both the GDP and life expectancy grown over time? Do you think these two could be related?***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now take the data from 2015, and will also be looking at the life expectancy and GDP of each of the countries. Because there are many countries, we will now visualise the data to see if the GDP and life expectancy are related, meaning if there is a correlation between them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#The .loc[a] function allows us to get data from a DataFrame that satisfies a condition a\n",
    "info_15 = selection_info.loc[selection_info['year'] == 2015]\n",
    "#In this case, we are getting the data that has 2015 as a value on the 'Year' column\n",
    "\n",
    "#Use the describe() function again to see the summary of our selected data\n",
    "__.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this activity, we will be working with the GDP and Life expectancy attributes. GDP stands for [Gross Domestic Product](https://www.investopedia.com/terms/g/gdp.asp), and it gives a sense of how rich a country is.\n",
    "\n",
    "In order to know more about these, we will now plot their [distribution](https://seaborn.pydata.org/generated/seaborn.distplot.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Plotting the distribution for GDP. Write the name of the correct column\n",
    "seaborn.distplot(info_15['__'], bins = 15)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Write the necessary code for plotting the distribution of the Life expectancy\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Seeing the GDP distribution, would you say that most countries have a high GDP? Why?***\n",
    "\n",
    "***Having seen the distribution for both life expectancy and GDP, can you guess from these if there is a correlation?***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know our data a bit better, let's try to find relationships in it. For doing this, we will be looking at the correlation between attributes by plotting pairs of them. Remember that an easy way to see if there is a correlation between two variables is to plot them and try to fit a line through the points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before going further with our data, we need to check for missing values. These appear as NaN (Not a Number). To check if any of our columns has missing values, we use the [isna()](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.isna.html) function to get a list of booleans (True if the value is missing, False if not), and then use [sum()](https://docs.python.org/3/library/functions.html#sum) to count the number of True values in the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Missing values in GDP: \",info_15['gdp'].isna().sum())\n",
    "print(\"Missing values in life ecpectancy: \", info_15['life_expectancy'].isna().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have seen that there are some data points for which we do not have some of the values. This means that we need to delete those for our representation.\n",
    "\n",
    "To drop the NaN values, we use the DataFrame function [.dropna(a)](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html), where a in this case stands for the subset of the names of the columns we want to act on.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Complete the function with the names of the columns in which we are deleting NaN values\n",
    "clean_info_15 = info_15.dropna(subset=['__'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " We will now use the function *plt.plot(x, y, 'o')* to plot the points.\n",
    "\n",
    "Note: The 'o' at the function arguments is for it to draw circular points. Other markers are available, check the [documentation](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.plot.html) for these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matplotlib.pyplot.plot(clean_info_15['gdp'], clean_info_15['life_expectancy'], 'o')\n",
    "matplotlib.pyplot.xlabel(\"gdp\")\n",
    "matplotlib.pyplot.ylabel(\"life_expectancy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Remember that each point represents a country. Does the distribution of these match the distribution we plotted earlier?***\n",
    "\n",
    "***From this plot, can you deduce if there is correlation between GDP and life expectancy?***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many points close to 0. This is because some GDPs are very big compared to others. So, in order to represent the data properly and see if there is in fact a correlation, we need to change the scale of the GDP axis. What scale would be the most appropriate to do this? Write the necessary code for representing the data in the new scale. Also note that the shape of the data points is very similar to that of a logarithmic function.\n",
    "\n",
    "Look at the documentation for the function [.xscale()](https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.xscale.html) to find what argument you need to use for your scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matplotlib.pyplot.plot(clean_info_15['gdp'], clean_info_15['life_expectancy'], 'o')\n",
    "#Fill in the correct argument\n",
    "matplotlib.pyplot.xscale('__')\n",
    "matplotlib.pyplot.xlabel(\"GDP\")\n",
    "matplotlib.pyplot.ylabel(\"Life expectancy\")\n",
    "matplotlib.pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Do you think there is a correlation here?***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will plot the fitting line. In this case, we will be using the seaborn [regplot()](https://seaborn.pydata.org/generated/seaborn.regplot.html) function, which fits a Machine Learning model known as linear regression and plots the data points and a fitting line.\n",
    "\n",
    "In the code, you will see that we are taking the logarithm of all the GDP values and feeding that into the function, instead of changing the scale, using a [list comprehension](https://docs.python.org/3/tutorial/datastructures.html). Here we are creating a new variable that will be more useful to us than the one we had before. We do this is because we want to fit the line to the transformed values, otherwise the line would be skewed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#You do not need to do anything here, just run the cell\n",
    "\n",
    "#We first get the logarithm of all the GDP values\n",
    "log_gdp = [math.log10(val) for val in clean_info_15['gdp']]\n",
    "#Plot the information and fit a line to the points\n",
    "seaborn.regplot(log_gdp, clean_info_15['life_expectancy'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we will now calculate the value of the correlation between both attributes. We will use the NumPy function [corrcoef(x,y)](https://realpython.com/numpy-scipy-pandas-correlation-python/). This function returns a 2x2 matrix with the correlation coefficients between the two arguments (i.e. \\[x-x, y-x\\], \\[x-y, y-y\\]).We are interested in the x-y or y-x coefficients.  We will again apply a logarithmic function to our GDP data before passing it into the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numpy.corrcoef(log_gdp, clean_info_15['__'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***What correlation coefficient did you obtain? Do you think this is a strong or a weak correlation? Remember that a correlation value can go from -1 to 1, 0 being no correlation at all.***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Activity 3: Mini challenge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keeping in mind what we have done until here, take now the GDP and the Schooling variables from the year 2010 and check if there is a correlation. Remember the following:\n",
    "    \n",
    "    -Delete the empty values where necessary \n",
    "    \n",
    "    -Label all the plots you make\n",
    "    \n",
    "    -Use an appropriate scale when necessary\n",
    "\n",
    "When you have your final result, make sure that you explain why do you think there is or there is not a correlation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Write your code here. Remember you can add any cells you need and you can go back to the previous code"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}