DS105-Intermediate-Statistics/DS105-L1-Python-Basic-Stats.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e804e15e",
   "metadata": {},
   "source": [
    "# DS105 Intermediate Statistics : Lesson One - Basic Stats in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98a31831",
   "metadata": {},
   "source": [
    "### Table of Contents <a class=\"anchor\" id=\"DS105L1_toc\"></a>\n",
    "\n",
    "* [Table of Contents](#DS105L1_toc)\n",
    "    * [Page 1 - Overview of this Module](#DS105L1_page_1)\n",
    "    * [Page 2 - Single Sample t-Test](#DS105L1_page_2)\n",
    "    * [Page 3 - Hands-On Single Sample t-Test](#DS105L1_page_3)\n",
    "    * [Page 4 - Hands-On Single Sample t-Test Solution](#DS105L1_page_4)\n",
    "    * [Page 5 - Independent t-Test](#DS105L1_page_5)\n",
    "    * [Page 6 - Hands-On Independent t-Test](#DS105L1_page_6)\n",
    "    * [Page 7 - Hands-On Independent t-Test Solution](#DS105L1_page_7)\n",
    "    * [Page 8 - Dependent t-Test](#DS105L1_page_8)\n",
    "    * [Page 9 - Hands-On Dependent t-Test](#DS105L1_page_9)\n",
    "    * [Page 10 - Hands-On Dependent t-Test Solution](#DS105L1_page_10)\n",
    "    * [Page 11 - Independent Chi-Square](#DS105L1_page_11)\n",
    "    * [Page 12 - Hands-On Independent Chi-Square](#DS105L1_page_12)\n",
    "    * [Page 13 - Hands-On Independent Chi-Square Solution](#DS105L1_page_13)\n",
    "    * [Page 14 - Correlations](#DS105L1_page_14)\n",
    "    * [Page 15 - Hands-On Correlations](#DS105L1_page_15)\n",
    "    * [Page 16 - Hands-On Correlations Solution](#DS105L1_page_16)\n",
    "    * [Page 17 - Key Terms](#DS105L1_page_17)\n",
    "    * [Page 18 - Hands-On Lesson 1 Review](#DS105L1_page_18)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4eedfc46",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 1 - Overview of this Module<a class=\"anchor\" id=\"DS105L1_page_1\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c3baa223",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/388627207\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x104d6f460>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Tutorial Video Name: Intermediate Statistics in Python\n",
    "VimeoVideo('388627207', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35c2ee72",
   "metadata": {},
   "source": [
    "The transcript for the above overview video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO105L01overview.zip)**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "357b586b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/444962753\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x104d6f3d0>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# OPTIONAL: Recoreded Live Workshop\n",
    "VimeoVideo('444962753', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6f59ff7",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "This lesson will cover the basic statistics you already know how to do in MS Excel and R in Python.  \n",
    "\n",
    "By the end of this lesson, you should be able to complete the following tasks in Python: \n",
    "\n",
    "* Single sample, independent, and dependent *t* tests\n",
    "* Chi-Squares\n",
    "* Correlation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f149a06",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 2 - Single Sample t-Test <a class=\"anchor\" id=\"DS105L1_page_2\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2664bc3c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/334045547\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x1043638b0>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Single Sample t-Test\n",
    "VimeoVideo('334045547', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "406dba0a",
   "metadata": {},
   "source": [
    "The approximate transcript and code for the above overview video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO105L01pg2tutorial.zip)**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b62c833",
   "metadata": {},
   "source": [
    "# Single Sample t-Test\n",
    "\n",
    "Remember that a single-sample *t*-test is meant to examine whether a particular value is different than the population mean.  You've already performed single sample *t*-tests by hand and using R.  Now it's time to learn how to complete them in Python!  \n",
    "\n",
    "---\n",
    "## Import Packages\n",
    "\n",
    "The very first thing you will need to do is import the packages you will be using.\n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy.stats import norm\n",
    "from scipy import stats\n",
    "```\n",
    "\n",
    "```scipy``` and ```stats``` are packages you will be working with a lot in this module, as they have a lot of useful statistics functions.\n",
    "\n",
    "---\n",
    "\n",
    "## Import Data\n",
    "\n",
    "You will be testing whether a cost of $25,000 for a hybrid vehicle in 2013 is different than the mean cost, using **[this data](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/hybrid2013.zip)**.\n",
    "---\n",
    "\n",
    "## Test Assumptions\n",
    "\n",
    "The only assumption for the single sample *t* test is that the data is normally distributed. You can test this just by creating a histogram: \n",
    "\n",
    "```python\n",
    "hybrid2013['msrp'].hist()\n",
    "```\n",
    "\n",
    "Here's the output:\n",
    "\n",
    "![The x-axis of a bar chart ranges from 20000 to 1 lakh in 8 units and the y-axis ranges from 0 to 10 in six units. The plot shows the highest peak in the categories of 20000 and 30000.](Media/python1.png)\n",
    "\n",
    "Looks like things aren't quite normally distributed, but you'll let it slide for now for learning purposes. \n",
    "\n",
    "---\n",
    "\n",
    "## Run the Analysis\n",
    "\n",
    "There is only one line of code needed to run a single sample *t*-test in Python.  The function ```stats.ttest_1samp()``` performs the function, and takes the argument of the data column that contains your population values, and then the x value that you are trying to test against, which is $25,000.\n",
    "\n",
    "```python\n",
    "stats.ttest_1samp(hybrid2013['msrp'], 25000)\n",
    "```\n",
    "\n",
    "Here is the result :\n",
    "\n",
    "```text\n",
    "Ttest_1sampResult(statistic=6.003733172775179, pvalue=3.9231807518835515e-07)\n",
    "```\n",
    "\n",
    "The statistic is your *t* value, and the *p* value is the one associated with that *t*-test.  Remember that the *p* value is written in scientific notation, so this is significant at *p* < .05.  That means that buying a hybrid car for $25,000 in 2013 is different than the population mean.  Is it higher or lower? To answer that question, you will need to examine the population mean: \n",
    "\n",
    "```python\n",
    "hybrid2013.msrp.mean()\n",
    "```\n",
    "\n",
    "It turns out that the average cost of a hybrid vehicle in 2013 was $42,943.49.   \n",
    "\n",
    "So, buying one for $25,000 would have been a great steal! \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea059b62",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 3 - Hands-On Single Sample t-Test<a class=\"anchor\" id=\"DS105L1_page_3\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d478d00",
   "metadata": {},
   "source": [
    "For this Activity, you will compute a single sample *t*-test in Python. `This Hands-On will not be graded`, but you are encouraged to complete it. The best way to become a great data scientist is to practice! Once you have submitted your project, you will be able to access the solution on the next page. Note that the solution will be slightly different from yours, but should look similar.\n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Do not submit your project until you have completed all requirements, as you will not be able to resubmit.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Requirements\n",
    "\n",
    "Using the **[hybrid2013 dataset](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/hybrid2013.zip)** you worked with in the lesson, determine whether a miles per gallon (mpg) rating of 40 is unusual for a hybrid car on the market in 2013. To do this, you will need to test for the assumption of normality by creating a histogram, and then run a single sample *t*test. \n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Be sure to zip and submit your entire directory when finished!</p>\n",
    "    </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ed97672",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 4 - Hands-On Single Sample t-Test Solution<a class=\"anchor\" id=\"DS105L1_page_4\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6030ad2b",
   "metadata": {},
   "source": [
    "## Test for Normality\n",
    "\n",
    "First, test for the assumption of normality by creating a histogram.  \n",
    "\n",
    "```python\n",
    "hybrid2013['mpg'].hist()\n",
    "```\n",
    "\n",
    "It should look like this: \n",
    "\n",
    "![The x-axis of a bar chart ranges from 20 to 50 in 6 units. The y-axis axis ranges from 0 to 8 in 4 units. The highest peak in the bar chart crosses 8 units on the y-axis.](Media/python2.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Single Sample *t*-Test\n",
    "\n",
    "```python\n",
    "stats.ttest_1samp(hybrid2013['mpg'], 40)\n",
    "```\n",
    "\n",
    "Here is the output:\n",
    "\n",
    "```text\n",
    "Ttest_1sampResult(statistic=-4.427320491687408, pvalue=6.67005084670698e-05)\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Check the Population Mean\n",
    "\n",
    "```python\n",
    "hybrid2013.mpg.mean()\n",
    "```\n",
    "\n",
    "And the output:\n",
    "\n",
    "```text\n",
    "33.48837209302326\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88a22a1a",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 5 - Independent t-Test<a class=\"anchor\" id=\"DS105L1_page_5\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a6816107",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/334044896\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x104363bb0>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Single Sample t-Test\n",
    "VimeoVideo('334044896', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfceec01",
   "metadata": {},
   "source": [
    "\n",
    "The approximate transcript and code for the above overview video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO105L01pg5tutorial.zip)**.\n",
    "\n",
    "# Independent t-Test\n",
    "\n",
    "An independent *t* test is used when you have one independent variable that is categorical and a grouping variable, and one dependent continuous variable.  Use an independent *t*-test when you want to determine whether the means of two different, unrelated groups are the same or different. \n",
    "\n",
    "---\n",
    "\n",
    "## Import Packages\n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy import stats\n",
    "from scipy.stats import ttest_ind\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Import Data\n",
    "\n",
    "You will continue to use the hybrid cars dataset from before.  However, this time, you are testing to see whether compact and mid-size hybrid cars differ in their average miles per gallon.\n",
    "\n",
    "---\n",
    "\n",
    "## Test Assumptions \n",
    "\n",
    "The only assumption that independent *t* has is normality.  You will need to test normality for each of your groups - compact and mid-sized hybrid cars.  \n",
    "\n",
    "This code is very similar to before, but has an extra layer of specifying which values from the ```carclass``` you want to examine: \n",
    "\n",
    "```python\n",
    "hybrid2013.mpg[hybrid2013.carclass == 'C'].hist()\n",
    "```\n",
    "\n",
    "So the name of the dataset and the name of the variable go first, but then you need to specify that you only want a histogram for the values that meet the condition ```C``` for compact cars. \n",
    "\n",
    "Here is the result: \n",
    "\n",
    "![The x-axis of a bar chart ranges from 30 to 50 in 5 units. The y-axis axis ranges from 0.00 to 2.00 in 8 units. The highest peak in the bar chart crosses 8 units on the y-axis.](Media/python3.png)\n",
    "\n",
    "Same thing for the mid-size hybrid cars:\n",
    "\n",
    "```python\n",
    "hybrid2013.mpg[hybrid2013.carclass == 'M'].hist()\n",
    "```\n",
    "\n",
    "And the result: \n",
    "\n",
    "![The x-axis of a bar chart ranges from 25 to 50 in 5 units. The y-axis axis ranges from 0.0 to 4.0 in 8 units. The plot shows the highest peak in categories 30 and 40.](Media/python4.png)\n",
    "\n",
    "It looks like neither of these are bell-shaped, and are thus not normal, but for the purposes of learning, you will continue.\n",
    "\n",
    "---\n",
    "\n",
    "## Run the Analysis\n",
    "\n",
    "You will use the function ```ttest_ind()``` to run an independent *t* test in Python. The arguments are two things you want to compare to each other.  If you happen to have those two things in separate columns, then it would simply look like this mock code: \n",
    "\n",
    "```python\n",
    "ttest_ind(data[column1], data[column2])\n",
    "```\n",
    "\n",
    "But since in this case, your data is all stored in the same ```carclass``` column, and you want to pull out the data for certain values, the code looks just a bit more complicated:\n",
    "\n",
    "```python\n",
    "ttest_ind(hybrid2013.mpg[hybrid2013.carclass == 'C'], hybrid2013.mpg[hybrid2013.carclass == 'M'])\n",
    "```\n",
    "\n",
    "And here is the results: \n",
    "\n",
    "```text\n",
    "Ttest_indResult(statistic=1.0751886097093057, pvalue=0.29216712457079796)\n",
    "```\n",
    "\n",
    "Looks like there is no significant difference between compact and mid-size hybrid cars in terms of miles per gallon, since the *p* value is not less than .05.  The *t* value is also pretty small, which is another good indication.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ac79378",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 6 - Hands-On Independent t-Test<a class=\"anchor\" id=\"DS105L1_page_6\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f025298",
   "metadata": {},
   "source": [
    "For your Activity, you will be computing an independent *t*-test to see if the miles per gallon differ between compact (```C```) and large (```L```) cars. `This Hands-On will not be graded`, but you are encouraged to complete it. The best way to become a great data scientist is to practice! Once you have submitted your project, you will be able to access the solution on the next page. Note that the solution will be slightly different from yours, but should look similar.\n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Do not submit your project until you have completed all requirements, as you will not be able to resubmit.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Requirements\n",
    "\n",
    "Using the **[hybrid2013 dataset](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/hybrid2013.zip)** you worked with in the lesson, determine if the mean miles per gallon for a compact and a large car differ from each other. To do this, you will need to test for the assumption of normality for both groups by creating a histogram, and then run an independent *t*test. \n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Be sure to zip and submit your entire directory when finished!</p>\n",
    "    </div>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bd5d174",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 7 - Hands-On Independent t-Test Solution<a class=\"anchor\" id=\"DS105L1_page_7\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80921c18",
   "metadata": {},
   "source": [
    "# Solution\n",
    "\n",
    "---\n",
    "\n",
    "## Test for Normality\n",
    "\n",
    "First, test for the assumption of normality by creating a histogram for both groups:  \n",
    "\n",
    "```python\n",
    "hybrid2013.mpg[hybrid2013.carclass == 'C'].hist()\n",
    "```\n",
    "\n",
    "It should look like this: \n",
    "\n",
    "![The x-axis of a bar chart ranges from 30 to 50 in 5 units. The y-axis axis ranges from 0.00 to 2.0 in 8 units. The highest peak in the bar chart reaches 2.00 on the y-axis.](Media/python3.png)\n",
    "\n",
    "```python\n",
    "hybrid2013.mpg[hybrid2013.carclass == 'L'].hist()\n",
    "```\n",
    "\n",
    "It should look like this: \n",
    "\n",
    "![The x-axis of a bar chart ranges from 20 to 40 in 5 units. The y-axis axis ranges from 0.00 to 2.0 in 8 units. The highest peak in the bar chart reaches 2.00 on the y-axis.](Media/python5.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Independent *t*-Test\n",
    "\n",
    "```python\n",
    "ttest_ind(hybrid2013.mpg[hybrid2013.carclass == 'C'], hybrid2013.mpg[hybrid2013.carclass == 'L'])\n",
    "```\n",
    "\n",
    "Here is the output:\n",
    "\n",
    "```text\n",
    "Ttest_indResult(statistic=2.598820461640718, pvalue=0.026545168887970098)\n",
    "```\n",
    "\n",
    "There is a significant difference between the mpg of compact cars versus large cars, since the *p* value is < .05.\n",
    "\n",
    "---\n",
    "\n",
    "## Examine the Means for Each Group to Determine Where the Significant Differences Lie\n",
    "\n",
    "To the means, you can call the ```.mean()``` function: \n",
    "\n",
    "```python\n",
    "hybrid2013.mpg[hybrid2013.carclass == 'L'].mean()\n",
    "```\n",
    "\n",
    "The mean for large cars is 28.5 mpg. \n",
    "\n",
    "```python\n",
    "hybrid2013.mpg[hybrid2013.carclass == 'C'].mean()\n",
    "```\n",
    "\n",
    "The mean for compact cars is 40.75 mpg.\n",
    "\n",
    "So, you will be getting significantly more miles to the gallon if you are driving a compact hybrid versus a large-sized hybrid in 2013. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c251f6b",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 8 - Dependent t-Test<a class=\"anchor\" id=\"DS105L1_page_8\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fa62d49a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/334044604\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x104363850>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Single Sample t-Test\n",
    "VimeoVideo('334044604', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45cd53b6",
   "metadata": {},
   "source": [
    "The approximate transcript and code for the above overview video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO105L01pg8tutorial.zip)**.\n",
    "\n",
    "# Dependent t-Test\n",
    "\n",
    "Now it's on to the last *t*-test! Dependent *t*-tests are used when your samples are related in some way, but you still want to see if the means change.  It may be change over time, or change with treatment, etc.  A dependent *t* requires an independent variable that is categorical (groups to compare) and a dependent variable that is continuous. \n",
    "\n",
    "---\n",
    "## Import Packages\n",
    "\n",
    "The only package you will need for this is ```pandas```, to import your data, and ```stats```, from ```scipy```. \n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "from scipy import stats\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Import Data\n",
    "\n",
    "The hybrid car data has been restructured for dependent *t*-tests and an additional year of data, 2012, has been added.  **[It is available here](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/hybrid2012-13.zip)**, and looks like this: \n",
    "\n",
    "![A table has 11 columns with 11 row entries. The columns are labeled vehicle, msrp2012, accelrate2012, mgp2012, mpgmpge2012, carclass2012, carclass_id2012, msrp2013, accelrate2013, mpg2013, mpgmpge2013.](Media/python6.png)\n",
    "\n",
    "Notice that it has the same variables repeated twice, once for 2012, and once for 2013. You'll also notice that the number of rows is greatly reduced form the dataset with only hybrid cars from 2013.  This is because only those cars that had an entry for both 2012 and 2013 were included.\n",
    "\n",
    "You will be testing to see if the price of hybrid cars changes from 2012 to 2013.  \n",
    "\n",
    "---\n",
    "\n",
    "## Test Assumptions\n",
    "\n",
    "Betcha can't guess what the assumptions are for dependent *t*-test! What? You guessed normality, and that you need a histogram! This game is less fun now, but good for you! \n",
    "\n",
    "As with independent *t*-tests, you'll need a histogram for each variable. \n",
    "\n",
    "```python\n",
    "hybrid201213['msrp2012'].hist()\n",
    "```\n",
    "\n",
    "This yields this graphic.  Not really normally distributed, but ignore for now for the purposes of learning. \n",
    "\n",
    "![The x-axis of a bar chart ranges from 20000 to 90000 in 8 units. The y-axis ranges from 0.0 to 3.0 in six units. The plot shows the highest peak in the categories of 30000 and 60000.](Media/python7.png)\n",
    "\n",
    "```python\n",
    "hybrid201213['msrp2013'].hist()\n",
    "```\n",
    "\n",
    "This yields this graphic.  Not really normally distributed, but ignore for now for the purposes of learning. \n",
    "\n",
    "![The x-axis of a bar chart ranges from 20000 to 70000 in 6 units. The y-axis ranges from 0.0 to 3.0 in six units. The highest peak in the bar chart reaches 3.0 on the y-axis.](Media/python8.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Run the Analysis\n",
    "\n",
    "You can use the function ```stats.ttest_rel()``` to compute a dependent *t*-test in Python.  Think of the ```_rel``` as standing for related, since the samples are paired.  The only arguments are the two columns of data you want to use.\n",
    "\n",
    "```python\n",
    "stats.ttest_rel(hybrid201213['msrp2012'], hybrid201213['msrp2013'])\n",
    "```\n",
    "\n",
    "This shows that there is so significant change in hybrid car price from 2012 to 2013, since the *p* value is not less than .05. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f73e4cec",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 9 - Hands-On Dependent t-Test<a class=\"anchor\" id=\"DS105L1_page_9\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d11a4936",
   "metadata": {},
   "source": [
    "For your Activity, you will be computing an dependent *t*-test to see if the miles per gallon changes between 2012 and 2013. `This Hands-On will not be graded`, but you are encouraged to complete it. The best way to become a great data scientist is to practice! Once you have submitted your project, you will be able to access the solution on the next page. Note that the solution will be slightly different from yours, but should look similar.\n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Do not submit your project until you have completed all requirements, as you will not be able to resubmit.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Requirements\n",
    "\n",
    "Using the **[hybrid2012-13 dataset](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/hybrid2012-13.zip)** you worked with in the lesson, determine if the mean miles per gallon changes from 2012 to 2013. To do this, you will need to test for the assumption of normality for both groups by creating a histogram, and then run a dependent *t*test. \n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Be sure to zip and submit your entire directory when finished!</p>\n",
    "    </div>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "768e7fb6",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 10 - Hands-On Dependent t-Test Solution<a class=\"anchor\" id=\"DS105L1_page_10\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e42f6798",
   "metadata": {},
   "source": [
    "# Solution\n",
    "\n",
    "---\n",
    "\n",
    "## Test for Normality\n",
    "\n",
    "First, test for the assumption of normality by creating a histogram for both groups:  \n",
    "\n",
    "```python\n",
    "hybrid201213['mpg2012'].hist()\n",
    "```\n",
    "\n",
    "It should look like this: \n",
    "\n",
    "![The x-axis of a bar chart ranges from 20 to 50 in 6 units. The y-axis ranges from 0.0 to 3.0 in six units. The highest peak in the bar chart reaches 3.0 on the y-axis.](Media/python9.png)\n",
    "\n",
    "```python\n",
    "hybrid201213['mpg2013'].hist()\n",
    "```\n",
    "\n",
    "It should look like this: \n",
    "\n",
    "![The x-axis of a bar chart ranges from 20 to 50 in 6 units. The y-axis ranges from 0 to 5 in 5 units. The highest peak in the bar chart reaches 5 on the y-axis.](Media/python10.png)\n",
    "\n",
    "Neither of these are normal, but you'll proceed for now.\n",
    "\n",
    "---\n",
    "\n",
    "## Dependent *t*-Test\n",
    "\n",
    "```python\n",
    "stats.ttest_rel(hybrid201213['mpg2012'], hybrid201213['mpg2013'])\n",
    "```\n",
    "\n",
    "Here is the output:\n",
    "\n",
    "```text\n",
    "Ttest_relResult(statistic=0.14466598084438312, pvalue=0.8873759030512348)\n",
    "```\n",
    "\n",
    "There is not a significant difference in the miles per gallon for hybrid cars from 2012 to 2013.  Guess that means you'd be good to buy a used model with no drop in gas mileage! This is because the *p* value is > .05.  Notice that the *t* value is tiny as well.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19b37aa8",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 11 - Independent Chi-Square<a class=\"anchor\" id=\"DS105L1_page_11\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "46378261",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/334042957\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x104363a30>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Single Sample t-Test\n",
    "VimeoVideo('334042957', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5876bcc8",
   "metadata": {},
   "source": [
    "The approximate transcript and code for the above overview video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO105L01pg11tutorial.zip)**.\n",
    "\n",
    "# Independent Chi-Square\n",
    "\n",
    "An independent Chi-Square is used when you want to determine whether two categorical variables influence each other.  \n",
    "\n",
    "---\n",
    "## Import Packages\n",
    "\n",
    "All you should need for an independent Chi-Square in Python is ```stats```, and of course ```pandas``` to load in your data:\n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "from scipy import stats\n",
    "```\n",
    "\n",
    "---\n",
    "## Import Data\n",
    "\n",
    "The data located **[here](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/lead_lipstick.zip)** is about the lipstick content in lead.  However, it contains some great categorical fields that you'll be using.  The first is product type, ```prodType``` and it has two levels: ```LP``` is lipstick, and ```LG``` is lip gloss.  The second is price category, ```priceCatgry```, and it has three levels: \n",
    "\n",
    "* 1: < 5 euros\n",
    "* 2: 5-15 euros\n",
    "* 3: > 15 euros\n",
    "\n",
    "You will test to see if the price of the product depends on whether it is a lip stick or a lip gloss. \n",
    "\n",
    "The data look like this initially: \n",
    "\n",
    "![A window displays 2 lines of source code. The source code reads, In open square bracket 58 close square bracket 1 lead_lipstick.head open and close bracket. Out open square bracket 58 close square bracket. A table has 8 columns and 5 row entries.](Media/python11.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Test Assumptions and Run the Analysis\n",
    "\n",
    "There is only one assumption for Chi-Square, and it is that when you are looking at the contingency tables, the expected frequencies for each cell need to have at least 5 entries per cell.  In Python, the only way to easily generate an expected frequencies table is actually to run the analysis.  So, you will conduct your independent Chi-Square first, and then make sure it meets this assumption!\n",
    "\n",
    "---\n",
    "\n",
    "### Create a Contingency Table\n",
    "\n",
    "The first thing that needs to be done, before you can run the independent Chi-Square analysis, is to create a contingency table, sometimes called a *crosstab*, which shows how each level of each variable crosses with the other variable levels.  ```pandas``` saves the day with an easy to use function called ```crosstab()```: \n",
    "\n",
    "```python\n",
    "lipstick_crosstab = pd.crosstab(lipstick['prodType'], lipstick['priceCatgry'])\n",
    "\n",
    "lipstick_crosstab\n",
    "```\n",
    "\n",
    "The arguments for this function is the columns in your data frame you want to use to create the crosstab.\n",
    "\n",
    "And this is the result:\n",
    "\n",
    "![A window displays 2 lines of source code. The source code reads, In open square bracket 65 close square bracket 1 lipstick_crosstab. Out open square bracket 58 close square bracket. A table has 4 columns and 2 rows. The column heading represents priceCatgry 1, 2, 3. Row heading represents prodType LG, LP. The row entries are as follows. Row 1, 19, 43, 12. Row 2, 34, 92, 23.](Media/python12.png)\n",
    "\n",
    "The three price categories are on the top, and the two different product types are along the side.  What is shown in the cells are how many products fit in both categories.  For instance, there are 19 lip glosses less than 5 euros.\n",
    "\n",
    "---\n",
    "\n",
    "### Running the Independent Chi-Square\n",
    "\n",
    "Once you have the contingency table, then you can run the function ```stats.chi2_contingency``` on the contingency table you have created: \n",
    "\n",
    "```python\n",
    "stats.chi2_contingency(lipstick_crosstab)\n",
    "```\n",
    "\n",
    "And here is the output you'll receive: \n",
    "\n",
    "```text\n",
    "(0.2969891724608704,\n",
    " 0.8620046738525345,\n",
    " 2,\n",
    " array([[17.58744395, 44.79820628, 11.61434978],\n",
    "        [35.41255605, 90.20179372, 23.38565022]]))\n",
    "```\n",
    "\n",
    "At first, it looks like a jumble of things.  But the first value is your Chi-Square statistic.  The second value is your *p* value associated with that Chi-Square statistic.  Looking at this, it looks like there is not a significant relationship between product type and product price. Neither lipstick nor lip gloss is pricier or cheaper than the other.\n",
    "\n",
    "---\n",
    "\n",
    "### Test the Assumption of 5 Cases per Expected Cell\n",
    "\n",
    "The last piece of the output, labeled ```array```, is your expected count contingency table, albeit not a very pretty one! The expected count is what you would expect to happen if there was no relationship between the two variables.  Since all of these values are over 5, this means that the assumption has been met, and you are free to present and discuss these results without any limitations!\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d589e91a",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 12 - Hands-On Independent Chi-Square<a class=\"anchor\" id=\"DS105L1_page_12\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33ab4f03",
   "metadata": {},
   "source": [
    "For your Activity, you will be computing an independent Chi-Square to see if the shade of lipstick and the price category are related. `This Hands-On will not be graded`, but you are encouraged to complete it. The best way to become a great data scientist is to practice! Once you have submitted your project, you will be able to access the solution on the next page. Note that the solution will be slightly different from yours, but should look similar.\n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Do not submit your project until you have completed all requirements, as you will not be able to resubmit.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Requirements\n",
    "\n",
    "Using the **[lipstick dataset](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/lead_lipstick.zip)** you worked with in the lesson, determine if the shade of lipstick and the price category are related. To do this, you will need to: \n",
    "\n",
    "* Create a contingency table\n",
    "* Test for the assumption of 5 per cell in the expected contingency table\n",
    "* Compute an independent Chi-Square \n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Be sure to zip and submit your entire directory when finished!</p>\n",
    "    </div>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f319ce4f",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 13 - Hands-On Independent Chi-Square Solution<a class=\"anchor\" id=\"DS105L1_page_13\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cc52393",
   "metadata": {},
   "source": [
    "# Solution\n",
    "\n",
    "---\n",
    "\n",
    "## Create a Contingency Table\n",
    "\n",
    "First, create a contingency table: \n",
    "\n",
    "```python\n",
    "lipstick2_crosstab = pd.crosstab(lipstick['shade'], lipstick['priceCatgry'])\n",
    "```\n",
    "\n",
    "It should look like this:\n",
    "\n",
    "![A window displays a line of source code that reads, 1 lipstick2_crosstab. A table has 4 columns and 4 entries. The column heading represents priceCatgry 1, 2, 3. Row heading represents the shade Brown, Pink, Purple, Red. The row entries are as follows. Row 1, 20, 30, 10. Row 2, 20, 49, 12. Row 3, 8, 23, 6. Row 4, 5, 33, 7.](Media/python13.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Independent Chi-Square\n",
    "\n",
    "Then create the independent Chi-Square:\n",
    "\n",
    "```python\n",
    "stats.chi2_contingency(lipstick2_crosstab)\n",
    "```\n",
    "\n",
    "Here is the output:\n",
    "\n",
    "```text\n",
    "(7.860569553614045,\n",
    " 0.2484973879479863,\n",
    " 6,\n",
    " array([[14.26008969, 36.32286996,  9.41704036],\n",
    "        [19.25112108, 49.03587444, 12.71300448],\n",
    "        [ 8.79372197, 22.39910314,  5.80717489],\n",
    "        [10.69506726, 27.24215247,  7.06278027]]))\n",
    "```\n",
    "\n",
    "Lipstick color does not significantly affect the price of the product.  This is because the *p* value is > .05. \n",
    "\n",
    "---\n",
    "\n",
    "## Test the Assumption\n",
    "\n",
    "The expected counts were all greater than 5 in the output below, so this assumption has been met, and you are free share these results without any limitations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7bcd091",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 14 - Correlations<a class=\"anchor\" id=\"DS105L1_page_14\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5a00a59b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"720\"\n",
       "            height=\"480\"\n",
       "            src=\"https://player.vimeo.com/video/334043602\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.VimeoVideo at 0x1716f5640>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import VimeoVideo\n",
    "# Single Sample t-Test\n",
    "VimeoVideo('334043602', width=720, height=480)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c43c35fd",
   "metadata": {},
   "source": [
    "The approximate transcript and code for the above overview video **[is located here](https://repo.exeterlms.com/documents/V2/DataScience/Video-Transcripts/DSO105L01pg14tutorial.zip)**.\n",
    "\n",
    "# Correlation\n",
    "\n",
    "Correlations can be done on two continuous variables, to determine the relationship between them.  As a reminder, a correlation can be between zero and one, and either positive or negative.  The larger it is, the more closely related the two variables are. \n",
    "\n",
    "---\n",
    "## Import Packages\n",
    "\n",
    "In addition to the usual ```pandas```, you will also need to import ```pyplot``` and ```seaborn``` to be able to graphically visualize correlations. \n",
    "\n",
    "```python\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "```\n",
    "\n",
    "---\n",
    "## Import Data\n",
    "\n",
    "The data you'll be looking at is [data from cruise ship companies](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/cruise_ship.zip).  It has information on the size, age, and number of passengers. \n",
    "\n",
    "Here's what the data currently looks like: \n",
    "\n",
    "![A window displays a line of source code that reads, 1 cruise_ship.head open and close bracket. A table has 10 columns and 5 rows. Column headings are Ship, Line, YearBit, Tonnage, passngrs, Length, Cabins, Crew, PassSpcR, outcab.](Media/python14.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Run One Correlation\n",
    "\n",
    "Using the function ```.corr()```, it's easy to run a correlation on a selected two variables.  For instance, do you think that the number of passengers and the number of cabins on a cruise ship would relate to each other? \n",
    "\n",
    "```python\n",
    "cruise_ship['passngrs'].corr(cruise_ship['Cabins'])\n",
    "```\n",
    "\n",
    "The first thing you type is one of your variables, then you call ```.cor()``` and list the second variable.\n",
    "\n",
    "Here is the output: \n",
    "\n",
    "```text\n",
    "0.9763413679845939\n",
    "```\n",
    "\n",
    "It should not come as a surprise to you that the number of passengers and the number of cabins are very related. .97 is about as strong as a correlation gets! This correlation is also positive, which means that as the number of passengers increase, so does the number of cabins, and vice versa.  \n",
    "\n",
    "---\n",
    "\n",
    "## Create a Correlation Matrix\n",
    "\n",
    "Running one correlation can be nice, but sometimes you'd like to know how all your data relates to each other! In that case, call in the big guns, and go for a correlation matrix! \n",
    "\n",
    "---\n",
    "\n",
    "### Drop Non-Continuous Variables\n",
    "\n",
    "When you are creating a correlation matrix, you are feeding the code your entire data set.  But you can only run Pearson's correlations on continuous variables! So it's important to drop anything that is categorical or a string first. \n",
    "\n",
    "```python\n",
    "cruise_ship1 = cruise_ship.drop(['Ship', 'Line'], axis=1)\n",
    "```\n",
    "\n",
    "As you'll recall, the ```.drop()``` function will remove any columns specified in the square brackets, and the ```axis=1``` arguments tells Python that these are the names of columns (not rows). So you are getting rid of the ```Ship``` and ```Line``` columns here.\n",
    "\n",
    "You can confirm that everything worked as expected: \n",
    "\n",
    "![A window displays a line of source code that reads, 1 cruise_ship1.head open and close bracket. A table has 8 columns and 5 rows. Column headings are YearBit, Tonnage, passngrs, Length, Cabins, Crew, PassSpcR, outcab.](Media/python15.png)\n",
    "\n",
    "---\n",
    "\n",
    "### Using .corr() on An Entire Dataset\n",
    "\n",
    "You will not believe how easy it is to get all correlations at your fingertips! \n",
    "\n",
    "```python\n",
    "cruise_ship1.corr(method='pearson')\n",
    "```\n",
    "\n",
    "Just put in the name of your cleaned up dataset, call the ```.corr()``` function, and specify ```method='pearson'``` and away you go! Here's the result: \n",
    "\n",
    "![A window displays a line of source code that reads, 1 cruise_ship1.corr(method='pearson'). A table has 8 columns and 5 rows. Column headings are YearBit, Tonnage, passngrs, Length, Cabins, Crew, PassSpcR, outcab.](Media/python16.png)\n",
    "\n",
    "Remember that you read only the top right or bottom left of this matrix; everything repeats after the diagonal row of ones. \n",
    "\n",
    "---\n",
    "\n",
    "### Make .corr() Pretty!\n",
    "\n",
    "Not pretty enough for you? Difficult to make sense of rows and rows of numbers? Well, you're in luck.  Adding a couple arguments can help you interpret things and add a little visual interest.\n",
    "\n",
    "```python\n",
    "cruise_ship1.corr(method='pearson').style.format(\"{:.2}\").background_gradient(cmap=plt.get_cmap('coolwarm'), axis=1)\n",
    "```\n",
    "\n",
    "Start by using the ```.style.format()``` function.  In it, you will place ```{:.2}``` to specify the gradients.  Then you can use ```background_gradient()``` to specify the colors.  This pulls the ```coolwarm``` palette from ```matplotlib pyplot```.  ```cmap=``` stands for color map.  Then lastly, you need the argument ```axis=1``` so that Python knows you are focused on columns.\n",
    "\n",
    "Here is the final product: \n",
    "\n",
    "![A table has 8 columns and 5 rows. The column headings are YearBit, Tonnage, passngrs, Length, Cabins, Crew, PassSpcR, outcab. The row headings are YearBit, Tonnage, passngrs, Length, Cabins, Crew, PassSpcR, outcab.](Media/python17.png)\n",
    "\n",
    "---\n",
    "\n",
    "### Use sns.heatmap()\n",
    "\n",
    "You can also do a correlation matrix easily using the ```seaborn``` package: \n",
    "\n",
    "```python\n",
    "sns.heatmap(cruise_ship1.corr(), annot=True)\n",
    "```\n",
    "\n",
    "Just put in your dataset name as an argument, then call the ```.corr()``` function again, and use the argument ```annot=True``` to have the values printed on the plot. There's a little less customization here, but it's also a little simpler, with fewer arguments. Here's the output:\n",
    "\n",
    "![An 11 by 11 grid box with column headings are labeled as YearBit, Tonnage, passngrs, Length, Cabins, Crew, PassSpcR, outcab. The row headings are YearBit, Tonnage, passngrs, Length, Cabins, Crew, PassSpcR, outcab.](Media/python18.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Summary\n",
    "\n",
    "In this lesson, you learned how to perform all the basic statistics you have learned in MS Excel and R, including *t*-tests, Chi-Squares, and correlations.  It is important that you become proficient with the basics in all three languages! \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1632962e",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 15 - Hands-On Correlations<a class=\"anchor\" id=\"DS105L1_page_15\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecf33fbc",
   "metadata": {},
   "source": [
    "For your Activity, you will be computing correlations on a dataset of power lifting records. `This Hands-On will not be graded`, but you are encouraged to complete it. The best way to become a great data scientist is to practice! Once you have submitted your project, you will be able to access the solution on the next page. Note that the solution will be slightly different from yours, but should look similar.\n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Do not submit your project until you have completed all requirements, as you will not be able to resubmit.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Requirements\n",
    "\n",
    "Using the **[power_lifting dataset](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/power_lifting.zip)**, you will explore how the different variables are related with each other.  Use any means of correlation you like, and correlate any variables you like.  Make sure to note anything interesting or unusual that stands out to you, and interpret those correlations.  \n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Be sure to zip and submit your entire directory when finished!</p>\n",
    "    </div>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43046a21",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 16 - Hands-On Correlations Solution<a class=\"anchor\" id=\"DS105L1_page_16\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfc5b4f3",
   "metadata": {},
   "source": [
    "# Solution\n",
    "\n",
    "---\n",
    "\n",
    "## Drop Any Variables That Aren't Continuous \n",
    "\n",
    "In order to run a correlation matrix, you will need to drop any variables that aren't continuous.  In this dataset, that includes ```Name```, ```Sex```, ```Equipment```, and ```Division```:   \n",
    "\n",
    "```python\n",
    "power_lifting1 = power_lifting.drop(['Name', 'Sex', 'Equipment', 'Division'], axis=1)\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "## Correlations\n",
    "\n",
    "You could have performed correlations any way you liked! Below you will find the code for all four options, and their results. \n",
    "\n",
    "---\n",
    "\n",
    "### Individual Correlations\n",
    "\n",
    "If you want, you can look at correlations one by one, though this can be tiresome.  Here's an example of one pair: \n",
    "\n",
    "```python\n",
    "power_lifting1['BestBenchKg'].corr(power_lifting1['BodyweightKg'])\n",
    "```\n",
    "\n",
    "Here is the output:\n",
    "\n",
    "```text\n",
    "0.5844312818128141\n",
    "```\n",
    "\n",
    "---\n",
    "\n",
    "### .corr()\n",
    "\n",
    "Here's one way to do a correlation matrix: \n",
    "\n",
    "```python\n",
    "power_lifting1.corr(method='pearson').style.format(\"{:.2}\").background_gradient(cmap=plt.get_cmap('coolwarm'), axis=1)\n",
    "```\n",
    "\n",
    "If you didn't want it styled, you could leave oof the arguments after the first set of parentheses. Here's the output result: \n",
    "\n",
    "![A table has 8 columns and 5 rows. The column headings are MeetID, Age, Bodyweightkg, Squat4Kg, BestsquatKg, Bench4Kg, BestBenchKg, Deadlift4Kg, BestDeadliftKg, TotalKgKg, Wilks. The row headings are MeetID, Age, Bodyweightkg, Squat4Kg, BestsquatKg, Bench4Kg, BestBenchKg, Deadlift4Kg, BestDeadliftKg, TotalKgKg, Wilks.](Media/python19.png)\n",
    "\n",
    "---\n",
    "\n",
    "### sns.heatmap()\n",
    "\n",
    "Here's the other way to do a correlation matrix.\n",
    "\n",
    "```python\n",
    "sns.heatmap(power_lifting.corr(), annot=True)\n",
    "```\n",
    "\n",
    "It yields a graph that's a little more difficult to read, just because of the number of variables:\n",
    "\n",
    "![An 11 by 11 grid box with column headings are labeled as MeetID, Age, Bodyweightkg, Squat4Kg, BestsquatKg, Bench4Kg, BestBenchKg, Deadlift4Kg, BestDeadliftKg, TotalKgKg, Wilks. The column headings are MeetID, Age, Bodyweightkg, Squat4Kg, BestsquatKg, Bench4Kg, BestBenchKg, Deadlift4Kg, BestDeadliftKg, TotalKgKg, Wilks. all the diagonal boxes are labeled as 1.](Media/python20.png)\n",
    "\n",
    "---\n",
    "\n",
    "## Interpretation / Discussion\n",
    "\n",
    "For either chart, larger correlations are shown in warmer colors.  Ignore the diagonal line of dark color down the center, since this is just all the variables correlating with themselves.  Then focus on the darker ones and move down.  Looks like both body weight and dead lifting weight correlate pretty highly with a number of other factors.  \n",
    "\n",
    "Body weight is correlated with: \n",
    "* Best squat weight\n",
    "* Best bench weight\n",
    "* Best deadlift weight \n",
    "\n",
    "Best dead lift weight is correlated with: \n",
    "* Best squat weight\n",
    "* Best bench weight\n",
    "* Total kilograms\n",
    "* Wilks\n",
    "\n",
    "In addition, best squat is correlated with:\n",
    "* Best bench weight\n",
    "* Total kilograms\n",
    "* Wilks\n",
    "\n",
    "From this, it looks like the 4kg performances aren't greatly correlated with anything. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8261d0f1",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 17 - Key Terms<a class=\"anchor\" id=\"DS105L1_page_17\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58bf8460",
   "metadata": {},
   "source": [
    "# Key Terms\n",
    "\n",
    "Below is a list and short description of the important keywords learned in this lesson. Please read through and go back and review any concepts you do not fully understand. Great Work!\n",
    "\n",
    "---\n",
    "\n",
    "## Key Python Code\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Keyword</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>.hist()</td>\n",
    "        <td>An argument in pandas that allows you to create a histogram even when data is missing.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>stats.ttest_1samp()</td>\n",
    "        <td>Computes a single sample t-test.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>ttest_ind()</td>\n",
    "        <td>Computes an independent t-test.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>stats.ttest_rel()</td>\n",
    "        <td>Computes a dependent t-test.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>pd.crosstab()</td>\n",
    "        <td>Creates a contingency table in Python.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>stats.chi2_contingency()</td>\n",
    "        <td>Calculates an independent Chi-Square using a contingency table.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>.corr()</td>\n",
    "        <td>Creates a correlation, either on a particular variable or a whole dataset that is numeric. Takes the argument method= to specify the type of correlation. </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>sns.heatmap()</td>\n",
    "        <td>Creates a heatmap of the relationship between variables in an all-numeric set.  An additional argument of annot=True will allow you to see the correlation values.</td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "---\n",
    "\n",
    "## Key Python Packages\n",
    "\n",
    "<table class=\"table table-striped\">\n",
    "    <tr>\n",
    "        <th>Keyword</th>\n",
    "        <th>Description</th>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>scipy</td>\n",
    "        <td>A package often used for machine learning and statistics. </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>stats</td>\n",
    "        <td>A package within scipy that will conduct many different statistical tests, including t-tests.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>seaborn</td>\n",
    "        <td>A data visualization package usually abbreviated as sns.</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td style=\"font-weight: bold;\" nowrap>pyplot</td>\n",
    "        <td>A package within matplotlib that does stastical graphing.  Often abbreviated as plt. </td>\n",
    "    </tr>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d00e520e",
   "metadata": {},
   "source": [
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">\n",
    "\n",
    "# Page 18 - Hands-On Lesson 1 Review<a class=\"anchor\" id=\"DS105L1_page_18\"></a>\n",
    "\n",
    "[Back to Top](#DS105L1_toc)\n",
    "\n",
    "<hr style=\"height:10px;border-width:0;color:gray;background-color:gray\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d453e5bd",
   "metadata": {},
   "source": [
    "For this Hands On, you will be analyzing data about Anime.  \n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Do not submit your project until you have completed all requirements, as you will not be able to resubmit.</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n",
    "---\n",
    "\n",
    "## Requirements\n",
    "\n",
    "Using the **[anime dataset](https://repo.exeterlms.com/documents/V2/DataScience/Intermediate-Stats/anime.zip)** you worked with in the lesson, perform the appropriate analyses and answer the following questions in your Python file.  \n",
    "\n",
    "---\n",
    "\n",
    "### Is a Rating Score of 6.2 Different from the Mean in this Dataset? \n",
    "\n",
    "Use the variable ```score```.  \n",
    "\n",
    "---\n",
    "\n",
    "### Does Anime that is Still Airing Differ in Popularity from Anime that is No Longer Airing? \n",
    "\n",
    "Use the variables ```status``` and ```popularity```.\n",
    "\n",
    "---\n",
    "\n",
    "### Does the Source of the Anime Influence the Type of Anime?\n",
    "\n",
    "Use the variable ```source```, recoded to have four levels: \n",
    "* Manga\n",
    "* Book\n",
    "* Game\n",
    "* Listening\n",
    "\n",
    "And use the variable ```type```.\n",
    "\n",
    "---\n",
    "\n",
    "### How do the Variables about Popularity / Ranking Relate to Each Other? \n",
    "\n",
    "Use the following variables: \n",
    "\n",
    "* ```score```\n",
    "* ```scored_by```\n",
    "* ```rank```\n",
    "* ```popularity```\n",
    "* ```members```\n",
    "* ```favorites```\n",
    "\n",
    "<div class=\"panel panel-danger\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Caution!</h3>\n",
    "    </div>\n",
    "    <div class=\"panel-body\">\n",
    "        <p>Be sure to zip and submit your entire directory when finished!</p>\n",
    "    </div>\n",
    "</div>\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}