c06-Statistics-for-Data-Science-with-Python/res/Regression_Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkST0151ENSkillsNetwork20531532-2022-01-01\" target=\"_blank\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Regression Analysis**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **30** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of regression analysis is to describe the relationship between one set of variables called the dependent variables, and another set of variables, called independent or explanatory variables. When there is only one explanatory variable, it is called simple regression.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After completing this lab you will be able to:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   Import Libraries\n",
    "*   Regression analysis in place of the t-test\n",
    "*   Regression analysis in place of ANOVA\n",
    "*   Regression analysis in place of correlation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Libraries\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All Libraries required for this lab are listed below. The libraries pre-installed on Skills Network Labs are commented. If you run this notebook in a different environment, e.g. your desktop, you may need to uncomment and install certain libraries.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#install specific version of libraries used in lab\n",
    "#! mamba install pandas==1.3.3\n",
    "#! mamba install numpy=1.21.2\n",
    "#! mamba install scipy=1.7.1-y\n",
    "#!  mamba install seaborn=0.9.0-y\n",
    "#!  mamba install matplotlib=3.4.3-y\n",
    "#!  mamba install statsmodels=0.12.0-y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the libraries we need for the lab\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import statsmodels.api as sm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read in the csv file from the URL using the request library\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>minority</th>\n",
       "      <th>age</th>\n",
       "      <th>gender</th>\n",
       "      <th>credits</th>\n",
       "      <th>beauty</th>\n",
       "      <th>eval</th>\n",
       "      <th>division</th>\n",
       "      <th>native</th>\n",
       "      <th>tenure</th>\n",
       "      <th>students</th>\n",
       "      <th>allstudents</th>\n",
       "      <th>prof</th>\n",
       "      <th>PrimaryLast</th>\n",
       "      <th>vismin</th>\n",
       "      <th>female</th>\n",
       "      <th>single_credit</th>\n",
       "      <th>upper_division</th>\n",
       "      <th>English_speaker</th>\n",
       "      <th>tenured_prof</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>36</td>\n",
       "      <td>female</td>\n",
       "      <td>more</td>\n",
       "      <td>0.289916</td>\n",
       "      <td>4.3</td>\n",
       "      <td>upper</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>24</td>\n",
       "      <td>43</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>36</td>\n",
       "      <td>female</td>\n",
       "      <td>more</td>\n",
       "      <td>0.289916</td>\n",
       "      <td>3.7</td>\n",
       "      <td>upper</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>86</td>\n",
       "      <td>125</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>36</td>\n",
       "      <td>female</td>\n",
       "      <td>more</td>\n",
       "      <td>0.289916</td>\n",
       "      <td>3.6</td>\n",
       "      <td>upper</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>76</td>\n",
       "      <td>125</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>yes</td>\n",
       "      <td>36</td>\n",
       "      <td>female</td>\n",
       "      <td>more</td>\n",
       "      <td>0.289916</td>\n",
       "      <td>4.4</td>\n",
       "      <td>upper</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>77</td>\n",
       "      <td>123</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>no</td>\n",
       "      <td>59</td>\n",
       "      <td>male</td>\n",
       "      <td>more</td>\n",
       "      <td>-0.737732</td>\n",
       "      <td>4.5</td>\n",
       "      <td>upper</td>\n",
       "      <td>yes</td>\n",
       "      <td>yes</td>\n",
       "      <td>17</td>\n",
       "      <td>20</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  minority  age  gender credits    beauty  eval division native tenure  \\\n",
       "0      yes   36  female    more  0.289916   4.3    upper    yes    yes   \n",
       "1      yes   36  female    more  0.289916   3.7    upper    yes    yes   \n",
       "2      yes   36  female    more  0.289916   3.6    upper    yes    yes   \n",
       "3      yes   36  female    more  0.289916   4.4    upper    yes    yes   \n",
       "4       no   59    male    more -0.737732   4.5    upper    yes    yes   \n",
       "\n",
       "   students  allstudents  prof  PrimaryLast  vismin  female  single_credit  \\\n",
       "0        24           43     1            0       1       1              0   \n",
       "1        86          125     1            0       1       1              0   \n",
       "2        76          125     1            0       1       1              0   \n",
       "3        77          123     1            1       1       1              0   \n",
       "4        17           20     2            0       0       0              0   \n",
       "\n",
       "   upper_division  English_speaker  tenured_prof  \n",
       "0               1                1             1  \n",
       "1               1                1             1  \n",
       "2               1                1             1  \n",
       "3               1                1             1  \n",
       "4               1                1             1  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings_url = 'https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ST0151EN-SkillsNetwork/labs/teachingratings.csv'\n",
    "ratings_df = pd.read_csv(ratings_url)\n",
    "ratings_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lab Exercises\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, you will learn how to run regression analysis in place of the t-test, ANOVA, and correlation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regression with T-test: Using the teachers rating data set, does gender affect teaching evaluation rates?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initially, we had used the t-test to test if there was a statistical difference in evaluations for males and females, we are now going to use regression. We will state the null hypothesis:\n",
    "\n",
    "*   $H\\_0: β1$ = 0 (Gender has no effect on teaching evaluation scores)\n",
    "*   $H\\_1: β1$ is not equal to 0 (Gender has an effect on teaching evaluation scores)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the female variable. female = 1 and male = 0\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jupyterlab/conda/envs/python/lib/python3.7/site-packages/statsmodels/tsa/tsatools.py:142: FutureWarning: In a future version of pandas all arguments of concat except for the argument 'objs' will be keyword-only\n",
      "  x = pd.concat(x[::order], 1)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>OLS Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>          <td>eval</td>       <th>  R-squared:         </th> <td>   0.022</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>   0.020</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>   10.56</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>             <td>Thu, 11 Aug 2022</td> <th>  Prob (F-statistic):</th>  <td>0.00124</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                 <td>16:48:43</td>     <th>  Log-Likelihood:    </th> <td> -378.50</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Observations:</th>      <td>   463</td>      <th>  AIC:               </th> <td>   761.0</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Residuals:</th>          <td>   461</td>      <th>  BIC:               </th> <td>   769.3</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Model:</th>              <td>     1</td>      <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "     <td></td>       <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>const</th>  <td>    4.0690</td> <td>    0.034</td> <td>  121.288</td> <td> 0.000</td> <td>    4.003</td> <td>    4.135</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>female</th> <td>   -0.1680</td> <td>    0.052</td> <td>   -3.250</td> <td> 0.001</td> <td>   -0.270</td> <td>   -0.066</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "  <th>Omnibus:</th>       <td>17.625</td> <th>  Durbin-Watson:     </th> <td>   1.209</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Prob(Omnibus):</th> <td> 0.000</td> <th>  Jarque-Bera (JB):  </th> <td>  18.970</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Skew:</th>          <td>-0.496</td> <th>  Prob(JB):          </th> <td>7.60e-05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Kurtosis:</th>      <td> 2.981</td> <th>  Cond. No.          </th> <td>    2.47</td>\n",
       "</tr>\n",
       "</table><br/><br/>Notes:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified."
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                            OLS Regression Results                            \n",
       "==============================================================================\n",
       "Dep. Variable:                   eval   R-squared:                       0.022\n",
       "Model:                            OLS   Adj. R-squared:                  0.020\n",
       "Method:                 Least Squares   F-statistic:                     10.56\n",
       "Date:                Thu, 11 Aug 2022   Prob (F-statistic):            0.00124\n",
       "Time:                        16:48:43   Log-Likelihood:                -378.50\n",
       "No. Observations:                 463   AIC:                             761.0\n",
       "Df Residuals:                     461   BIC:                             769.3\n",
       "Df Model:                           1                                         \n",
       "Covariance Type:            nonrobust                                         \n",
       "==============================================================================\n",
       "                 coef    std err          t      P>|t|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "const          4.0690      0.034    121.288      0.000       4.003       4.135\n",
       "female        -0.1680      0.052     -3.250      0.001      -0.270      -0.066\n",
       "==============================================================================\n",
       "Omnibus:                       17.625   Durbin-Watson:                   1.209\n",
       "Prob(Omnibus):                  0.000   Jarque-Bera (JB):               18.970\n",
       "Skew:                          -0.496   Prob(JB):                     7.60e-05\n",
       "Kurtosis:                       2.981   Cond. No.                         2.47\n",
       "==============================================================================\n",
       "\n",
       "Notes:\n",
       "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
       "\"\"\""
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## X is the input variables (or independent variables)\n",
    "X = ratings_df['female']\n",
    "## y is the target/dependent variable\n",
    "y = ratings_df['eval']\n",
    "## add an intercept (beta_0) to our model\n",
    "X = sm.add_constant(X) \n",
    "\n",
    "model = sm.OLS(y, X).fit()\n",
    "predictions = model.predict(X)\n",
    "\n",
    "# Print out the statistics\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Conclusion:** Like the t-test, the p-value is less than the alpha (α) level = 0.05, so we reject the null hypothesis as there is evidence that there is a difference in mean evaluation scores based on gender. The coefficient -0.1680 means that females get 0.168 scores less than men.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Regression with ANOVA: Using the teachers' rating data set, does beauty  score for instructors  differ by age?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "State the Hypothesis:\n",
    "\n",
    "*   $H\\_0: µ1 = µ2 = µ3$ (the three population means are equal)\n",
    "*   $H\\_1:$ At least one of the means differ\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we group the data like we did with ANOVA\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "ratings_df.loc[(ratings_df['age'] <= 40), 'age_group'] = '40 years and younger'\n",
    "ratings_df.loc[(ratings_df['age'] > 40)&(ratings_df['age'] < 57), 'age_group'] = 'between 40 and 57 years'\n",
    "ratings_df.loc[(ratings_df['age'] >= 57), 'age_group'] = '57 years and older'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use OLS function from the statsmodel library\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              df      sum_sq    mean_sq          F        PR(>F)\n",
      "age_group    2.0   20.422744  10.211372  17.597559  4.322549e-08\n",
      "Residual   460.0  266.925153   0.580272        NaN           NaN\n"
     ]
    }
   ],
   "source": [
    "from statsmodels.formula.api import ols\n",
    "lm = ols('beauty ~ age_group', data = ratings_df).fit()\n",
    "table= sm.stats.anova_lm(lm)\n",
    "print(table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Conclusion:** We can also see the same values for ANOVA like before and we will reject the null hypothesis since the p-value is less than 0.05 there is significant evidence that at least one of the means differ.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regression with ANOVA option 2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create dummy variables - A dummy variable is a numeric variable that represents categorical data, such as gender, race, etc. Dummy variables are dichotomous, i.e they can take on only two quantitative values.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = pd.get_dummies(ratings_df[['age_group']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age_group_40 years and younger</th>\n",
       "      <th>age_group_57 years and older</th>\n",
       "      <th>age_group_between 40 and 57 years</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>458</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>459</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>460</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>461</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>462</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>463 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     age_group_40 years and younger  age_group_57 years and older  \\\n",
       "0                                 1                             0   \n",
       "1                                 1                             0   \n",
       "2                                 1                             0   \n",
       "3                                 1                             0   \n",
       "4                                 0                             1   \n",
       "..                              ...                           ...   \n",
       "458                               1                             0   \n",
       "459                               0                             1   \n",
       "460                               0                             0   \n",
       "461                               1                             0   \n",
       "462                               0                             0   \n",
       "\n",
       "     age_group_between 40 and 57 years  \n",
       "0                                    0  \n",
       "1                                    0  \n",
       "2                                    0  \n",
       "3                                    0  \n",
       "4                                    0  \n",
       "..                                 ...  \n",
       "458                                  0  \n",
       "459                                  0  \n",
       "460                                  1  \n",
       "461                                  0  \n",
       "462                                  1  \n",
       "\n",
       "[463 rows x 3 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jupyterlab/conda/envs/python/lib/python3.7/site-packages/statsmodels/tsa/tsatools.py:142: FutureWarning: In a future version of pandas all arguments of concat except for the argument 'objs' will be keyword-only\n",
      "  x = pd.concat(x[::order], 1)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>OLS Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>         <td>beauty</td>      <th>  R-squared:         </th> <td>   0.071</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>   0.067</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>   17.60</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>             <td>Thu, 11 Aug 2022</td> <th>  Prob (F-statistic):</th> <td>4.32e-08</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                 <td>16:56:33</td>     <th>  Log-Likelihood:    </th> <td> -529.47</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Observations:</th>      <td>   463</td>      <th>  AIC:               </th> <td>   1065.</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Residuals:</th>          <td>   460</td>      <th>  BIC:               </th> <td>   1077.</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Model:</th>              <td>     2</td>      <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "                  <td></td>                     <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>const</th>                             <td>    0.0138</td> <td>    0.028</td> <td>    0.496</td> <td> 0.620</td> <td>   -0.041</td> <td>    0.069</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>age_group_40 years and younger</th>    <td>    0.3224</td> <td>    0.058</td> <td>    5.574</td> <td> 0.000</td> <td>    0.209</td> <td>    0.436</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>age_group_57 years and older</th>      <td>   -0.2596</td> <td>    0.056</td> <td>   -4.621</td> <td> 0.000</td> <td>   -0.370</td> <td>   -0.149</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>age_group_between 40 and 57 years</th> <td>   -0.0489</td> <td>    0.045</td> <td>   -1.081</td> <td> 0.280</td> <td>   -0.138</td> <td>    0.040</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "  <th>Omnibus:</th>       <td>11.586</td> <th>  Durbin-Watson:     </th> <td>   0.434</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Prob(Omnibus):</th> <td> 0.003</td> <th>  Jarque-Bera (JB):  </th> <td>  12.114</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Skew:</th>          <td> 0.394</td> <th>  Prob(JB):          </th> <td> 0.00234</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Kurtosis:</th>      <td> 2.913</td> <th>  Cond. No.          </th> <td>8.36e+15</td>\n",
       "</tr>\n",
       "</table><br/><br/>Notes:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.<br/>[2] The smallest eigenvalue is 9.18e-30. This might indicate that there are<br/>strong multicollinearity problems or that the design matrix is singular."
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                            OLS Regression Results                            \n",
       "==============================================================================\n",
       "Dep. Variable:                 beauty   R-squared:                       0.071\n",
       "Model:                            OLS   Adj. R-squared:                  0.067\n",
       "Method:                 Least Squares   F-statistic:                     17.60\n",
       "Date:                Thu, 11 Aug 2022   Prob (F-statistic):           4.32e-08\n",
       "Time:                        16:56:33   Log-Likelihood:                -529.47\n",
       "No. Observations:                 463   AIC:                             1065.\n",
       "Df Residuals:                     460   BIC:                             1077.\n",
       "Df Model:                           2                                         \n",
       "Covariance Type:            nonrobust                                         \n",
       "=====================================================================================================\n",
       "                                        coef    std err          t      P>|t|      [0.025      0.975]\n",
       "-----------------------------------------------------------------------------------------------------\n",
       "const                                 0.0138      0.028      0.496      0.620      -0.041       0.069\n",
       "age_group_40 years and younger        0.3224      0.058      5.574      0.000       0.209       0.436\n",
       "age_group_57 years and older         -0.2596      0.056     -4.621      0.000      -0.370      -0.149\n",
       "age_group_between 40 and 57 years    -0.0489      0.045     -1.081      0.280      -0.138       0.040\n",
       "==============================================================================\n",
       "Omnibus:                       11.586   Durbin-Watson:                   0.434\n",
       "Prob(Omnibus):                  0.003   Jarque-Bera (JB):               12.114\n",
       "Skew:                           0.394   Prob(JB):                      0.00234\n",
       "Kurtosis:                       2.913   Cond. No.                     8.36e+15\n",
       "==============================================================================\n",
       "\n",
       "Notes:\n",
       "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
       "[2] The smallest eigenvalue is 9.18e-30. This might indicate that there are\n",
       "strong multicollinearity problems or that the design matrix is singular.\n",
       "\"\"\""
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = ratings_df['beauty']\n",
    "## add an intercept (beta_0) to our model\n",
    "X = sm.add_constant(X) \n",
    "\n",
    "model = sm.OLS(y, X).fit()\n",
    "predictions = model.predict(X)\n",
    "\n",
    "# Print out the statistics\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will get the same results and conclusion\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlation: Using the teachers' rating dataset, Is teaching evaluation score correlated with beauty score?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jupyterlab/conda/envs/python/lib/python3.7/site-packages/statsmodels/tsa/tsatools.py:142: FutureWarning: In a future version of pandas all arguments of concat except for the argument 'objs' will be keyword-only\n",
      "  x = pd.concat(x[::order], 1)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>OLS Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>          <td>eval</td>       <th>  R-squared:         </th> <td>   0.036</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>   0.034</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>   17.08</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>             <td>Thu, 11 Aug 2022</td> <th>  Prob (F-statistic):</th> <td>4.25e-05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                 <td>16:57:13</td>     <th>  Log-Likelihood:    </th> <td> -375.32</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Observations:</th>      <td>   463</td>      <th>  AIC:               </th> <td>   754.6</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Residuals:</th>          <td>   461</td>      <th>  BIC:               </th> <td>   762.9</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Model:</th>              <td>     1</td>      <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "     <td></td>       <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>const</th>  <td>    3.9983</td> <td>    0.025</td> <td>  157.727</td> <td> 0.000</td> <td>    3.948</td> <td>    4.048</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>beauty</th> <td>    0.1330</td> <td>    0.032</td> <td>    4.133</td> <td> 0.000</td> <td>    0.070</td> <td>    0.196</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "  <th>Omnibus:</th>       <td>15.399</td> <th>  Durbin-Watson:     </th> <td>   1.238</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Prob(Omnibus):</th> <td> 0.000</td> <th>  Jarque-Bera (JB):  </th> <td>  16.405</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Skew:</th>          <td>-0.453</td> <th>  Prob(JB):          </th> <td>0.000274</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Kurtosis:</th>      <td> 2.831</td> <th>  Cond. No.          </th> <td>    1.27</td>\n",
       "</tr>\n",
       "</table><br/><br/>Notes:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified."
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                            OLS Regression Results                            \n",
       "==============================================================================\n",
       "Dep. Variable:                   eval   R-squared:                       0.036\n",
       "Model:                            OLS   Adj. R-squared:                  0.034\n",
       "Method:                 Least Squares   F-statistic:                     17.08\n",
       "Date:                Thu, 11 Aug 2022   Prob (F-statistic):           4.25e-05\n",
       "Time:                        16:57:13   Log-Likelihood:                -375.32\n",
       "No. Observations:                 463   AIC:                             754.6\n",
       "Df Residuals:                     461   BIC:                             762.9\n",
       "Df Model:                           1                                         \n",
       "Covariance Type:            nonrobust                                         \n",
       "==============================================================================\n",
       "                 coef    std err          t      P>|t|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "const          3.9983      0.025    157.727      0.000       3.948       4.048\n",
       "beauty         0.1330      0.032      4.133      0.000       0.070       0.196\n",
       "==============================================================================\n",
       "Omnibus:                       15.399   Durbin-Watson:                   1.238\n",
       "Prob(Omnibus):                  0.000   Jarque-Bera (JB):               16.405\n",
       "Skew:                          -0.453   Prob(JB):                     0.000274\n",
       "Kurtosis:                       2.831   Cond. No.                         1.27\n",
       "==============================================================================\n",
       "\n",
       "Notes:\n",
       "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
       "\"\"\""
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## X is the input variables (or independent variables)\n",
    "X = ratings_df['beauty']\n",
    "## y is the target/dependent variable\n",
    "y = ratings_df['eval']\n",
    "## add an intercept (beta_0) to our model\n",
    "X = sm.add_constant(X) \n",
    "\n",
    "model = sm.OLS(y, X).fit()\n",
    "predictions = model.predict(X)\n",
    "\n",
    "# Print out the statistics\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Conclusion:** p < 0.05 there is evidence of correlation between beauty and evaluation scores\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practice Questions\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1: Using the teachers' rating data set, does tenure affect beauty scores?\n",
    "\n",
    "*   Use α = 0.05\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jupyterlab/conda/envs/python/lib/python3.7/site-packages/statsmodels/tsa/tsatools.py:142: FutureWarning: In a future version of pandas all arguments of concat except for the argument 'objs' will be keyword-only\n",
      "  x = pd.concat(x[::order], 1)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>OLS Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>         <td>beauty</td>      <th>  R-squared:         </th> <td>   0.000</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>  -0.002</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>  0.1689</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>             <td>Thu, 11 Aug 2022</td> <th>  Prob (F-statistic):</th>  <td> 0.681</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                 <td>17:05:04</td>     <th>  Log-Likelihood:    </th> <td> -546.45</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Observations:</th>      <td>   463</td>      <th>  AIC:               </th> <td>   1097.</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Residuals:</th>          <td>   461</td>      <th>  BIC:               </th> <td>   1105.</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Model:</th>              <td>     1</td>      <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td></td>          <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>const</th>        <td>    0.0284</td> <td>    0.078</td> <td>    0.363</td> <td> 0.717</td> <td>   -0.125</td> <td>    0.182</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>tenured_prof</th> <td>   -0.0364</td> <td>    0.089</td> <td>   -0.411</td> <td> 0.681</td> <td>   -0.210</td> <td>    0.138</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "  <th>Omnibus:</th>       <td>23.184</td> <th>  Durbin-Watson:     </th> <td>   0.461</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Prob(Omnibus):</th> <td> 0.000</td> <th>  Jarque-Bera (JB):  </th> <td>  23.229</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Skew:</th>          <td> 0.507</td> <th>  Prob(JB):          </th> <td>9.03e-06</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Kurtosis:</th>      <td> 2.583</td> <th>  Cond. No.          </th> <td>    4.05</td>\n",
       "</tr>\n",
       "</table><br/><br/>Notes:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified."
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                            OLS Regression Results                            \n",
       "==============================================================================\n",
       "Dep. Variable:                 beauty   R-squared:                       0.000\n",
       "Model:                            OLS   Adj. R-squared:                 -0.002\n",
       "Method:                 Least Squares   F-statistic:                    0.1689\n",
       "Date:                Thu, 11 Aug 2022   Prob (F-statistic):              0.681\n",
       "Time:                        17:05:04   Log-Likelihood:                -546.45\n",
       "No. Observations:                 463   AIC:                             1097.\n",
       "Df Residuals:                     461   BIC:                             1105.\n",
       "Df Model:                           1                                         \n",
       "Covariance Type:            nonrobust                                         \n",
       "================================================================================\n",
       "                   coef    std err          t      P>|t|      [0.025      0.975]\n",
       "--------------------------------------------------------------------------------\n",
       "const            0.0284      0.078      0.363      0.717      -0.125       0.182\n",
       "tenured_prof    -0.0364      0.089     -0.411      0.681      -0.210       0.138\n",
       "==============================================================================\n",
       "Omnibus:                       23.184   Durbin-Watson:                   0.461\n",
       "Prob(Omnibus):                  0.000   Jarque-Bera (JB):               23.229\n",
       "Skew:                           0.507   Prob(JB):                     9.03e-06\n",
       "Kurtosis:                       2.583   Cond. No.                         4.05\n",
       "==============================================================================\n",
       "\n",
       "Notes:\n",
       "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
       "\"\"\""
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### insert code here\n",
    "## X is the input variables (or independent variables)\n",
    "X = ratings_df['tenured_prof']\n",
    "## y is the target/dependent variable\n",
    "y = ratings_df['beauty']\n",
    "## add an intercept (beta_0) to our model\n",
    "X = sm.add_constant(X) \n",
    "\n",
    "model = sm.OLS(y, X).fit()\n",
    "predictions = model.predict(X)\n",
    "\n",
    "# Print out the statistics\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-click **here** for a hint.\n",
    "\n",
    "<!-- The hint is below:\n",
    "## State Hypothesis\n",
    "Null Hypothesis: Mean beauty scores for tenured and non-tenured instructors are equal\n",
    "Alternative Hypothesis: There is a difference in mean beauty scores for tenured and non-tenured instructors\n",
    "\n",
    "## use the dummy variable for tenure - the OLS library doesn't recognize texts\n",
    "X = ratings_df['tenured_prof']\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-click **here** for the solution.\n",
    "\n",
    "<!-- The answer is below:\n",
    "## put beauty scores in a list\n",
    "y = ratings_df['beauty']\n",
    "## add an intercept (beta_0) to our model\n",
    "X = sm.add_constant(X) \n",
    "\n",
    "model = sm.OLS(y, X).fit()\n",
    "predictions = model.predict(X)\n",
    "\n",
    "# Print out the statistics\n",
    "model.summary()\n",
    "p-value is greater than 0.05, so we fail to reject the null hypothesis as there is no evidence that the mean difference of tenured and untenured instructors are different\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2: Using the teachers' rating data set, does being an English speaker affect the number of students assigned to professors?\n",
    "\n",
    "*   Use \"allstudents\"\n",
    "*   Use α = 0.05 and α = 0.1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jupyterlab/conda/envs/python/lib/python3.7/site-packages/statsmodels/tsa/tsatools.py:142: FutureWarning: In a future version of pandas all arguments of concat except for the argument 'objs' will be keyword-only\n",
      "  x = pd.concat(x[::order], 1)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>OLS Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>       <td>allstudents</td>   <th>  R-squared:         </th> <td>   0.007</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>   0.005</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>   3.476</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>             <td>Thu, 11 Aug 2022</td> <th>  Prob (F-statistic):</th>  <td>0.0629</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                 <td>17:06:55</td>     <th>  Log-Likelihood:    </th> <td> -2654.2</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Observations:</th>      <td>   463</td>      <th>  AIC:               </th> <td>   5312.</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Residuals:</th>          <td>   461</td>      <th>  BIC:               </th> <td>   5321.</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Model:</th>              <td>     1</td>      <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "         <td></td>            <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>const</th>           <td>   29.6071</td> <td>   14.150</td> <td>    2.092</td> <td> 0.037</td> <td>    1.802</td> <td>   57.413</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>English_speaker</th> <td>   27.2158</td> <td>   14.598</td> <td>    1.864</td> <td> 0.063</td> <td>   -1.471</td> <td>   55.902</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "  <th>Omnibus:</th>       <td>429.792</td> <th>  Durbin-Watson:     </th> <td>   0.708</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Prob(Omnibus):</th> <td> 0.000</td>  <th>  Jarque-Bera (JB):  </th> <td>10527.126</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Skew:</th>          <td> 4.129</td>  <th>  Prob(JB):          </th> <td>    0.00</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Kurtosis:</th>      <td>24.852</td>  <th>  Cond. No.          </th> <td>    8.01</td> \n",
       "</tr>\n",
       "</table><br/><br/>Notes:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified."
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                            OLS Regression Results                            \n",
       "==============================================================================\n",
       "Dep. Variable:            allstudents   R-squared:                       0.007\n",
       "Model:                            OLS   Adj. R-squared:                  0.005\n",
       "Method:                 Least Squares   F-statistic:                     3.476\n",
       "Date:                Thu, 11 Aug 2022   Prob (F-statistic):             0.0629\n",
       "Time:                        17:06:55   Log-Likelihood:                -2654.2\n",
       "No. Observations:                 463   AIC:                             5312.\n",
       "Df Residuals:                     461   BIC:                             5321.\n",
       "Df Model:                           1                                         \n",
       "Covariance Type:            nonrobust                                         \n",
       "===================================================================================\n",
       "                      coef    std err          t      P>|t|      [0.025      0.975]\n",
       "-----------------------------------------------------------------------------------\n",
       "const              29.6071     14.150      2.092      0.037       1.802      57.413\n",
       "English_speaker    27.2158     14.598      1.864      0.063      -1.471      55.902\n",
       "==============================================================================\n",
       "Omnibus:                      429.792   Durbin-Watson:                   0.708\n",
       "Prob(Omnibus):                  0.000   Jarque-Bera (JB):            10527.126\n",
       "Skew:                           4.129   Prob(JB):                         0.00\n",
       "Kurtosis:                      24.852   Cond. No.                         8.01\n",
       "==============================================================================\n",
       "\n",
       "Notes:\n",
       "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
       "\"\"\""
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## insert code here\n",
    "## X is the input variables (or independent variables)\n",
    "X = ratings_df['English_speaker']\n",
    "## y is the target/dependent variable\n",
    "y = ratings_df['allstudents']\n",
    "## add an intercept (beta_0) to our model\n",
    "X = sm.add_constant(X) \n",
    "\n",
    "model = sm.OLS(y, X).fit()\n",
    "predictions = model.predict(X)\n",
    "\n",
    "# Print out the statistics\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-click **here** for a hint.\n",
    "\n",
    "<!-- The hint is below:\n",
    "## State Hypothesis\n",
    "Null Hypothesis: Mean number of students assigned to native English speakers vs non-native English speakers are equal\n",
    "Alternative Hypothesis: There is a difference in mean number of students assigned to native English speakers vs non-native English speakers\n",
    "\n",
    "## Is the instructor a native English speaker - make sure to use the binary variable \"English speaker\"\n",
    "X = ratings_df['English_speaker']\n",
    "## You can use the students or all students variable\n",
    "y = ratings_df['allstudents']\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-click **here** for the solution.\n",
    "\n",
    "<!-- The answer is below:\n",
    "## add an intercept (beta_0) to our model\n",
    "X = sm.add_constant(X) \n",
    "\n",
    "model = sm.OLS(y, X).fit()\n",
    "predictions = model.predict(X)\n",
    "\n",
    "# Print out the statistics\n",
    "model.summary()\n",
    "\n",
    "At α = 0.05, p-value is greater, we fail to reject the null hypothesis as there is no evidence that being a native English speaker or a non-native English speaker affects the number of students assigned to an instructor.\n",
    "At α = 0.1, p-value is less, we reject the null hypothesis as there is evidence that there is a significant difference of mean number of students assigned to native English speakers vs non-native English speakers.\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3: Using the teachers' rating data set, what is the correlation between the number of students who participated in the evaluation survey and evaluation scores?\n",
    "\n",
    "*   Use \"students\" variable\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jupyterlab/conda/envs/python/lib/python3.7/site-packages/statsmodels/tsa/tsatools.py:142: FutureWarning: In a future version of pandas all arguments of concat except for the argument 'objs' will be keyword-only\n",
      "  x = pd.concat(x[::order], 1)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>OLS Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>          <td>eval</td>       <th>  R-squared:         </th> <td>   0.001</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>  -0.001</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>  0.5806</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>             <td>Thu, 11 Aug 2022</td> <th>  Prob (F-statistic):</th>  <td> 0.446</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                 <td>17:08:27</td>     <th>  Log-Likelihood:    </th> <td> -383.46</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Observations:</th>      <td>   463</td>      <th>  AIC:               </th> <td>   770.9</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Residuals:</th>          <td>   461</td>      <th>  BIC:               </th> <td>   779.2</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Model:</th>              <td>     1</td>      <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>        <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>const</th>    <td>    3.9823</td> <td>    0.033</td> <td>  119.689</td> <td> 0.000</td> <td>    3.917</td> <td>    4.048</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>students</th> <td>    0.0004</td> <td>    0.001</td> <td>    0.762</td> <td> 0.446</td> <td>   -0.001</td> <td>    0.002</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "  <th>Omnibus:</th>       <td>15.259</td> <th>  Durbin-Watson:     </th> <td>   1.198</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Prob(Omnibus):</th> <td> 0.000</td> <th>  Jarque-Bera (JB):  </th> <td>  16.283</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Skew:</th>          <td>-0.456</td> <th>  Prob(JB):          </th> <td>0.000291</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Kurtosis:</th>      <td> 2.888</td> <th>  Cond. No.          </th> <td>    74.8</td>\n",
       "</tr>\n",
       "</table><br/><br/>Notes:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified."
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                            OLS Regression Results                            \n",
       "==============================================================================\n",
       "Dep. Variable:                   eval   R-squared:                       0.001\n",
       "Model:                            OLS   Adj. R-squared:                 -0.001\n",
       "Method:                 Least Squares   F-statistic:                    0.5806\n",
       "Date:                Thu, 11 Aug 2022   Prob (F-statistic):              0.446\n",
       "Time:                        17:08:27   Log-Likelihood:                -383.46\n",
       "No. Observations:                 463   AIC:                             770.9\n",
       "Df Residuals:                     461   BIC:                             779.2\n",
       "Df Model:                           1                                         \n",
       "Covariance Type:            nonrobust                                         \n",
       "==============================================================================\n",
       "                 coef    std err          t      P>|t|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "const          3.9823      0.033    119.689      0.000       3.917       4.048\n",
       "students       0.0004      0.001      0.762      0.446      -0.001       0.002\n",
       "==============================================================================\n",
       "Omnibus:                       15.259   Durbin-Watson:                   1.198\n",
       "Prob(Omnibus):                  0.000   Jarque-Bera (JB):               16.283\n",
       "Skew:                          -0.456   Prob(JB):                     0.000291\n",
       "Kurtosis:                       2.888   Cond. No.                         74.8\n",
       "==============================================================================\n",
       "\n",
       "Notes:\n",
       "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
       "\"\"\""
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## insert code here\n",
    "## X is the input variables (or independent variables)\n",
    "X = ratings_df['students']\n",
    "## y is the target/dependent variable\n",
    "y = ratings_df['eval']\n",
    "## add an intercept (beta_0) to our model\n",
    "X = sm.add_constant(X) \n",
    "\n",
    "model = sm.OLS(y, X).fit()\n",
    "predictions = model.predict(X)\n",
    "\n",
    "# Print out the statistics\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-click **here** for a hint.\n",
    "\n",
    "<!-- The hint is below:\n",
    "## create a list of students and evaluation socres\n",
    "X = ratings_df['students']\n",
    "y = ratings_df['eval']\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Double-click **here** for the solution.\n",
    "\n",
    "<!-- The rest of the answer is below:\n",
    "## add an intercept (beta_0) to our model\n",
    "X = sm.add_constant(X) \n",
    "\n",
    "model = sm.OLS(y, X).fit()\n",
    "predictions = model.predict(X)\n",
    "\n",
    "# Print out the statistics\n",
    "model.summary()\n",
    "\n",
    "R-square is 0.001, R will be √0.001, correlation coefficient is 0.03 (close to 0). There is a very weak correlation between the number of students who participated in the evaluation survey and evaluation scores\n",
    "-->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Aije Egwaikhide](https://www.linkedin.com/in/aije-egwaikhide/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkST0151ENSkillsNetwork20531532-2022-01-01) is a Data Scientist at IBM who holds a degree in Economics and Statistics from the University of Manitoba and a Post-grad in Business Analytics from St. Lawrence College, Kingston. She is a current employee of IBM where she started as a Junior Data Scientist at the Global Business Services (GBS) in 2018. Her main role was making meaning out of data for their Oil and Gas clients through basic statistics and advanced Machine Learning algorithms. The highlight of her time in GBS was creating a customized end-to-end Machine learning and Statistics solution on optimizing operations in the Oil and Gas wells. She moved to the Cognitive Systems Group as a Senior Data Scientist where she will be providing the team with actionable insights using Data Science techniques and further improve processes through building machine learning solutions. She recently joined the IBM Developer Skills Network group where she brings her real-world experience to the courses she creates.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Date (YYYY-MM-DD) | Version | Changed By      | Change Description                     |\n",
    "| ----------------- | ------- | --------------- | -------------------------------------- |\n",
    "| 2020-08-14        | 0.1     | Aije Egwaikhide | Created the initial version of the lab |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2020 IBM Corporation. This notebook and its source code are released under the terms of the [MIT License](https://cognitiveclass.ai/mit-license/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkST0151ENSkillsNetwork20531532-2022-01-01).\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}