02-linear-models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Linear models\n",
    "\n",
    "In this exercise, we will implement the linear models that appear in the lecture slides ourselves.\n",
    "We will start with the Iris data that you already know from the lecture.   \n",
    "\n",
    "*Exercise*: Load the dataset \"iris.csv\" into a dataframe using the `pd.read_csv()` method from the pandas package!   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt \n",
    "import seaborn as sns\n",
    "\n",
    "# load the iris dataset into a dataframe\n",
    "df = \"Implement me!\"\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's investigate how our data actually looks.   \n",
    "\n",
    "*Exercise*: Do some exploratory data analysis (EDA). Look for example what columns `df` has, what datatypes the single columns are made off and check for missing values. What else can you say about the dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"Implement me!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1. Simple linear regression\n",
    "\n",
    "First, let's look at the simple linear model that represents the relationship between PetalLength and SepalWidth. We will use the Python package statsmodels (`https://www.statsmodels.org`) to fit linear models. The specification of linear models works very similarly to the R examples in the lecture.\n",
    "\n",
    "To better understand the `ols` formulas, it is worth taking a look here: (https://patsy.readthedocs.io/en/latest/formulas.html)\n",
    "\n",
    "*Exercise*: Write down the general equation for regression in this markdown cell. Afterwards, modify the formula to represent the relationship between PetalLength and SepalWidth and plug it into `smf.ols()`.   \n",
    "\n",
    "*Solution*:   \n",
    "* Regression equation = Todo   \n",
    "* For our example: Todo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import statsmodels.api as sm\n",
    "import statsmodels.formula.api as smf\n",
    "\n",
    "# initialize the linear regression model\n",
    "model = smf.ols('Implement me!', data=df)\n",
    "# fit/train the model\n",
    "results = model.fit()\n",
    "# print the results of the model\n",
    "print(results.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Exercise*: Now fit the model and interpret the results! What do the coefficients, $R^2$, p-values mean? (see slide 17) Write everything down in this markdown cell!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2. Regularization\n",
    "\n",
    "Now import the data from 'reg_data.csv' as a DataFrame and look what types of columns we have!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2 = 'Implement me!'\n",
    "df2.info()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Exercise*: Fit a linear model with y as the dependent variable and x1 and x2 as independent variables. Interpret the results again. What is the problem?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize and train the model in one step -> save results\n",
    "results = smf.ols('Implement me!', data=df2).fit()\n",
    "# get the parameters of the trained model\n",
    "results.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just a markdown cell to take notes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Exercise*: Have a deeper look into your data and tell me what problem we have with our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "'Implement me!'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Exercise*: Use ridge regression/regularization to fit your linear model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize the model\n",
    "model = smf.ols('Implement me!', data=df)\n",
    "#train the model and save the results\n",
    "results = model.fit_regularized(L1_wt=0.1,alpha=1.9)\n",
    "# get the parameters of the trained model\n",
    "results.params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3. Logitic regression\n",
    "\n",
    "In machine learning, we are often not so much interested in the parameters of the model, but in the fact that the model provides good predictions. For such purposes, the logistic regression from the `sklearn` package is better suited -- there, things that are quite tedious in statsmodels, such as encoding the class, dealing with more than two classes, etc., are done directly. Have a look at the following documentation: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html\n",
    "\n",
    "*Exercise*: Divide the Iris dataset that you already imported in `df` into two sets which contain the feature $X$ and target values $y$ respectively. Further seperate these two sets into two subsets: A training set that contains 80% of the data and a test set that contains the remaining 20% of the data. Use the `train_test_split()`-method of `sklearn` for this purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# split the iris data from 'df' into features (X) and target (y) value\n",
    "X = \"Implement me!\"\n",
    "y = \"Implement me!\"\n",
    "# split X and y into train and test sets\n",
    "X_train, X_test, y_train, y_test = \"Implement me!\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Exercise*: Train a logit model with the training data and generate predictions for the test data. Use the `LogisticRegression` class and its respective methods for this purpose. Afterwards look at the results by using the methods provided by `sklearn.metrics`. How many test examples have been classified correctly by your trained model?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "\n",
    "# Train the classifier\n",
    "clf = \"Implement me!\"\n",
    "\n",
    "# compute predictions with your trained model and save these predictions in yhat\n",
    "yhat = \"Implement me!\"\n",
    "\n",
    "# compute and print some classification metrics\n",
    "print(\"Implement me!\")\n",
    "print(\"Implement me!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Additional exercise*: There is a way to also get the coefficients of a `LogisticRegression` model in sklearn. Try to figure out how to get these parameters. How many $\\beta_i$  do we have in total?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameter = ['theta_'+str(i) for i in range(X_train.shape[1]+1)]\n",
    "columns = ['intersect:x_0=1'] + list(X.columns.values)\n",
    "sk_theta = []\n",
    "for i in range (len(clf.intercept_)):\n",
    "    theta = [clf.intercept_[i]]\n",
    "    theta.extend(clf.coef_[i])\n",
    "    sk_theta.append(theta)\n",
    "parameter_df = pd.DataFrame({'Parameter':parameter,'Columns':columns,'c1':sk_theta[0], 'c2':sk_theta[1], 'c3':sk_theta[2]})\n",
    "parameter_df"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}