lecture_7_machine_learning_intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Why Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[40 Must-Read AI](https://www.springboard.com/blog/machine-learning-blog/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Best Science of 21st century\n",
    "behind almost every technology\n",
    "High demand of machine learning professionals\n",
    "Interesting field of computer science\n",
    "applies in many sciences and drives research\n",
    "capable of beating human intelligence in some cases\n",
    "Many research is done in that field\n",
    "Gives too much benefit to business\n",
    "high income\n",
    "thousands of jobs\n",
    "many opportunities\n",
    "huge outsourcing staff\n",
    "creates modern and most powerful technologies\n",
    "makes our daily life easy\n",
    "creates intelligent agents\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Quotes of \"AI Fathers\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/andrew_ng.jpg\" style=\"height: 500px;\">\n",
    "\n",
    "- [Andrew NG](https://www.brainyquote.com/authors/andrew-ng-quotes): We're making this analogy that AI is the new electricity. Electricity transformed industries: agriculture, transportation, communication, manufacturing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/yan_lecun.jpg\" style=\"height: 500px;\">\n",
    "\n",
    "- [Yan Lecun](https://www.forbes.com/sites/bernardmarr/2017/09/22/12-ai-quotes-everyone-should-read/#5a9c661b58a9): Unsupervised learning –  teaching machines to learn for themselves without having to be explicitly told if everything they do is right or wrong – is the key to “true” AI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/yoshua_bengio.jpeg\" style=\"width: 500px;\">\n",
    "\n",
    "- [Yoshua Bengio](https://medium.com/@argodesign/7-quotes-that-will-shape-the-way-you-think-about-artificial-intelligence-ac143be6b42b): “I do really believe that creativity is computational…It is something we understand the principles behind. So, it’s only a matter of having…neural nets or models that are smarter. That understand the world better...\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/geoffrey_hinton.jpg\" style=\"height: 500px;\">\n",
    "\n",
    "- [Geoffrey Hinton](https://www.brainyquote.com/authors/geoffrey-hinton-quotes): \"Early AI was mainly based on logic. You're trying to make computers that reason like people. The second route is from biology: You're trying to make computers that can perceive and act and adapt like animals.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "### Worth to see:\n",
    "- [Architects of AI](http://www.softouch.on.ca/kb/data/Architects%20of%20Intelligence.pdf)\n",
    "- [AI_software_and_frameworks_2019](https://www.ubuntupit.com/best-ai-and-machine-learning-software-and-frameworks/)\n",
    "- [7_quotes_on_AI](https://medium.com/@argodesign/7-quotes-that-will-shape-the-way-you-think-about-artificial-intelligence-ac143be6b42b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Relations to other fields"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/cs_stats_business_venn_diagram.png\" style=\"height: 700px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Relation to data mining"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Machine Learning**\n",
    "- Focuses on prediction\n",
    "- Uses knowledge about the data\n",
    "- Uses extracted properties of the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Data Mining**\n",
    "- Focuses on discovery\n",
    "- Discovers unknown properties in data\n",
    "- Uses Machine Learning methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Machine learning and data mining often employ the same methods and overlap significantly, but while machine learning focuses on prediction, based on known properties learned from the training data, data mining focuses on the discovery of (previously) unknown properties in the data (this is the analysis step of knowledge discovery in databases). Data mining uses many machine learning methods, but with different goals; on the other hand, machine learning also employs data mining methods as \"unsupervised learning\" or as a preprocessing step to improve learner accuracy. Much of the confusion between these two research communities (which do often have separate conferences and separate journals, ECML PKDD being a major exception) comes from the basic assumptions they work with: in machine learning, performance is usually evaluated with respect to the ability to reproduce known knowledge, while in knowledge discovery and data mining (KDD) the key task is the discovery of previously unknown knowledge. Evaluated with respect to known knowledge, an uninformed (unsupervised) method will easily be outperformed by other supervised methods, while in a typical KDD task, supervised methods cannot be used due to the unavailability of training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Relation to optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Machine Learning**\n",
    "- Has optimization objective\n",
    "- Minimizes discrepancy between predicted and pre-assigned labels \n",
    "- Tries to generalize (Minimizes loss on unseen examples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Optimization**\n",
    "- Has optimization objective\n",
    "- Minimizes discrepancy between predicted and pre-assigned labels \n",
    "- Minimizes loss on training set only"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Machine learning also has intimate ties to optimization: many learning problems are formulated as minimization of some loss function on a training set of examples. Loss functions express the discrepancy between the predictions of the model being trained and the actual problem instances (for example, in classification, one wants to assign a label to instances, and models are trained to correctly predict the pre-assigned labels of a set of examples). The difference between the two fields arises from the goal of generalization: while optimization algorithms can minimize the loss on a training set, machine learning is concerned with minimizing the loss on unseen samples.[15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Relation to statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Statistics**\n",
    "- Draws population inferences from a sample\n",
    "- Uses machine learning as \"algorithmic model\" (e.g. Random Forest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Machine Learning**\n",
    "- Finds generalizable predictive patterns.\n",
    "- Is continuation of statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Machine learning and statistics are closely related fields in terms of methods, but distinct in their principal goal: statistics draws population inferences from a sample, while machine learning finds generalizable predictive patterns. According to Michael I. Jordan, the ideas of machine learning, from methodological principles to theoretical tools, have had a long pre-history in statistics. He also suggested the term data science as a placeholder to call the overall field.\n",
    "\n",
    "Leo Breiman distinguished two statistical modeling paradigms: data model and algorithmic model, wherein \"algorithmic model\" means more or less the machine learning algorithms like Random forest.\n",
    "\n",
    "Some statisticians have adopted methods from machine learning, leading to a combined field that they call statistical learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## History"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/arthur_samuel.jpg\" style=\"width:1000;\">\n",
    "\n",
    "The name \"Machine Learning\" was coined in 1959 by [Arthur Samuel](https://en.wikipedia.org/wiki/Arthur_Samuel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/tom_mitchel.jpg\" style=\"height: 500px;\">\n",
    "\n",
    "[Tom M. Mitchell](https://en.wikipedia.org/wiki/Tom_M._Mitchell): \"A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Main types of Algorithms of Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Supervised Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Intuitive explanation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/supervised_learning.png\" style=\"height: 700px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Supervised Learning Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- classification\n",
    "    - spam detection\n",
    "    - classifying images\n",
    "    - document classification\n",
    "    - medical diagnosis\n",
    "- regression\n",
    "    - predicting prices\n",
    "    - vehicle details life cycle prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Theoretical Background"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Given set of N training examples: $\\{(x_1, y_1), ..., (x_N,\\; y_N)\\}$\n",
    "\n",
    "$x_i$ is the **feature** vector of the i-th example and $y_i$ is its **label**, **target** or **class**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Need to find a function $g: X \\to Y$, such that outputs of that function are very close to input labels ($y_i$'s)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$X$ is the input space.\n",
    "\n",
    "$Y$ is the output space.\n",
    "\n",
    "The function $g$ is an element of some space of possible functions $G$ (called the **\"hypothesis space\"**)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Unsupervised Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Intuitive explanation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/unsupervised_learning.png\" style=\"height: 700px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Unsupervised Learning Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- dimensionality reduction\n",
    "    - data compression\n",
    "    - visualization\n",
    "    - topic modeling\n",
    "- clustering\n",
    "    - recommender systems\n",
    "    - social network analysis\n",
    "    - targeted marketing\n",
    "    - big data analysis\n",
    "    - detecting outliers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Theoretical Background"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Given set of N training examples: $\\{x_1, ..., x_N\\}$\n",
    "\n",
    "$x_i$ is the **feature** vector of the i-th example (there are no labels here)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "There are undefined number of problems that can be solved using self-supervised learning algorithm, in general. One example can be clustering where we need to define classes for each input sample, hence we are going to model some function $g: X \\to N$, such that outputs of that function gives class id (classes are enumerated with undefined order)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$X$ is the input space.\n",
    "\n",
    "$N$ is the output space (Just integers of class IDs).\n",
    "\n",
    "**We don't have strong mathematical definition of Self-Supervised (Unsupervised) Learning!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Reinforcement Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Intuitive explanation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/reinforcement_learning.png\" style=\"height: 700px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Reinforcement Learning Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Game AI\n",
    "    - playing games (OpenAI)\n",
    "- Robot Control\n",
    "    - robot navigation\n",
    "    - self-driving cars\n",
    "- Real Time Decisions\n",
    "    - recommendation systems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/machine_learning_algorithms_diagram.png\" style=\"height: 600px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Yan Lecun's cake example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/yan_lecun_cake_example.png\" style=\"height: 700px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "**\"Most of human and animal learning is unsupervised learning. If intelligence was a cake, unsupervised learning would be the cake, supervised learning would be the icing on the cake, and reinforcement learning would be the cherry on the cake. We know how to make the icing and the cherry, but we don't know how to make the cake. We need to solve the unsupervised learning problem before we can even think of getting to true AI.\"**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Machine Learning in Industry and Research"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[AI changes our lives](https://towardsdatascience.com/how-artificial-intelligence-is-impacting-our-everyday-lives-eae3b63379e1)\n",
    "\n",
    "[Investments in AI (2018)](https://www.forbes.com/sites/louiscolumbus/2018/01/12/10-charts-that-will-change-your-perspective-on-artificial-intelligences-growth/#149c1d4f4758)\n",
    "\n",
    "[AI changes 9 industries](https://www.businessinsider.com/sc/artificial-intelligence-companies?IR=T)\n",
    "\n",
    "([IBM](https://www.research.ibm.com/artificial-intelligence/), [Intel](https://www.intel.ai/research/), [Google](https://ai.google/research/), [Microsoft](https://www.microsoft.com/en-us/research/lab/microsoft-research-ai/))\n",
    "\n",
    "[AI statistics](https://blog.zoominfo.com/statistics-about-artificial-intelligence/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Regression vs Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Two interesting types of problems of supervised learning for continuous&discrete target prediction "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## What is Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In **statistical modeling**, regression analysis is a set of statistical processes for estimating the relationships between a **dependent variable** (often called the 'outcome variable') and one or more **independent** variables (often called 'predictors', 'covariates', or 'features').\n",
    "\n",
    "In **Machine Learning** we are estimating relationship between **target** and **features**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## What is Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In **machine learning** and **statistics**, classification is the problem of identifying to which of a set of categories (sub-populations) a new observation belongs, on the basis of a training set of data containing observations (or instances) whose category membership is known.\n",
    "\n",
    "So, we are predicting **target class** from given input **features**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Regression vs Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Regression**\n",
    "- **target** is continuous"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Classification**\n",
    "- **target** is discrete"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Linear regression and loss function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In **statistics**, **linear regression** is a linear approach to modeling the relationship between a scalar response and explanatory variables . "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The case of one explanatory variable is called **simple linear regression**. \n",
    "For more than one explanatory variable, the process is called **multiple linear regression**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In **linear regression**, the relationships are modeled using **linear predictor functions** whose unknown model parameters are estimated from the data. Such models are called **linear models**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Like all forms of regression analysis, linear regression focuses on the **conditional probability distribution** of the response given the values of the predictors, rather than on the joint probability distribution of all of these variables, which is the domain of **multivariate analysis**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## OLS (Ordinary Least Squares)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Consider an overdetermined system (more equations than unknowns)\n",
    "\n",
    "$$\\sum_{j=1}^{p} X_{ij}\\beta_j = y_i,\\ (i=1, 2, \\dots, n),$$\n",
    "\n",
    "with \"p\" features (one is bias term) and \"n\" linear equations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In vectorized form:\n",
    "$$\\mathbf {X} \\boldsymbol {\\beta} = \\mathbf {y},$$\n",
    "\n",
    "where\n",
    "\n",
    "$$\\mathbf {X}=\\begin{bmatrix}\n",
    "X_{11} & X_{12} & \\cdots & X_{1p} \\\\\n",
    "X_{21} & X_{22} & \\cdots & X_{2p} \\\\\n",
    "\\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
    "X_{n1} & X_{n2} & \\cdots & X_{np}\n",
    "\\end{bmatrix} ,\n",
    "\\qquad \\boldsymbol \\beta = \\begin{bmatrix}\n",
    "\\beta_1 \\\\ \\beta_2 \\\\ \\vdots \\\\ \\beta_p \\end{bmatrix} ,\n",
    "\\qquad \\mathbf y = \\begin{bmatrix}\n",
    "y_1 \\\\ y_2 \\\\ \\vdots \\\\ y_n\n",
    "\\end{bmatrix}. $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Such a system usually has no exact solution, so the goal is instead to find the coefficients $\\boldsymbol{\\beta}$ which fit the equations \"best\", in the sense of solving the quadratic minimization problem: \n",
    "\n",
    "$$\\hat{\\boldsymbol{\\beta}} = \\underset{\\boldsymbol{\\beta}}{\\operatorname{arg\\,min}}\\,S(\\boldsymbol{\\beta}), $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The **OLS** estimator is identical to the maximum likelihood estimator (MLE) under the normality assumption for the error terms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "A justification for choosing this criterion is given in [Properties](https://en.wikipedia.org/wiki/Ordinary_least_squares#Properties) below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Normal Equation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**OLS** problem has a unique solution, provided that the \"p\" columns of the matrix $\\mathbf X$ are linearly independent, given by solving the normal equations\n",
    "\n",
    "$$(\\mathbf X^{\\rm T} \\mathbf X )\\hat{\\boldsymbol{\\beta}}= \\mathbf X^{\\rm T} \\mathbf y.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$\\hat{\\boldsymbol\\beta} = (\\mathbf{X}^\\mathsf{T}\\mathbf{X})^{-1} \\mathbf{X}^\\mathsf{T} \\mathbf{y}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Loss Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In **mathematical optimization** and **decision theory**, a loss function or cost function is a function that maps an event or values of one or more variables onto a real number intuitively representing some \"cost\" associated with the event.\n",
    "\n",
    "An optimization problem seeks to minimize a loss function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "objective function (loss) \"S\" of **OLS** is given by:\n",
    "\n",
    "$$S(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\bigl| y_i - \\sum_{j=1}^p X_{ij}\\beta_j\\bigr|^2 = \\bigl\\|\\mathbf y - \\mathbf X \\boldsymbol \\beta \\bigr\\|^2.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Gradient descent** is a first-order iterative optimization algorithm for finding the minimum of a function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "To find a **local minimum** of a function using **gradient descent**, one takes steps proportional to the negative of the gradient (or approximate gradient) of the function at the current point."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/gradient_descent.png\" style=\"height: 700px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Batch, Mini-Batch and Stochastic Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/different_batch_gradient_descent.png\" style=\"height: 700px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Why not using normal equation?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- OLS problem can't be always solved"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Takes $O(n^3)$ time to calculate inverse of $X$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Simple Linear Regression Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/simple_linear_regression.png\" style=\"height: 700px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Multi Dimensional Linear Regression Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/multivariable_linear_regression.png\" style=\"height: 800px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Training linear regression using NumPy and Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## What is NumPy?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Numpy** - package for vector and matrix manipulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "How do \"those guys\" make things run faster?\n",
    "- Read on AVX instruction set (SIMD) and structure of x86 and RISC\n",
    "- Read on OpenMP and CUDA for multiprocessing\n",
    "- Read on assembly-level optimization, memory stride, caching, etc.\n",
    "- Or even about memory management, virtualization\n",
    "- More bare metal FPGA, TPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## What is Pandas?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "What is pandas?\n",
    "\n",
    "- It's an open source data analysis library for providing easy-to-use data structures and data analysis tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Benefits of pandas:\n",
    "- A lot of functionality\n",
    "- Active community\n",
    "- Extensive documentation\n",
    "- Built on top of NumPy\n",
    "- Works well with scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Training Loop (pseudocode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**D** - data,\n",
    "**L** - loss function,\n",
    "**J** - cost function, \n",
    "**E** - number of epochs (full runs on data)\n",
    "\n",
    "Initialize $w,b$\n",
    "\n",
    "for epoch e∈(1,...,E):\n",
    "-  shuffle $D$ to prevent cycles\n",
    "-  for every $(x_i,y_i)∈D$:\n",
    "    - compute prediction $\\hat{y_i}=h(x_i)$\n",
    "-  compute loss $L=\\dfrac{1}{n}\\sum_{n=1}^{n}J(\\hat{y_i},y_i)$\n",
    "-  compute gradients $\\Delta{w}=−\\triangledown_{L^i} w, \\Delta{b}=−\\dfrac{\\partial{L}}{\\partial{b}}$\n",
    "-  update parameters $w=w+\\Delta{w},b=+\\Delta{b}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Logistic regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Why Logistic?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Uses Logistic function\n",
    "- Used for binary targets\n",
    "- Predicts probabilities instead of some quantities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Logistic Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/logistic_function.png\" style=\"weight: 800px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "A \"logistic function\" or \"logistic curve\" is a common \"S\" shape (sigmoid curve), with equation:\n",
    "\n",
    "$$f(x) = \\frac{L}{1 + e^{-k(x-x_0)}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- $e$ = the natural logarithm base (also known as Euler's number),\n",
    "- $x_0$ = the $x$-value of the sigmoid's midpoint,\n",
    "- $L$ = the curve's maximum value, and\n",
    "- $k$ = the logistic growth rate or steepness of the curve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Logistic function that we use in Machine Learning: \n",
    "$$f(x) = \\frac{1}{1 + e^{-(x)}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Derivative of logistic function is easily calculated:\n",
    "$$\\frac{d}{dx}f(x) = f(x)(1-f(x))$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Linear Regression  vs Logistic Regression Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/logistic vs linear model predictions.png\" style=\"height: 700px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Cross Entropy Loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Sigmoid: $g(z) = \\frac{1}{1 + e^{-z}} $\n",
    "\n",
    "Hypothesis: $h_\\theta(x) = \\frac{1}{1 + e^{-\\theta^T x}} $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$Cost(h_\\theta(x), y) = -log(h_\\theta(x))$ if y=1\n",
    "\n",
    "$Cost(h_\\theta(x), y) = -log(1- h_\\theta(x))$ if y=0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/cross_entropy_loss.png\" style=\"height: 600px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Full Cost Function:\n",
    "$$Cost(h_\\theta(x), y) = -{y}log(h_\\theta(x)) - {(1-y)}log(1 - h_\\theta(x))$$\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Cross Entropy Loss:\n",
    "$$J(\\theta) = -\\dfrac{1}{m}\\sum_{1}^{m}[{y^{(i)}}log(h_\\theta(x^{(i)})) + {(1-y^{(i)})}log(1 - h_\\theta(x^{(i)}))]$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Data organization and hyperparameters optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Data Organization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Crucial for Learning\n",
    "- Should Match learning algorithm\n",
    "- Data Specific (sequence,  time-series)\n",
    "- Enables performance measuring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Train/Validation/Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/train_valid_test_splits.png\" style=\"height: 300px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Equally Distributed\n",
    "- Train/Valid/Test: 70%/15%/15% on small datasets\n",
    "- Train/Valid/Test: 90%/5%/5% on huge datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Kaggle Data Splitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/kaggle_data_splitting.jpg\" style=\"height: 700px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### k-fold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/k-fold.png\" style=\"height: 600px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Why we need such data splitting?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Train**: Fit model on data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Valid**: Tune hyperparameters, choose best model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Test**: Score model and estimate generalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Hyperparameters Optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### What is Hyperparameter?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In **machine learning**, a **hyperparameter** is a parameter whose value is set before the learning process begins."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "By contrast, the values of **other parameters** are derived via training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Model Hyperparameters**: cannot be inferred while fitting the machine to the training set (Size of Model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- **Algorithm Hyperparameters**: have no influence on the performance of the model but affect the speed and quality of the learning process (Batch size, Learning Rate, etc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Hyperparameters Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/hyperparameter_tuning.png\" style=\"height: 700px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Search in Hyperparameters Space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Here Important and Unimportant parameters are explained by their improvements on score. We have big increase in scores for important parameter (shown on top of grid) and little increase on unimportant parameter (shown on the left of grid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/grid_and_random_search.png\" style=\"width: 1400px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Scoring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Detailed Information About Scoring Different types of models can be found [here](https://scikit-learn.org/stable/modules/model_evaluation.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Confusion Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/confusion_matrix.jpeg\" style=\"height: 700px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Confusion matrix value name consists of two words:\n",
    "- **prediction is correct?**: the answer is True or False\n",
    "- **what was predicted?**: the answer can be True/False or 1/0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Example:\n",
    "- **True Positive**: It means that our prediction was correct and we predicted Positive class or 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$Accuracy = \\dfrac{TP+TN}{TP+TN+FP+FN}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Precision&Recall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$Precision=\\dfrac{TP}{TP+FP}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$$Recall=\\dfrac{TP}{TP+FN}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/precision_and_recall.jpg\" style=\"height: 700px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### All Other Available Terms of Measures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<tbody><tr>\n",
    "<td style=\"border:none;\" colspan=\"2\">\n",
    "</td>\n",
    "<td style=\"background:#eeeebb;\" colspan=\"2\"><b>True condition</b>\n",
    "</td></tr>\n",
    "<tr>\n",
    "<td style=\"border:none;\">\n",
    "</td>\n",
    "<td style=\"background:#dddddd;\"><a href=\"/wiki/Statistical_population\" title=\"Statistical population\">Total population</a>\n",
    "</td>\n",
    "<td style=\"background:#ffffcc;\">Condition positive\n",
    "</td>\n",
    "<td style=\"background:#ddddaa;\">Condition negative\n",
    "</td>\n",
    "<td style=\"background:#eeeecc;font-size:90%;\"><a href=\"/wiki/Prevalence\" title=\"Prevalence\">Prevalence</a> <span style=\"font-size:118%;white-space:nowrap;\">= <span role=\"math\" class=\"sfrac nowrap tion\" style=\"display:inline-block; vertical-align:-0.5em; font-size:85%; text-align:center;\"><span class=\"num\" style=\"display:block; line-height:1em; margin:0 0.1em;\">Σ&nbsp;Condition positive</span><span class=\"slash visualhide\">/</span><span class=\"den\" style=\"display:block; line-height:1em; margin:0 0.1em; border-top:1px solid;\">Σ&nbsp;Total population</span></span></span>\n",
    "</td>\n",
    "<td style=\"background:#cceecc;border-left:double silver;font-size:90%;\" colspan=\"2\"><a href=\"/wiki/Accuracy_and_precision\" title=\"Accuracy and precision\">Accuracy</a> (ACC) = <span style=\"font-size:118%;\"><span role=\"math\" class=\"sfrac nowrap tion\" style=\"display:inline-block; vertical-align:-0.5em; font-size:85%; text-align:center;\"><span class=\"num\" style=\"display:block; line-height:1em; margin:0 0.1em;\">Σ&nbsp;True positive + Σ True negative</span><span class=\"slash visualhide\">/</span><span class=\"den\" style=\"display:block; line-height:1em; margin:0 0.1em; border-top:1px solid;\">Σ&nbsp;Total population</span></span></span>\n",
    "</td></tr>\n",
    "<tr>\n",
    "<td style=\"background:#bbeeee;\" rowspan=\"2\"><b>Predicted<br>condition</b>\n",
    "</td>\n",
    "<td style=\"background:#ccffff;\">Predicted condition<br>positive\n",
    "</td>\n",
    "<td style=\"background:#ccffcc;\"><span style=\"color:#006600;\"><b><a href=\"/wiki/True_positive\" class=\"mw-redirect\" title=\"True positive\">True positive</a></b></span>\n",
    "</td>\n",
    "<td style=\"background:#eedddd;\"><span style=\"color:#cc0000;\"><b><a href=\"/wiki/False_positive\" class=\"mw-redirect\" title=\"False positive\">False positive</a></b>,<br><a href=\"/wiki/Type_I_error\" class=\"mw-redirect\" title=\"Type I error\">Type I error</a></span>\n",
    "</td>\n",
    "<td style=\"background:#ccffee;border-top:double silver;font-size:90%;\"><a href=\"/wiki/Positive_predictive_value\" class=\"mw-redirect\" title=\"Positive predictive value\">Positive predictive value</a> (PPV), <a href=\"/wiki/Precision_(information_retrieval)\" class=\"mw-redirect\" title=\"Precision (information retrieval)\">Precision</a> = <span style=\"font-size:118%;white-space:nowrap;\"><span role=\"math\" class=\"sfrac nowrap tion\" style=\"display:inline-block; vertical-align:-0.5em; font-size:85%; text-align:center;\"><span class=\"num\" style=\"display:block; line-height:1em; margin:0 0.1em;\">Σ True positive</span><span class=\"slash visualhide\">/</span><span class=\"den\" style=\"display:block; line-height:1em; margin:0 0.1em; border-top:1px solid;\">Σ&nbsp;Predicted&nbsp;condition&nbsp;positive</span></span></span>\n",
    "</td>\n",
    "<td style=\"background:#cceeff;border-top:double silver;font-size:90%;\" colspan=\"2\"><a href=\"/wiki/False_discovery_rate\" title=\"False discovery rate\">False discovery rate</a> (FDR) = <span style=\"font-size:118%;white-space:nowrap;\"><span role=\"math\" class=\"sfrac nowrap tion\" style=\"display:inline-block; vertical-align:-0.5em; font-size:85%; text-align:center;\"><span class=\"num\" style=\"display:block; line-height:1em; margin:0 0.1em;\">Σ False positive</span><span class=\"slash visualhide\">/</span><span class=\"den\" style=\"display:block; line-height:1em; margin:0 0.1em; border-top:1px solid;\">Σ&nbsp;Predicted&nbsp;condition&nbsp;positive</span></span></span>\n",
    "</td></tr>\n",
    "<tr>\n",
    "<td style=\"background:#aadddd;\">Predicted condition<br>negative\n",
    "</td>\n",
    "<td style=\"background:#ffdddd;\"><span style=\"color:#cc0000;\"><b><a href=\"/wiki/False_negative\" class=\"mw-redirect\" title=\"False negative\">False negative</a></b>,<br><a href=\"/wiki/Type_II_error\" class=\"mw-redirect\" title=\"Type II error\">Type II error</a></span>\n",
    "</td>\n",
    "<td style=\"background:#bbeebb;\"><span style=\"color:#006600;\"><b><a href=\"/wiki/True_negative\" class=\"mw-redirect\" title=\"True negative\">True negative</a></b></span>\n",
    "</td>\n",
    "<td style=\"background:#eeddee;border-bottom:double silver;font-size:90%;\"><a href=\"/wiki/False_omission_rate\" class=\"mw-redirect\" title=\"False omission rate\">False omission rate</a> (FOR) = <span style=\"font-size:118%;white-space:nowrap;\"><span role=\"math\" class=\"sfrac nowrap tion\" style=\"display:inline-block; vertical-align:-0.5em; font-size:85%; text-align:center;\"><span class=\"num\" style=\"display:block; line-height:1em; margin:0 0.1em;\">Σ False negative</span><span class=\"slash visualhide\">/</span><span class=\"den\" style=\"display:block; line-height:1em; margin:0 0.1em; border-top:1px solid;\">Σ&nbsp;Predicted&nbsp;condition&nbsp;negative</span></span></span>\n",
    "</td>\n",
    "<td style=\"background:#aaddcc;border-bottom:double silver;font-size:90%;\" colspan=\"2\"><a href=\"/wiki/Negative_predictive_value\" class=\"mw-redirect\" title=\"Negative predictive value\">Negative predictive value</a> (NPV) = <span style=\"font-size:118%;white-space:nowrap;\"><span role=\"math\" class=\"sfrac nowrap tion\" style=\"display:inline-block; vertical-align:-0.5em; font-size:85%; text-align:center;\"><span class=\"num\" style=\"display:block; line-height:1em; margin:0 0.1em;\">Σ True negative</span><span class=\"slash visualhide\">/</span><span class=\"den\" style=\"display:block; line-height:1em; margin:0 0.1em; border-top:1px solid;\">Σ&nbsp;Predicted&nbsp;condition&nbsp;negative</span></span></span>\n",
    "</td></tr>\n",
    "<tr style=\"font-size:90%;\">\n",
    "<td style=\"border:none;vertical-align:bottom;padding:0 2px 0 0;color:#999999;\" colspan=\"2\" rowspan=\"2\">\n",
    "</td>\n",
    "<td style=\"background:#eeffcc;\"><a href=\"/wiki/True_positive_rate\" class=\"mw-redirect\" title=\"True positive rate\">True positive rate</a> (TPR), <a href=\"/wiki/Recall_(information_retrieval)\" class=\"mw-redirect\" title=\"Recall (information retrieval)\">Recall</a>, <a href=\"/wiki/Sensitivity_(tests)\" class=\"mw-redirect\" title=\"Sensitivity (tests)\">Sensitivity</a>, probability&nbsp;of&nbsp;detection, <a href=\"/wiki/Statistical_power\" class=\"mw-redirect\" title=\"Statistical power\">Power</a> <span style=\"font-size:118%;white-space:nowrap;\">= <span role=\"math\" class=\"sfrac nowrap tion\" style=\"display:inline-block; vertical-align:-0.5em; font-size:85%; text-align:center;\"><span class=\"num\" style=\"display:block; line-height:1em; margin:0 0.1em;\">Σ True positive</span><span class=\"slash visualhide\">/</span><span class=\"den\" style=\"display:block; line-height:1em; margin:0 0.1em; border-top:1px solid;\">Σ&nbsp;Condition&nbsp;positive</span></span></span>\n",
    "</td>\n",
    "<td style=\"background:#eeddbb;\"><a href=\"/wiki/False_positive_rate\" title=\"False positive rate\">False positive rate</a> (FPR), <a href=\"/wiki/Information_retrieval\" title=\"Information retrieval\"><span class=\"nowrap\">Fall-out</span></a>, probability&nbsp;of&nbsp;false&nbsp;alarm <span style=\"font-size:118%;white-space:nowrap;\">= <span role=\"math\" class=\"sfrac nowrap tion\" style=\"display:inline-block; vertical-align:-0.5em; font-size:85%; text-align:center;\"><span class=\"num\" style=\"display:block; line-height:1em; margin:0 0.1em;\">Σ False positive</span><span class=\"slash visualhide\">/</span><span class=\"den\" style=\"display:block; line-height:1em; margin:0 0.1em; border-top:1px solid;\">Σ&nbsp;Condition&nbsp;negative</span></span></span>\n",
    "</td>\n",
    "<td style=\"background:#eeeeee;\"><a href=\"/wiki/Positive_likelihood_ratio\" class=\"mw-redirect\" title=\"Positive likelihood ratio\">Positive likelihood ratio</a> <span class=\"nowrap\">(LR+)</span> <span style=\"font-size:118%;white-space:nowrap;\">= <span role=\"math\" class=\"sfrac nowrap tion\" style=\"display:inline-block; vertical-align:-0.5em; font-size:85%; text-align:center;\"><span class=\"num\" style=\"display:block; line-height:1em; margin:0 0.1em;\">TPR</span><span class=\"slash visualhide\">/</span><span class=\"den\" style=\"display:block; line-height:1em; margin:0 0.1em; border-top:1px solid;\">FPR</span></span></span>\n",
    "</td>\n",
    "<td style=\"background:#dddddd;\" rowspan=\"2\"><a href=\"/wiki/Diagnostic_odds_ratio\" title=\"Diagnostic odds ratio\">Diagnostic odds ratio</a> (DOR) <span style=\"font-size:118%;white-space:nowrap;\">= <span role=\"math\" class=\"sfrac nowrap tion\" style=\"display:inline-block; vertical-align:-0.5em; font-size:85%; text-align:center;\"><span class=\"num\" style=\"display:block; line-height:1em; margin:0 0.1em;\">LR+</span><span class=\"slash visualhide\">/</span><span class=\"den\" style=\"display:block; line-height:1em; margin:0 0.1em; border-top:1px solid;\">LR−</span></span></span>\n",
    "</td>\n",
    "<td style=\"background:#ddffdd;border-left:double silver;line-height:2;\" rowspan=\"2\"><a href=\"/wiki/F1_score\" title=\"F1 score\">F<sub>1</sub> score</a> = <span style=\"font-size:118%;white-space:nowrap;\">2 · <span role=\"math\" class=\"sfrac nowrap tion\" style=\"display:inline-block; vertical-align:-0.5em; font-size:85%; text-align:center;\"><span class=\"num\" style=\"display:block; line-height:1em; margin:0 0.1em;\">Precision · Recall</span><span class=\"slash visualhide\">/</span><span class=\"den\" style=\"display:block; line-height:1em; margin:0 0.1em; border-top:1px solid;\">Precision + Recall</span></span></span>\n",
    "</td></tr>\n",
    "<tr style=\"font-size:90%;\">\n",
    "<td style=\"background:#ffeecc;\"><a href=\"/wiki/False_negative_rate\" class=\"mw-redirect\" title=\"False negative rate\">False negative rate</a> (FNR), Miss&nbsp;rate <span style=\"font-size:118%;white-space:nowrap;\">= <span role=\"math\" class=\"sfrac nowrap tion\" style=\"display:inline-block; vertical-align:-0.5em; font-size:85%; text-align:center;\"><span class=\"num\" style=\"display:block; line-height:1em; margin:0 0.1em;\">Σ False negative</span><span class=\"slash visualhide\">/</span><span class=\"den\" style=\"display:block; line-height:1em; margin:0 0.1em; border-top:1px solid;\">Σ&nbsp;Condition&nbsp;positive</span></span></span>\n",
    "</td>\n",
    "<td style=\"background:#ddeebb;\"><a href=\"/wiki/Specificity_(tests)\" class=\"mw-redirect\" title=\"Specificity (tests)\">Specificity</a> (SPC), Selectivity, <a href=\"/wiki/True_negative_rate\" class=\"mw-redirect\" title=\"True negative rate\">True negative rate</a> (TNR) <span style=\"font-size:118%;white-space:nowrap;\">= <span role=\"math\" class=\"sfrac nowrap tion\" style=\"display:inline-block; vertical-align:-0.5em; font-size:85%; text-align:center;\"><span class=\"num\" style=\"display:block; line-height:1em; margin:0 0.1em;\">Σ True negative</span><span class=\"slash visualhide\">/</span><span class=\"den\" style=\"display:block; line-height:1em; margin:0 0.1em; border-top:1px solid;\">Σ&nbsp;Condition&nbsp;negative</span></span></span>\n",
    "</td>\n",
    "<td style=\"background:#cccccc;\"><a href=\"/wiki/Negative_likelihood_ratio\" class=\"mw-redirect\" title=\"Negative likelihood ratio\">Negative likelihood ratio</a> <span class=\"nowrap\">(LR−)</span> <span style=\"font-size:118%;white-space:nowrap;\">= <span role=\"math\" class=\"sfrac nowrap tion\" style=\"display:inline-block; vertical-align:-0.5em; font-size:85%; text-align:center;\"><span class=\"num\" style=\"display:block; line-height:1em; margin:0 0.1em;\">FNR</span><span class=\"slash visualhide\">/</span><span class=\"den\" style=\"display:block; line-height:1em; margin:0 0.1em; border-top:1px solid;\">TNR</span></span></span>\n",
    "</td></tr></tbody>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Comparing Models using Precision&Recall"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We need to have one measure instead of two to be able to compare several models by scores.\n",
    "\n",
    "Here are some possible combinations of Precision and Recall to get combined score:\n",
    "\n",
    "$$F1(harmonic) = \\dfrac{2*Precision*Recall}{Precision+Recall}$$\n",
    "\n",
    "$$geometric = \\sqrt{Precision*Recall}$$\n",
    "\n",
    "$$arithmetic = \\dfrac{Precision+Recall}{2}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/f_score_vs_others.png\" style=\"height: 700px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "From some empirical tests, explorations and experience we can say that F1 score better describes the overall performance of model.\n",
    "Maybe we choose other measure depending on our problem and requirements but F1 score is said to be best in most scenarios."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "F1 Score is harmonic mean of Precision and Recall:\n",
    "$$F1=\\dfrac{2}{\\dfrac{1}{Precision}+\\dfrac{1}{Recall}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Well known form:\n",
    "$$F1=\\dfrac{2*Precision*Recall}{Precision + Recall}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Choose Optimal Threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/decision_threshold.png\" style=\"width: 1400px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### ROC-AUC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Short [video](https://www.youtube.com/watch?v=4jRBRDbJemM) explaining ROC-AUC "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"images/ml_intro/roc_auc.png\" style=\"height: 700px;\" class=\"center\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- MSE (Mean Squared Error): Measures average squared difference between predictions and targets\n",
    "\n",
    "- Explained Variance Score: How well our model explains target variable variance using given data (features) "
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "367px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}