notebooks/3.2 Exploratory data analysis and working with texts.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.2 Exploratory data analysis and working with texts\n",
    "\n",
    "In this notebook, we learn about:\n",
    "1. descriptive statistics to explore data;\n",
    "2. working with texts (hints)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: descriptive statistics\n",
    "\n",
    "*The goal of exploratory data analysis is to develop an understanding of your data. EDA is fundamentally a creative process. And like most creative processes, the key to asking quality questions is to generate a large quantity of questions.* \n",
    "\n",
    "Key questions:\n",
    "* Which kind of variation occurs within variables?\n",
    "* Which kind of co-variation occurs between variables?\n",
    "\n",
    "https://r4ds.had.co.nz/exploratory-data-analysis.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "\n",
    "import os, codecs\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import the dataset\n",
    "Let us import the Venetian apprenticeship contracts dataset in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "root_folder = \"../data/apprenticeship_venice/\"\n",
    "df_contracts = pd.read_csv(codecs.open(os.path.join(root_folder,\"professions_data.csv\"), encoding=\"utf8\"), sep=\";\")\n",
    "df_professions = pd.read_csv(codecs.open(os.path.join(root_folder,\"professions_classification.csv\"), encoding=\"utf8\"), sep=\",\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take another look to the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 9653 entries, 0 to 9652\n",
      "Data columns (total 47 columns):\n",
      " #   Column                    Non-Null Count  Dtype  \n",
      "---  ------                    --------------  -----  \n",
      " 0   page_title                9653 non-null   object \n",
      " 1   register                  9653 non-null   object \n",
      " 2   annual_salary             7870 non-null   float64\n",
      " 3   a_profession              9653 non-null   object \n",
      " 4   profession_code_strict    9618 non-null   object \n",
      " 5   profession_code_gen       9614 non-null   object \n",
      " 6   profession_cat            9597 non-null   object \n",
      " 7   corporation               9350 non-null   object \n",
      " 8   keep_profession_a         9653 non-null   int64  \n",
      " 9   complete_profession_a     9653 non-null   int64  \n",
      " 10  enrolmentY                9628 non-null   float64\n",
      " 11  enrolmentM                9631 non-null   float64\n",
      " 12  startY                    9533 non-null   float64\n",
      " 13  startM                    9539 non-null   float64\n",
      " 14  length                    9645 non-null   float64\n",
      " 15  has_fled                  9653 non-null   int64  \n",
      " 16  m_profession              9535 non-null   object \n",
      " 17  m_profession_code_strict  9508 non-null   object \n",
      " 18  m_profession_code_gen     9506 non-null   object \n",
      " 19  m_profession_cat          9489 non-null   object \n",
      " 20  m_corporation             9276 non-null   object \n",
      " 21  keep_profession_m         9653 non-null   int64  \n",
      " 22  complete_profession_m     9653 non-null   int64  \n",
      " 23  m_gender                  9554 non-null   float64\n",
      " 24  m_name                    9623 non-null   object \n",
      " 25  m_surname                 6960 non-null   object \n",
      " 26  m_patronimic              2620 non-null   object \n",
      " 27  m_atelier                 1434 non-null   object \n",
      " 28  m_coords                  9639 non-null   object \n",
      " 29  a_name                    9653 non-null   object \n",
      " 30  a_age                     9303 non-null   float64\n",
      " 31  a_gender                  9522 non-null   float64\n",
      " 32  a_geo_origins             7149 non-null   object \n",
      " 33  a_geo_origins_std         4636 non-null   object \n",
      " 34  a_coords                  9610 non-null   object \n",
      " 35  a_quondam                 7848 non-null   float64\n",
      " 36  accommodation_master      9653 non-null   int64  \n",
      " 37  personal_care_master      9653 non-null   int64  \n",
      " 38  clothes_master            9653 non-null   int64  \n",
      " 39  generic_expenses_master   9653 non-null   int64  \n",
      " 40  salary_in_kind_master     9653 non-null   int64  \n",
      " 41  pledge_goods_master       9653 non-null   int64  \n",
      " 42  pledge_money_master       9653 non-null   int64  \n",
      " 43  salary_master             9653 non-null   int64  \n",
      " 44  female_guarantor          9653 non-null   int64  \n",
      " 45  period_cat                7891 non-null   float64\n",
      " 46  incremental_salary        9653 non-null   int64  \n",
      "dtypes: float64(11), int64(15), object(21)\n",
      "memory usage: 3.5+ MB\n"
     ]
    }
   ],
   "source": [
    "df_contracts.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>page_title</th>\n",
       "      <th>register</th>\n",
       "      <th>annual_salary</th>\n",
       "      <th>a_profession</th>\n",
       "      <th>profession_code_strict</th>\n",
       "      <th>profession_code_gen</th>\n",
       "      <th>profession_cat</th>\n",
       "      <th>corporation</th>\n",
       "      <th>keep_profession_a</th>\n",
       "      <th>complete_profession_a</th>\n",
       "      <th>...</th>\n",
       "      <th>personal_care_master</th>\n",
       "      <th>clothes_master</th>\n",
       "      <th>generic_expenses_master</th>\n",
       "      <th>salary_in_kind_master</th>\n",
       "      <th>pledge_goods_master</th>\n",
       "      <th>pledge_money_master</th>\n",
       "      <th>salary_master</th>\n",
       "      <th>female_guarantor</th>\n",
       "      <th>period_cat</th>\n",
       "      <th>incremental_salary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Carlo Della sosta (Orese) 1592-08-03</td>\n",
       "      <td>asv, giustizia vecchia, accordi dei garzoni, 1...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>orese</td>\n",
       "      <td>orese</td>\n",
       "      <td>orefice</td>\n",
       "      <td>orefice</td>\n",
       "      <td>Oresi</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Antonio quondam Andrea (squerariol) 1583-01-09</td>\n",
       "      <td>asv, giustizia vecchia, accordi dei garzoni, 1...</td>\n",
       "      <td>12.5</td>\n",
       "      <td>squerariol</td>\n",
       "      <td>squerariol</td>\n",
       "      <td>lavori allo squero</td>\n",
       "      <td>lavori allo squero</td>\n",
       "      <td>Squerarioli</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Cristofollo di Zuane (batioro in carta) 1591-0...</td>\n",
       "      <td>asv, giustizia vecchia, accordi dei garzoni, 1...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>batioro</td>\n",
       "      <td>batioro</td>\n",
       "      <td>battioro</td>\n",
       "      <td>fabbricatore di foglie/fili/cordelle d'oro o a...</td>\n",
       "      <td>Battioro</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 47 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          page_title  \\\n",
       "0               Carlo Della sosta (Orese) 1592-08-03   \n",
       "1     Antonio quondam Andrea (squerariol) 1583-01-09   \n",
       "2  Cristofollo di Zuane (batioro in carta) 1591-0...   \n",
       "\n",
       "                                            register  annual_salary  \\\n",
       "0  asv, giustizia vecchia, accordi dei garzoni, 1...            NaN   \n",
       "1  asv, giustizia vecchia, accordi dei garzoni, 1...           12.5   \n",
       "2  asv, giustizia vecchia, accordi dei garzoni, 1...            NaN   \n",
       "\n",
       "  a_profession profession_code_strict profession_code_gen  \\\n",
       "0        orese                  orese             orefice   \n",
       "1   squerariol             squerariol  lavori allo squero   \n",
       "2      batioro                batioro            battioro   \n",
       "\n",
       "                                      profession_cat  corporation  \\\n",
       "0                                            orefice        Oresi   \n",
       "1                                 lavori allo squero  Squerarioli   \n",
       "2  fabbricatore di foglie/fili/cordelle d'oro o a...     Battioro   \n",
       "\n",
       "   keep_profession_a  complete_profession_a  ...  personal_care_master  \\\n",
       "0                  1                      1  ...                     1   \n",
       "1                  1                      1  ...                     0   \n",
       "2                  1                      1  ...                     0   \n",
       "\n",
       "   clothes_master  generic_expenses_master  salary_in_kind_master  \\\n",
       "0               1                        1                      0   \n",
       "1               0                        1                      0   \n",
       "2               0                        0                      0   \n",
       "\n",
       "   pledge_goods_master  pledge_money_master salary_master female_guarantor  \\\n",
       "0                    0                    0             0                0   \n",
       "1                    0                    0             1                0   \n",
       "2                    0                    0             0                0   \n",
       "\n",
       "  period_cat incremental_salary  \n",
       "0        NaN                  0  \n",
       "1        1.0                  0  \n",
       "2        NaN                  0  \n",
       "\n",
       "[3 rows x 47 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_contracts.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['page_title', 'register', 'annual_salary', 'a_profession',\n",
       "       'profession_code_strict', 'profession_code_gen', 'profession_cat',\n",
       "       'corporation', 'keep_profession_a', 'complete_profession_a',\n",
       "       'enrolmentY', 'enrolmentM', 'startY', 'startM', 'length', 'has_fled',\n",
       "       'm_profession', 'm_profession_code_strict', 'm_profession_code_gen',\n",
       "       'm_profession_cat', 'm_corporation', 'keep_profession_m',\n",
       "       'complete_profession_m', 'm_gender', 'm_name', 'm_surname',\n",
       "       'm_patronimic', 'm_atelier', 'm_coords', 'a_name', 'a_age', 'a_gender',\n",
       "       'a_geo_origins', 'a_geo_origins_std', 'a_coords', 'a_quondam',\n",
       "       'accommodation_master', 'personal_care_master', 'clothes_master',\n",
       "       'generic_expenses_master', 'salary_in_kind_master',\n",
       "       'pledge_goods_master', 'pledge_money_master', 'salary_master',\n",
       "       'female_guarantor', 'period_cat', 'incremental_salary'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_contracts.columns"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Every row represents an apprenticeship contract. Contracts were registered both at the guild's and at a public office. This is a sample of contracts from a much larger set of records.\n",
    "\n",
    "Some of the variables we will work with are:\n",
    "* `annual_salary`: the annual salary paid to the apprencice, if any (in Venetian ducats).\n",
    "* `a_profession` to `corporation`: increasingly generic classifications for the apprentice's stated profession.\n",
    "* `startY` and `enrolmentY`: contract start and registration year respectively.\n",
    "* `length`: of the contract, in years.\n",
    "* `m_gender` and `a_gender`: of master and apprentice respectively.\n",
    "* `a_age`: age of the apprentice at entry, in years.\n",
    "* `female_guarantor`: if at least one of the contract's guarantors was female, boolean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0    9424\n",
       "0.0     130\n",
       "Name: m_gender, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_contracts.m_gender.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Trascrizione</th>\n",
       "      <th>Standard</th>\n",
       "      <th>Gruppo 0</th>\n",
       "      <th>Gruppo 1</th>\n",
       "      <th>Gruppo 2</th>\n",
       "      <th>Gruppo 3</th>\n",
       "      <th>Gruppo 4</th>\n",
       "      <th>Corporazione</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>al negotio del libraro</td>\n",
       "      <td>librer</td>\n",
       "      <td>libraio</td>\n",
       "      <td>librai - diverse specializzazioni</td>\n",
       "      <td>stampa</td>\n",
       "      <td>altre lavorazioni manifatturiere</td>\n",
       "      <td>beni</td>\n",
       "      <td>libreri, stampatori e ligadori</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>arte de far arpicordi</td>\n",
       "      <td>arte de far arpicordi</td>\n",
       "      <td>fabbricatore di arpicordi</td>\n",
       "      <td>fabbricatore di strumenti musicali</td>\n",
       "      <td>musica</td>\n",
       "      <td>altri servizi</td>\n",
       "      <td>servizi</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>arte de' colori</td>\n",
       "      <td>arte dei colori</td>\n",
       "      <td>fabbricazione/vendita di colori</td>\n",
       "      <td>colori</td>\n",
       "      <td>colori</td>\n",
       "      <td>decorazioni e mestieri dell'arte</td>\n",
       "      <td>beni</td>\n",
       "      <td>spezieri</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Trascrizione               Standard  \\\n",
       "0  al negotio del libraro                 librer   \n",
       "1   arte de far arpicordi  arte de far arpicordi   \n",
       "2         arte de' colori        arte dei colori   \n",
       "\n",
       "                          Gruppo 0                            Gruppo 1  \\\n",
       "0                          libraio   librai - diverse specializzazioni   \n",
       "1        fabbricatore di arpicordi  fabbricatore di strumenti musicali   \n",
       "2  fabbricazione/vendita di colori                              colori   \n",
       "\n",
       "  Gruppo 2                          Gruppo 3 Gruppo 4  \\\n",
       "0   stampa  altre lavorazioni manifatturiere     beni   \n",
       "1   musica                     altri servizi  servizi   \n",
       "2   colori  decorazioni e mestieri dell'arte     beni   \n",
       "\n",
       "                     Corporazione  \n",
       "0  libreri, stampatori e ligadori  \n",
       "1                             NaN  \n",
       "2                        spezieri  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_professions.head(3)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The professions data frame contains a classification system for each profession as found in the records (transcription, first column). The last column is the guild (or corporation) which governed the given profession. This work was performed manually by historians. We don't use it here as the classifications we need are already part of the main dataframe."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Questions\n",
    "\n",
    "1. Plot the distribution (histogram) of the apprentices' age, annual salary and start year.\n",
    "2. Calculate the proportion of female apprentices and masters, and of contracts with a female guarantor."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 1: plot distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAU4UlEQVR4nO3df4xdZ53f8feHJPwQXuGEZEep49apcLsKpBgYJVnBH5MgEiesmqxEUVAKDpuVt1UigZq2GKQ2S9hIWWkXWlQ2qnfjYloWbwSksUK2qWsyovwRkhhCEiegeMGUWCbWrp3AQJvK9Ns/7uPdW3c8cz0zd2zP835JV/ec5zz33Oebe/O5x88990yqCklSH151qgcgSVo+hr4kdcTQl6SOGPqS1BFDX5I6cvapHsBczj///Fq3bt2cfX7+85/z+te/fnkGdJrptXbr7ot1n7w9e/b8ZVVdMNu20zr0161bxxNPPDFnn+npaaamppZnQKeZXmu37r5Y98lL8qMTbXN6R5I6YuhLUkfmDf0kr03yWJLvJtmb5JOt/fNJfpjkyXbb0NqT5LNJ9iV5Ksnbh/a1Kcnz7bZpbFVJkmY1ypz+K8BVVTWT5Bzgm0n+vG37F1X15eP6Xwusb7fLgXuAy5OcB9wBTAIF7Emys6qOLEUhkqT5zXukXwMzbfWcdpvrgj3XA19oj3sUWJ3kQuAaYFdVHW5BvwvYuLjhS5JOxkhn7yQ5C9gDvAn4XFV9K8k/Be5K8q+B3cCWqnoFWAP8eOjhL7S2E7Uf/1ybgc0AExMTTE9Pzzm2mZmZefusVL3Wbt19se6lNVLoV9UvgQ1JVgP3J3kL8HHgJ8Crga3Ax4A7Fzugqtra9sfk5GTNd8pSr6dzQb+1W3dfrHtpndTZO1X1EvAIsLGqDrYpnFeA/wBc1rodANYOPeyi1naidknSMhnl7J0L2hE+SV4HvAf4XpunJ0mAG4Bn2kN2Ah9qZ/FcAbxcVQeBh4Grk5yb5Fzg6tYmSVomo0zvXAhsb/P6rwLuq6oHk3w9yQVAgCeBf9L6PwRcB+wDfgF8GKCqDif5FPB463dnVR1eskp0UtZt+dpI/fbf/d4xj0TScpo39KvqKeBts7RfdYL+Bdx6gm3bgG0nOUZJ0hLxF7mS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHZk39JO8NsljSb6bZG+ST7b2i5N8K8m+JH+W5NWt/TVtfV/bvm5oXx9v7d9Pcs3YqpIkzWqUI/1XgKuq6q3ABmBjkiuA3wc+U1VvAo4At7T+twBHWvtnWj+SXALcCLwZ2Aj8UZKzlrAWSdI85g39Gphpq+e0WwFXAV9u7duBG9ry9W2dtv3dSdLad1TVK1X1Q2AfcNlSFCFJGs3Zo3RqR+R7gDcBnwP+Anipqo62Li8Aa9ryGuDHAFV1NMnLwBtb+6NDux1+zPBzbQY2A0xMTDA9PT3n2GZmZubts1ItpvbbLz06fyc4Lf/b9vqaW3dfxlX3SKFfVb8ENiRZDdwP/NqSj+RvnmsrsBVgcnKypqam5uw/PT3NfH1WqsXUfvOWr43Ub/9NC9v/OPX6mlt3X8ZV90mdvVNVLwGPAL8OrE5y7EPjIuBAWz4ArAVo298A/NVw+yyPkSQtg1HO3rmgHeGT5HXAe4DnGIT/+1q3TcADbXlnW6dt/3pVVWu/sZ3dczGwHnhsieqQJI1glOmdC4HtbV7/VcB9VfVgkmeBHUl+D/gOcG/rfy/wH5PsAw4zOGOHqtqb5D7gWeAocGubNpIkLZN5Q7+qngLeNkv7D5jl7Juq+l/APzrBvu4C7jr5YUqSloK/yJWkjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqyLyhn2RtkkeSPJtkb5KPtPbfTXIgyZPtdt3QYz6eZF+S7ye5Zqh9Y2vbl2TLeEqSJJ3I2SP0OQrcXlXfTvIrwJ4ku9q2z1TVHwx3TnIJcCPwZuBvAf8tyd9rmz8HvAd4AXg8yc6qenYpCpEkzW/e0K+qg8DBtvyzJM8Ba+Z4yPXAjqp6Bfhhkn3AZW3bvqr6AUCSHa2voS9JyyRVNXrnZB3wDeAtwD8DbgZ+CjzB4F8DR5L8O+DRqvpP7TH3An/edrGxqn67tX8QuLyqbjvuOTYDmwEmJibesWPHjjnHNDMzw6pVq0auYSVZTO1PH3h5pH6XrnnDgvY/Tr2+5tbdl8XUfeWVV+6pqsnZto0yvQNAklXAV4CPVtVPk9wDfAqodv+HwG8taIRDqmorsBVgcnKypqam5uw/PT3NfH1WqsXUfvOWr43Ub/9NC9v/OPX6mlt3X8ZV90ihn+QcBoH/xar6KkBVvTi0/Y+BB9vqAWDt0MMvam3M0S5JWgajnL0T4F7guar69FD7hUPdfhN4pi3vBG5M8pokFwPrgceAx4H1SS5O8moGX/buXJoyJEmjGOVI/53AB4GnkzzZ2j4BfCDJBgbTO/uB3wGoqr1J7mPwBe1R4Naq+iVAktuAh4GzgG1VtXfJKpEkzWuUs3e+CWSWTQ/N8Zi7gLtmaX9orsdJksbLX+RKUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdWTe0E+yNskjSZ5NsjfJR1r7eUl2JXm+3Z/b2pPks0n2JXkqyduH9rWp9X8+yabxlSVJms0oR/pHgdur6hLgCuDWJJcAW4DdVbUe2N3WAa4F1rfbZuAeGHxIAHcAlwOXAXcc+6CQJC2PeUO/qg5W1bfb8s+A54A1wPXA9tZtO3BDW74e+EINPAqsTnIhcA2wq6oOV9URYBewcSmLkSTNLVU1eudkHfAN4C3A/6iq1a09wJGqWp3kQeDuqvpm27Yb+BgwBby2qn6vtf8r4H9W1R8c9xybGfwLgYmJiXfs2LFjzjHNzMywatWqkWtYSRZT+9MHXh6p36Vr3rCg/Y9Tr6+5dfdlMXVfeeWVe6pqcrZtZ4+6kySrgK8AH62qnw5yfqCqKsnonx5zqKqtwFaAycnJmpqamrP/9PQ08/VZqRZT+81bvjZSv/03LWz/49Tra27dfRlX3SOdvZPkHAaB/8Wq+mprfrFN29DuD7X2A8DaoYdf1NpO1C5JWiajnL0T4F7guar69NCmncCxM3A2AQ8MtX+oncVzBfByVR0EHgauTnJu+wL36tYmSVomo0zvvBP4IPB0kidb2yeAu4H7ktwC/Ah4f9v2EHAdsA/4BfBhgKo6nORTwOOt351VdXgpipAkjWbe0G9fyOYEm989S/8Cbj3BvrYB205mgJKkpeMvciWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUkZH/iIrODOtG/OMokvrkkb4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI/OGfpJtSQ4leWao7XeTHEjyZLtdN7Tt40n2Jfl+kmuG2je2tn1Jtix9KZKk+YxypP95YOMs7Z+pqg3t9hBAkkuAG4E3t8f8UZKzkpwFfA64FrgE+EDrK0laRvNehqGqvpFk3Yj7ux7YUVWvAD9Msg+4rG3bV1U/AEiyo/V99uSHLElaqMVce+e2JB8CngBur6ojwBrg0aE+L7Q2gB8f1375bDtNshnYDDAxMcH09PScg5iZmZm3z0o1W+23X3p0SZ/jdPxv2+trbt19GVfdCw39e4BPAdXu/xD4raUYUFVtBbYCTE5O1tTU1Jz9p6enma/PSjVb7Tcv8QXX9t80NW+f5dbra27dfRlX3QsK/ap68dhykj8GHmyrB4C1Q10vam3M0S5JWiYLOmUzyYVDq78JHDuzZydwY5LXJLkYWA88BjwOrE9ycZJXM/iyd+fChy1JWoh5j/STfAmYAs5P8gJwBzCVZAOD6Z39wO8AVNXeJPcx+IL2KHBrVf2y7ec24GHgLGBbVe1d6mIkSXMb5eydD8zSfO8c/e8C7pql/SHgoZManSRpSfmLXEnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktSRxVx7R/pr60a8/MP+u9875pFImotH+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI/OGfpJtSQ4leWao7bwku5I83+7Pbe1J8tkk+5I8leTtQ4/Z1Po/n2TTeMqRJM1llCP9zwMbj2vbAuyuqvXA7rYOcC2wvt02A/fA4EMCuAO4HLgMuOPYB4UkafnMG/pV9Q3g8HHN1wPb2/J24Iah9i/UwKPA6iQXAtcAu6rqcFUdAXbx/3+QSJLGbKFz+hNVdbAt/wSYaMtrgB8P9XuhtZ2oXZK0jBb95xKrqpLUUgwGIMlmBlNDTExMMD09PWf/mZmZefusVLPVfvulR5f0OUb9bzvq8y7Fa9Xra27dfRlX3QsN/ReTXFhVB9v0zaHWfgBYO9TvotZ2AJg6rn16th1X1VZgK8Dk5GRNTU3N1u2vTU9PM1+flWq22m8e8W/Vjuzpn4/YcbS30v6bphY8lGN6fc2tuy/jqnuh0zs7gWNn4GwCHhhq/1A7i+cK4OU2DfQwcHWSc9sXuFe3NknSMpr38CzJlxgcpZ+f5AUGZ+HcDdyX5BbgR8D7W/eHgOuAfcAvgA8DVNXhJJ8CHm/97qyq478cliSN2byhX1UfOMGmd8/St4BbT7CfbcC2kxqdJGlJ+YtcSeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHFv2H0aWTsW7Ev+G7/+73jnkkUp880pekjhj6ktQRQ1+SOmLoS1JHDH1J6siiQj/J/iRPJ3kyyROt7bwku5I83+7Pbe1J8tkk+5I8leTtS1GAJGl0S3Gkf2VVbaiqyba+BdhdVeuB3W0d4FpgfbttBu5ZgueWJJ2EcUzvXA9sb8vbgRuG2r9QA48Cq5NcOIbnlySdQKpq4Q9OfggcAQr491W1NclLVbW6bQ9wpKpWJ3kQuLuqvtm27QY+VlVPHLfPzQz+JcDExMQ7duzYMecYZmZmWLVq1YJrOJPNVvvTB14+RaNZWpeuecMJt/X6mlt3XxZT95VXXrlnaPbl/7HYX+S+q6oOJPlVYFeS7w1vrKpKclKfKlW1FdgKMDk5WVNTU3P2n56eZr4+K9Vstd884i9eT3f7b5o64bZeX3Pr7su46l7U9E5VHWj3h4D7gcuAF49N27T7Q637AWDt0MMvam2SpGWy4CP9JK8HXlVVP2vLVwN3AjuBTcDd7f6B9pCdwG1JdgCXAy9X1cHFDL4XJ7peze2XHl0xR/aSlsdipncmgPsH0/acDfxpVf2XJI8D9yW5BfgR8P7W/yHgOmAf8Avgw4t4bknSAiw49KvqB8BbZ2n/K+Dds7QXcOtCn0+StHj+IleSOuL19HVamuu6+8PfZXjdfenkeKQvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSNehuEUmutSA5I0Dh7pS1JHDH1J6oihL0kdMfQlqSOGviR1xLN3dEY7mTOg/IMrkkf6ktQVQ1+SOrLs0ztJNgL/FjgL+JOqunu5xyDNZdQpI6eLdCZa1tBPchbwOeA9wAvA40l2VtWzyzmOcfOXtpJOV8t9pH8ZsK+qfgCQZAdwPbCiQl+np6X+MF7uD/fbLz3KzVu+5r8wtCipquV7suR9wMaq+u22/kHg8qq6bajPZmBzW/37wPfn2e35wF+OYbhngl5rt+6+WPfJ+ztVdcFsG067UzaraiuwddT+SZ6oqskxDum01Wvt1t0X615ay332zgFg7dD6Ra1NkrQMljv0HwfWJ7k4yauBG4GdyzwGSerWsk7vVNXRJLcBDzM4ZXNbVe1d5G5HngpagXqt3br7Yt1LaFm/yJUknVr+IleSOmLoS1JHzujQT7IxyfeT7Euy5VSPZ1ySbEtyKMkzQ23nJdmV5Pl2f+6pHOM4JFmb5JEkzybZm+QjrX1F157ktUkeS/LdVvcnW/vFSb7V3u9/1k6GWHGSnJXkO0kebOu91L0/ydNJnkzyRGtb8vf6GRv6Q5d0uBa4BPhAkktO7ajG5vPAxuPatgC7q2o9sLutrzRHgdur6hLgCuDW9hqv9NpfAa6qqrcCG4CNSa4Afh/4TFW9CTgC3HLqhjhWHwGeG1rvpW6AK6tqw9D5+Uv+Xj9jQ5+hSzpU1f8Gjl3SYcWpqm8Ah49rvh7Y3pa3Azcs55iWQ1UdrKpvt+WfMQiCNazw2mtgpq2e024FXAV8ubWvuLoBklwEvBf4k7YeOqh7Dkv+Xj+TQ38N8OOh9RdaWy8mqupgW/4JMHEqBzNuSdYBbwO+RQe1tymOJ4FDwC7gL4CXqupo67JS3+//BviXwP9p62+kj7ph8MH+X5PsaZejgTG810+7yzDo5FVVJVmx594mWQV8BfhoVf10cPA3sFJrr6pfAhuSrAbuB37t1I5o/JL8BnCoqvYkmTrFwzkV3lVVB5L8KrAryfeGNy7Ve/1MPtLv/ZIOLya5EKDdHzrF4xmLJOcwCPwvVtVXW3MXtQNU1UvAI8CvA6uTHDtQW4nv93cC/zDJfgbTtVcx+NsbK71uAKrqQLs/xOCD/jLG8F4/k0O/90s67AQ2teVNwAOncCxj0eZz7wWeq6pPD21a0bUnuaAd4ZPkdQz+/sRzDML/fa3biqu7qj5eVRdV1ToG/z9/vapuYoXXDZDk9Ul+5dgycDXwDGN4r5/Rv8hNch2DOcBjl3S469SOaDySfAmYYnCp1ReBO4D/DNwH/G3gR8D7q+r4L3vPaEneBfx34Gn+Zo73Ewzm9Vds7Un+AYMv7c5icGB2X1XdmeTvMjgCPg/4DvCPq+qVUzfS8WnTO/+8qn6jh7pbjfe31bOBP62qu5K8kSV+r5/RoS9JOjln8vSOJOkkGfqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI/8XPx8/NkAzKbwAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's plot the age of apprentice\n",
    "df_contracts.a_age.hist(bins=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    9303.000000\n",
       "mean       14.266688\n",
       "std         2.902770\n",
       "min         1.000000\n",
       "25%        12.000000\n",
       "50%        14.000000\n",
       "75%        16.000000\n",
       "max        50.000000\n",
       "Name: a_age, dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_contracts.a_age.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAU90lEQVR4nO3df5BdZ13H8ffXhoLTYJNa3Mkk0QSJOIUOkO60dURmQzVNC5Kq2CnToQHjZJwpDIx1JMhokR8zRa1IZ7AaaSRFINRipxlASwy9Mv2jpYSW/qRkaVObTNpoE4I3RTD16x/3WbxddrM3t2fv3c3zfs3c2XOe89xzvufs5nPOfe65N5GZSJLq8BPDLkCSNDiGviRVxNCXpIoY+pJUEUNfkiqyYNgFHM+ZZ56ZK1as6Ou5R48e5bTTTmu2oFkyX2q1zubNl1qts1mzXefu3bv/MzNfMuXCzJyzj3POOSf7dfvtt/f93EGbL7VaZ/PmS63W2azZrhP4ek6Tqw7vSFJFDH1JqoihL0kVMfQlqSKGviRVxNCXpIoY+pJUkZ5CPyIWRcTNEfGtiHg4In4pIs6IiJ0Rsaf8XFz6RkRcFxHjEXFfRKzuWs+G0n9PRGyYrZ2SJE2t1yv9jwH/kpm/CLwKeBjYDOzKzFXArjIPcBGwqjw2AdcDRMQZwNXAecC5wNUTJwpJ0mDM+DUMEXE68DrgbQCZ+UPghxGxHhgr3bYBLeA9wHrgxvKpsDvLq4Qlpe/OzDxU1rsTWAd8trndea4Vm7/YU7+917xhtkqQpDmllyv9lcB/AH8fEfdExCci4jRgJDMPlD5PAiNleinwRNfz95W26dolSQPSyxeuLQBWA+/MzLsi4mP8/1AOAJmZEdHI/7sYEZvoDAsxMjJCq9Xqaz3tdpurzn62p779bqMp7XZ76DX0wjqbN19qtc5mDbPOXkJ/H7AvM+8q8zfTCf2nImJJZh4owzcHy/L9wPKu5y8rbfv5/+GgifbW5I1l5hZgC8Do6GiOjY1N7tKTVqvFtXcc7anv3sv720ZTWq0W/e7nIFln8+ZLrdbZrGHWOePwTmY+CTwRES8vTRcADwE7gIk7cDYAt5bpHcAV5S6e84EjZRjoNmBtRCwub+CuLW2SpAHp9fv03wl8OiJOBR4F3k7nhHFTRGwEHgcuLX2/BFwMjAPPlL5k5qGI+CBwd+n3gYk3dSVJg9FT6GfmvcDoFIsumKJvAldOs56twNYTqE+S1CA/kStJFTH0Jakihr4kVcTQl6SKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKGviRVxNCXpIoY+pJUEUNfkipi6EtSRQx9SaqIoS9JFTH0JakiPYV+ROyNiPsj4t6I+HppOyMidkbEnvJzcWmPiLguIsYj4r6IWN21ng2l/56I2DA7uyRJms6JXOmvycxXZ+Zomd8M7MrMVcCuMg9wEbCqPDYB10PnJAFcDZwHnAtcPXGikCQNxvMZ3lkPbCvT24BLutpvzI47gUURsQS4ENiZmYcy8zCwE1j3PLYvSTpBkZkzd4p4DDgMJPC3mbklIr6bmYvK8gAOZ+aiiPgCcE1m3lGW7QLeA4wBL8rMD5X2Pwa+n5l/MWlbm+i8QmBkZOSc7du397Vj7Xabx44821Pfs5ee3tc2mtJut1m4cOFQa+iFdTZvvtRqnc2a7TrXrFmzu2tU5jkW9LiO12bm/oj4GWBnRHyre2FmZkTMfPboQWZuAbYAjI6O5tjYWF/rabVaXHvH0Z767r28v200pdVq0e9+DpJ1Nm++1GqdzRpmnT0N72Tm/vLzIHALnTH5p8qwDeXnwdJ9P7C86+nLStt07ZKkAZkx9CPitIh48cQ0sBZ4ANgBTNyBswG4tUzvAK4od/GcDxzJzAPAbcDaiFhc3sBdW9okSQPSy/DOCHBLZ9ieBcBnMvNfIuJu4KaI2Ag8Dlxa+n8JuBgYB54B3g6QmYci4oPA3aXfBzLzUGN7Ikma0Yyhn5mPAq+aov1p4IIp2hO4cpp1bQW2nniZkqQm+IlcSaqIoS9JFTH0Jakihr4kVcTQl6SKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKGviRVxNCXpIoY+pJUEUNfkipi6EtSRQx9SapIz6EfEadExD0R8YUyvzIi7oqI8Yj4XEScWtpfWObHy/IVXet4b2l/JCIubHxvJEnHdSJX+u8CHu6a/wjw0cx8GXAY2FjaNwKHS/tHSz8i4izgMuAVwDrgryPilOdXviTpRPQU+hGxDHgD8IkyH8DrgZtLl23AJWV6fZmnLL+g9F8PbM/MH2TmY8A4cG4D+yBJ6tGCHvv9FfCHwIvL/E8D383MY2V+H7C0TC8FngDIzGMRcaT0Xwrc2bXO7uf8SERsAjYBjIyM0Gq1eizxudrtNled/WxPffvdRlPa7fbQa+iFdTZvvtRqnc0aZp0zhn5EvBE4mJm7I2JstgvKzC3AFoDR0dEcG+tvk61Wi2vvONpT372X97eNprRaLfrdz0GyzubNl1qts1nDrLOXK/1fBt4UERcDLwJ+CvgYsCgiFpSr/WXA/tJ/P7Ac2BcRC4DTgae72id0P0eSNAAzjuln5nszc1lmrqDzRuxXMvNy4HbgzaXbBuDWMr2jzFOWfyUzs7RfVu7uWQmsAr7W2J5IkmbU65j+VN4DbI+IDwH3ADeU9huAT0XEOHCIzomCzHwwIm4CHgKOAVdmZm+D7pKkRpxQ6GdmC2iV6UeZ4u6bzPxv4Lenef6HgQ+faJGSpGb4iVxJqoihL0kVMfQlqSKGviRVxNCXpIoY+pJUEUNfkipi6EtSRQx9SaqIoS9JFTH0Jakihr4kVcTQl6SKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqsiMoR8RL4qIr0XENyPiwYj409K+MiLuiojxiPhcRJxa2l9Y5sfL8hVd63pvaX8kIi6ctb2SJE2plyv9HwCvz8xXAa8G1kXE+cBHgI9m5suAw8DG0n8jcLi0f7T0IyLOAi4DXgGsA/46Ik5pcF8kSTOYMfSzo11mX1AeCbweuLm0bwMuKdPryzxl+QUREaV9e2b+IDMfA8aBc5vYCUlSbyIzZ+7UuSLfDbwM+Djw58Cd5WqeiFgO/HNmvjIiHgDWZea+suw7wHnA+8tz/qG031Cec/OkbW0CNgGMjIycs3379r52rN1u89iRZ3vqe/bS0/vaRlPa7TYLFy4cag29sM7mzZdarbNZs13nmjVrdmfm6FTLFvSygsx8Fnh1RCwCbgF+sbnyfmxbW4AtAKOjozk2NtbXelqtFtfecbSnvnsv728bTWm1WvS7n4Nknc2bL7VaZ7OGWecJ3b2Tmd8Fbgd+CVgUERMnjWXA/jK9H1gOUJafDjzd3T7FcyRJA9DL3TsvKVf4RMRPAr8GPEwn/N9cum0Abi3TO8o8ZflXsjOGtAO4rNzdsxJYBXytof2QJPWgl+GdJcC2Mq7/E8BNmfmFiHgI2B4RHwLuAW4o/W8APhUR48AhOnfskJkPRsRNwEPAMeDKMmwkSRqQGUM/M+8DXjNF+6NMcfdNZv438NvTrOvDwIdPvExJUhP8RK4kVcTQl6SKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKGviRVxNCXpIoY+pJUEUNfkipi6EtSRQx9SaqIoS9JFTH0Jakihr4kVWTG0I+I5RFxe0Q8FBEPRsS7SvsZEbEzIvaUn4tLe0TEdRExHhH3RcTqrnVtKP33RMSG2dstSdJUernSPwZclZlnAecDV0bEWcBmYFdmrgJ2lXmAi4BV5bEJuB46JwngauA84Fzg6okThSRpMGYM/cw8kJnfKNP/BTwMLAXWA9tKt23AJWV6PXBjdtwJLIqIJcCFwM7MPJSZh4GdwLomd0aSdHyRmb13jlgBfBV4JfDvmbmotAdwODMXRcQXgGsy846ybBfwHmAMeFFmfqi0/zHw/cz8i0nb2ETnFQIjIyPnbN++va8da7fbPHbk2Z76nr309L620ZR2u83ChQuHWkMvrLN586VW62zWbNe5Zs2a3Zk5OtWyBb2uJCIWAp8H3p2Z3+vkfEdmZkT0fvY4jszcAmwBGB0dzbGxsb7W02q1uPaOoz313Xt5f9toSqvVot/9HCTrbN58qdU6mzXMOnu6eyciXkAn8D+dmf9Ump8qwzaUnwdL+35gedfTl5W26dolSQPSy907AdwAPJyZf9m1aAcwcQfOBuDWrvYryl085wNHMvMAcBuwNiIWlzdw15Y2SdKA9DK888vAW4H7I+Le0vZHwDXATRGxEXgcuLQs+xJwMTAOPAO8HSAzD0XEB4G7S78PZOahJnZCktSbGUO/vCEb0yy+YIr+CVw5zbq2AltPpEBJUnP8RK4kVcTQl6SKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKGviRVxNCXpIoY+pJUkV7+Y/ST3orNX+yp395r3jDLlUjS7PJKX5IqYuhLUkUMfUmqiKEvSRUx9CWpIjOGfkRsjYiDEfFAV9sZEbEzIvaUn4tLe0TEdRExHhH3RcTqrudsKP33RMSG2dkdSdLx9HKl/0lg3aS2zcCuzFwF7CrzABcBq8pjE3A9dE4SwNXAecC5wNUTJwpJ0uDMGPqZ+VXg0KTm9cC2Mr0NuKSr/cbsuBNYFBFLgAuBnZl5KDMPAzv58ROJJGmW9TumP5KZB8r0k8BImV4KPNHVb19pm65dkjRAz/sTuZmZEZFNFAMQEZvoDA0xMjJCq9Xqaz3tdpurzn62qbIA+q5lJu12e9bW3STrbN58qdU6mzXMOvsN/aciYklmHijDNwdL+35geVe/ZaVtPzA2qb011YozcwuwBWB0dDTHxsam6jajVqvFtXcc7eu509l7eX+1zKTVatHvfg6SdTZvvtRqnc0aZp39Du/sACbuwNkA3NrVfkW5i+d84EgZBroNWBsRi8sbuGtLmyRpgGa80o+Iz9K5Sj8zIvbRuQvnGuCmiNgIPA5cWrp/CbgYGAeeAd4OkJmHIuKDwN2l3wcyc/Kbw5KkWTZj6GfmW6ZZdMEUfRO4cpr1bAW2nlB1kqRG+YlcSaqIoS9JFTH0Jakihr4kVcTQl6SKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRUx9CWpIoa+JFXE0Jekihj6klQRQ1+SKmLoS1JFDH1JqoihL0kVMfQlqSKGviRVxNCXpIosGHYB88mKzV/sqd/ea94wy5VIUn+80pekihj6klQRQ1+SKuKY/ixw7F/SXDXw0I+IdcDHgFOAT2TmNYOuYa6YODlcdfYx3nacE4UnB0lNGejwTkScAnwcuAg4C3hLRJw1yBokqWaDvtI/FxjPzEcBImI7sB54aMB1zCu9DhediF5fPXRve6ZXJE1uV9LsiMwc3MYi3gysy8zfLfNvBc7LzHd09dkEbCqzLwce6XNzZwL/+TzKHaT5Uqt1Nm++1GqdzZrtOn8uM18y1YI590ZuZm4Btjzf9UTE1zNztIGSZt18qdU6mzdfarXOZg2zzkHfsrkfWN41v6y0SZIGYNChfzewKiJWRsSpwGXAjgHXIEnVGujwTmYei4h3ALfRuWVza2Y+OEube95DRAM0X2q1zubNl1qts1lDq3Ogb+RKkobLr2GQpIoY+pJUkZMy9CNiXUQ8EhHjEbF52PVMiIjlEXF7RDwUEQ9GxLtK+/sjYn9E3FseF8+BWvdGxP2lnq+XtjMiYmdE7Ck/F8+BOl/eddzujYjvRcS758IxjYitEXEwIh7oapvyGEbHdeVv9r6IWD3kOv88Ir5VarklIhaV9hUR8f2u4/o3g6rzOLVO+7uOiPeWY/pIRFw45Do/11Xj3oi4t7QP9phm5kn1oPMG8XeAlwKnAt8Ezhp2XaW2JcDqMv1i4Nt0vo7i/cAfDLu+SbXuBc6c1PZnwOYyvRn4yLDrnOJ3/yTwc3PhmAKvA1YDD8x0DIGLgX8GAjgfuGvIda4FFpTpj3TVuaK73xw5plP+rsu/rW8CLwRWllw4ZVh1Tlp+LfAnwzimJ+OV/o++6iEzfwhMfNXD0GXmgcz8Rpn+L+BhYOlwqzoh64FtZXobcMnwSpnSBcB3MvPxYRcCkJlfBQ5Nap7uGK4HbsyOO4FFEbFkWHVm5pcz81iZvZPOZ2qGbppjOp31wPbM/EFmPgaM08mHWXe8OiMigEuBzw6ilslOxtBfCjzRNb+PORisEbECeA1wV2l6R3kpvXUuDJsACXw5InaXr8YAGMnMA2X6SWBkOKVN6zKe+w9prh1TmP4YzuW/29+h8ypkwsqIuCci/i0ifmVYRU0y1e96rh7TXwGeysw9XW0DO6YnY+jPeRGxEPg88O7M/B5wPfDzwKuBA3Re+g3bazNzNZ1vRL0yIl7XvTA7r0vnzP2+5cN+bwL+sTTNxWP6HHPtGE4lIt4HHAM+XZoOAD+bma8Bfh/4TET81LDqK+b873qSt/Dci5OBHtOTMfTn9Fc9RMQL6AT+pzPznwAy86nMfDYz/xf4Owb0EvR4MnN/+XkQuIVOTU9NDDmUnweHV+GPuQj4RmY+BXPzmBbTHcM593cbEW8D3ghcXk5QlKGSp8v0bjrj5L8wtCI57u96Lh7TBcBvAp+baBv0MT0ZQ3/OftVDGcu7AXg4M/+yq7177PY3gAcmP3eQIuK0iHjxxDSdN/UeoHMcN5RuG4Bbh1PhlJ5z9TTXjmmX6Y7hDuCKchfP+cCRrmGggYvOf3b0h8CbMvOZrvaXROf/xSAiXgqsAh4dTpU/qmm63/UO4LKIeGFErKRT69cGXd8kvwp8KzP3TTQM/JgO6h3jQT7o3AnxbTpnzPcNu56uul5L5+X8fcC95XEx8Cng/tK+A1gy5DpfSueuh28CD04cQ+CngV3AHuBfgTOGfUxLXacBTwOnd7UN/ZjSOQkdAP6HznjyxumOIZ27dj5e/mbvB0aHXOc4nfHwib/Tvyl9f6v8TdwLfAP49TlwTKf9XQPvK8f0EeCiYdZZ2j8J/N6kvgM9pn4NgyRV5GQc3pEkTcPQl6SKGPqSVBFDX5IqYuhLUkUMfUmqiKEvSRX5P+pfGFy48xAEAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's plot the distribution of annual salary\n",
    "df_contracts.annual_salary.hist(bins=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:ylabel='Frequency'>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAD4CAYAAAAD6PrjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAARTUlEQVR4nO3df7BndV3H8edLEH+QYgqZww8XXCKZ0ZKu1kzRLzVBWlA0g7HJjIEoaXKaptZ0in+aoZp+WSityaCmIJba7rAOZpPyDwkLooCIrLjGIgHhzFLmSNi7P75nj5fLvbtnl/3c8z17n4+ZO3u+n/v9fvc9n+/d+9rPj3NOqgpJkgCeNHYBkqT5YShIknqGgiSpZyhIknqGgiSpd+jYBTwRRx55ZK1bt27sMiRpUm666ab/rKqjlvveJEMhyQZgw/r169m2bdvY5UjSpCT52krfm+T0UVVtqaoLjjjiiLFLkaSDyiRDQZLUxiRDIcmGJJt27do1dimSdFCZZCg4fSRJbUwyFCRJbUwyFJw+kqQ2JhkKTh9JUhuTDAVJUhuTP3ltf63beM2g5+245Iz9/jskaWomOVJw+kiS2phkKEiS2jAUJEk9Q0GS1JtkKHiegiS1MclQcKFZktqYZChIktowFCRJPUNBktSbZCi40CxJbUwyFFxolqQ2JhkKkqQ2DAVJUs9QkCT1DAVJUm+SoeDuI0lqY5Kh4O4jSWpjkqEgSWrDUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVJvkqHgyWuS1MYkQ8GT1ySpjUmGgiSpDUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJvUPHLmDerdt4zeDn7rjkjIaVSFJ7jhQkSb25CYUkL0xyWZJ/SPLrY9cjSWtR01BIcnmSB5LctqT9tCR3JtmeZCNAVd1RVRcCbwB+vGVdkqTltR4pXAGctrghySHApcDpwMnAuUlO7r53JnANsLVxXZKkZTQNhaq6DvjGkuaXAdur6u6qegS4Cjire/7mqjodeONK75nkgiTbkmx78MEHW5UuSWvSGLuPjgbuWfR4J/CjSX4aOBt4CnsYKVTVJmATwMLCQjWrUpLWoLnZklpVnwY+PXIZkrSmjbH76F7g2EWPj+naBvN2nJLUxhihcCNwYpLjkxwGnANs3pc38HacktRG6y2pVwLXAycl2ZnkvKp6FLgIuBa4A7i6qm7fx/d1pCBJDTRdU6iqc1do38oT2HZaVVuALQsLC+fv73tIkh5vbs5oliSNb5Kh4PSRJLUxyVBwoVmS2phkKEiS2jAUJEm9SYaCawqS1MYkQ8E1BUlqY5KhIElqw1CQJPUmGQquKUhSG5MMBdcUJKmNSYaCJKkNQ0GS1DMUJEm9SYaCC82S1MYkQ8GFZklqY5KhIElqw1CQJPUMBUlSz1CQJPUmGQruPpKkNiYZCu4+kqQ2JhkKkqQ2DAVJUs9QkCT1DAVJUs9QkCT1Dh27gIPJuo3XDHrejkvOaFyJJO2fQSOFJC9qXYgkaXxDp4/eleSGJL+RxJMDJOkgNSgUqupU4I3AscBNST6U5JVNK9sDz2iWpDYGLzRX1V3AO4DfA34KeGeSLyU5u1Vxe6jFM5olqYGhawovTvIXwB3AzwIbquqF3fFfNKxPkrSKhu4++mvg74Dfr6pv7W6sqq8neUeTyiRJq25oKJwBfKuqvgOQ5EnAU6vqf6rqA82qkyStqqFrCp8Cnrbo8dO7NknSQWRoKDy1qv5794Pu+OltSpIkjWVoKHwzySm7HyT5EeBbe3i+JGmChq4pvBX4SJKvAwG+H/jFVkVJksYxKBSq6sYkPwic1DXdWVX/264sSdIY9uWCeC8F1nWvOSUJVfX+JlVJkkYxKBSSfAB4AXAL8J2uuQBDYT94NVVJ82roSGEBOLmqqmUxSV7D7JyIZwLvrapPtvz7JEmPNXT30W3MFpf3WZLLkzyQ5LYl7acluTPJ9iQbAarq41V1PnAhLmRL0qobOlI4EvhikhuAb+9urKozB7z2CuBvWDTVlOQQ4FLglcBO4MYkm6vqi91T3tF9X5K0ioaGwsX7+xdU1XVJ1i1pfhmwvaruBkhyFXBWkjuAS4BPVNXNy71fkguACwCOO+64/S1LkrSMofdT+AywA3hyd3wjsOwv7YGOBu5Z9Hhn1/abwCuA1ye5cIVaNlXVQlUtHHXUUU+gBEnSUkN3H53P7H/nz2a2C+lo4DLg5QeymKp6J/DOA/mekqThhk4fvYXZlM9nYXbDnSTf9wT+3nuZ3cVtt2O6tkGSbAA2rF+//gmUMP/cuipptQ3dffTtqnpk94MkhzI7T2F/3QicmOT4JIcB5wCbh77YO69JUhtDQ+EzSX4feFp3b+aPAFuGvDDJlcD1wElJdiY5r6oeBS4CrmV2N7erq+r2oUV7j2ZJamPo9NFG4DzgVuDXgK3M7sS2V1V17grtW7v32WdVtQXYsrCwcP7+vF6StLyhF8T7P+A93Zck6SA1dPfRV1lmDaGqTjjgFQ2wVhaaJWm17cu1j3Z7KvALzLanjsLpI0lqY+jJaw8t+rq3qv6S2YXrJEkHkaHTR6csevgkZiOHfbkXgyRpAob+Yv+zRcePMrvkxRsOeDUDuaYgSW0M3X30M60L2ReuKUhSG0Onj357T9+vqj8/MOVIksa0L7uPXsp3L0WxAbgBuKtFUZKkcQwNhWOAU6rqvwCSXAxcU1W/1KqwPXFNQZLaGHrto+cCjyx6/EjXNgoviCdJbQwdKbwfuCHJx7rHrwHe16QiSdJohu4++qMknwBO7ZreXFWfa1eWJGkMQ6ePAJ4OPFxVfwXsTHJ8o5okSSMZFApJ/hD4PeBtXdOTgb9vVdSAeryfgiQ1MHSk8FrgTOCbAFX1deAZrYraGxeaJamNoaHwSFUV3eWzkxzeriRJ0liGhsLVSf4WeFaS84FP4Q13JOmgs9fdR0kCfBj4QeBh4CTgD6rqnxvXpgNs3cZrBj1vxyVeFV1aq/YaClVVSbZW1YsAg0CSDmJDp49uTvLSppVIkkY39IzmHwV+KckOZjuQwmwQ8eJWhe2J1z6SpDb2GApJjquqfwdetUr1DOL9FCSpjb2NFD7O7OqoX0vyj1X1ulWoSZI0kr2tKWTR8QktC5EkjW9voVArHEuSDkJ7mz76oSQPMxsxPK07hu8uND+zaXWSpFW1x1CoqkNWqxBJ0vj25dLZkqSDnKEgSeoNPXltrnjy2mMNvaaRJO3NJEcK3k9BktqYZChIktowFCRJPUNBktSb5EKz5sO+LHB74x5pGhwpSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqTc3W1KTnAC8HTiiql4/dj06sIZuX3XrqjSupiOFJJcneSDJbUvaT0tyZ5LtSTYCVNXdVXVey3okSXvWevroCuC0xQ1JDgEuBU4HTgbOTXJy4zokSQM0DYWqug74xpLmlwHbu5HBI8BVwFkt65AkDTPGQvPRwD2LHu8Ejk7ynCSXAS9J8raVXpzkgiTbkmx78MEHW9cqSWvK3Cw0V9VDwIUDnrcJ2ASwsLBQreuSpLVkjJHCvcCxix4f07UNlmRDkk27du06oIVJ0lo3RijcCJyY5PgkhwHnAJv35Q2885oktdF6S+qVwPXASUl2Jjmvqh4FLgKuBe4Arq6q21vWIUkapumaQlWdu0L7VmDr/r5vkg3AhvXr1+/vW0iSljHJy1w4fSRJbUwyFCRJbUwyFNx9JEltTDIUnD6SpDYmGQqSpDbm5ozmfeHuI3kpbqmNSY4UnD6SpDYmGQqSpDYMBUlSzzUFPc7Q+XpJB59JjhRcU5CkNiYZCpKkNgwFSVLPUJAk9SYZCl77SJLamGQouNAsSW1MMhQkSW0YCpKknqEgSeoZCpKknqEgSep57SPNlXm/7pL3cdDBbpIjBbekSlIbkwwFSVIbhoIkqWcoSJJ6hoIkqWcoSJJ6hoIkqWcoSJJ6nrwmMf8nzUmrZZIjBU9ek6Q2JhkKkqQ2DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT1DAVJUs9QkCT15uaCeEkOB94FPAJ8uqo+OHJJkrTmNB0pJLk8yQNJblvSflqSO5NsT7Kxaz4b+IeqOh84s2VdkqTltZ4+ugI4bXFDkkOAS4HTgZOBc5OcDBwD3NM97TuN65IkLaPp9FFVXZdk3ZLmlwHbq+pugCRXAWcBO5kFwy3sIaySXABcAHDccccd+KJ1UJnCfRKG1rjjkjMaV6Kx7cvPa6ufhzEWmo/muyMCmIXB0cBHgdcleTewZaUXV9WmqlqoqoWjjjqqbaWStMbMzUJzVX0TePOQ53rnNUlqY4yRwr3AsYseH9O1Dead1ySpjTFC4UbgxCTHJzkMOAfYPEIdkqQlWm9JvRK4Hjgpyc4k51XVo8BFwLXAHcDVVXX7Pr7vhiSbdu3adeCLlqQ1rPXuo3NXaN8KbH0C77sF2LKwsHD+/r6HJOnxvMyFJKk3yVBw+kiS2phkKLj7SJLaSFWNXcN+S/Ig8LX9fPmRwH8ewHJamUqdMJ1arfPAm0qt1jnz/Kpa9uzfSYfCE5FkW1UtjF3H3kylTphOrdZ54E2lVuvcu0lOH0mS2jAUJEm9tRwKm8YuYKCp1AnTqdU6D7yp1Gqde7Fm1xQkSY+3lkcKkqQlDAVJUm/NhcIK94eeC0mOTfKvSb6Y5PYkv9W1X5zk3iS3dF+vnoNadyS5tatnW9f27CT/nOSu7s/vHbnGkxb12S1JHk7y1nnpz+XuYb5SH2bmnd3P7ReSnDJynX+a5EtdLR9L8qyufV2Sby3q28tGrnPFzzrJ27r+vDPJq1arzj3U+uFFde5IckvXvrp9WlVr5gs4BPgKcAJwGPB54OSx61pU3/OAU7rjZwBfZnYf64uB3xm7viW17gCOXNL2J8DG7ngj8Mdj17nks/8P4Pnz0p/ATwKnALftrQ+BVwOfAAL8GPDZkev8OeDQ7viPF9W5bvHz5qA/l/2su39XnweeAhzf/V44ZMxal3z/z4A/GKNP19pIob8/dFU9Auy+P/RcqKr7qurm7vi/mF1a/Ohxq9onZwHv647fB7xmvFIe5+XAV6pqf8+AP+Cq6jrgG0uaV+rDs4D318y/Ac9K8ryx6qyqT9bsMvgA/8bsZlmjWqE/V3IWcFVVfbuqvgpsZ/b7YVXsqdYkAd4AXLla9Sy21kJhpftDz50k64CXAJ/tmi7qhuqXjz0t0yngk0luSnJB1/bcqrqvO/4P4LnjlLasc3jsP7J568/dVurDef7Z/VVmo5jdjk/yuSSfSXLqWEUtstxnPc/9eSpwf1Xdtaht1fp0rYXCJCT5HuAfgbdW1cPAu4EXAD8M3MdsaDm2n6iqU4DTgbck+cnF36zZuHcu9jtndoe/M4GPdE3z2J+PM099uJIkbwceBT7YNd0HHFdVLwF+G/hQkmeOVR8T+ayXOJfH/gdmVft0rYXCE74/dGtJnswsED5YVR8FqKr7q+o7VfV/wHtYxWHuSqrq3u7PB4CPMavp/t1TGt2fD4xX4WOcDtxcVffDfPbnIiv14dz97Cb5FeDngTd2AUY3HfNQd3wTs7n6Hxirxj181nPXnwBJDgXOBj68u221+3SthcJc3x+6m0t8L3BHVf35ovbFc8evBW5b+trVlOTwJM/Yfcxs0fE2Zn35pu5pbwL+aZwKH+cx//Oat/5cYqU+3Az8crcL6ceAXYummVZdktOA3wXOrKr/WdR+VJJDuuMTgBOBu8epco+f9WbgnCRPSXI8szpvWO36lvEK4EtVtXN3w6r36WqtaM/LF7NdHF9mlrZvH7ueJbX9BLPpgi8At3RfrwY+ANzatW8GnjdynScw27nxeeD23f0IPAf4F+Au4FPAs+egTw8HHgKOWNQ2F/3JLKjuA/6X2Zz2eSv1IbNdR5d2P7e3Agsj17md2Zz87p/Ty7rnvq77mbgFuBnYMHKdK37WwNu7/rwTOH3sz75rvwK4cMlzV7VPvcyFJKm31qaPJEl7YChIknqGgiSpZyhIknqGgiSpZyhIknqGgiSp9/+16QhAEUZLcQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's plot the age of apprentice\n",
    "# when plotting very skewed distribution is generally\n",
    "# better to use a logarithmic scale for the y axis\n",
    "# as this allows to 'see' data points that otherwise\n",
    "# will remain hidden\n",
    "df_contracts.annual_salary.plot(kind='hist', bins=30, logy=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    7870.000000\n",
       "mean        5.916921\n",
       "std         6.985214\n",
       "min         0.166667\n",
       "25%         3.000000\n",
       "50%         4.000000\n",
       "75%         6.000000\n",
       "max       180.000000\n",
       "Name: annual_salary, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_contracts.annual_salary.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAVq0lEQVR4nO3df5Bd5X3f8fe3yGDCupKAdEslTYUbjTMENQnsAKlTzypKsQCPRTsOg4exJYeMxlNISa1MkOtpyKT1VG5KPHaauqNGjOWG8eIQuyj8iK3IbD3+Q8SIYsQP2yxErrUji9pgOWtIHaXf/nGfde6z7K97z713d+H9mtnZc57znHO+99yz96Pz4x5FZiJJ0rS/s9QFSJKWF4NBklQxGCRJFYNBklQxGCRJlVVLXcB8Lrzwwty4cWNH8/zgBz/gvPPO609BPWB9zVhfM9bXzEqp7+jRo9/JzB/vekGZuWx/Lr/88uzUww8/3PE8g2R9zVhfM9bXzEqpD3g0G3z2eipJklQxGCRJFYNBklQxGCRJFYNBklQxGCRJFYNBklQxGCRJFYNBklRZ1o/EeK3ZuOcBdm8+w849Dyyq//G91/W5Ikl6NY8YJEkVg0GSVDEYJEkVg0GSVDEYJEkVg0GSVDEYJEkVg0GSVDEYJEkVg0GSVFkwGCLiroh4ISKebGv7nYj4WkQ8ERGfi4g1bdM+GBETEfH1iHh7W/u20jYREXt6/kokST2xmCOGTwLbZrQdAi7NzH8MfAP4IEBEXALcCPxUmee/RMRZEXEW8PvANcAlwLtLX0nSMrNgMGTml4AXZ7R9ITPPlNEjwPoyvB0Yy8z/m5l/AUwAV5Sficx8PjN/CIyVvpKkZSYyc+FOERuB+zPz0lmm/QlwT2b+YUT8Z+BIZv5hmbYfeKh03ZaZv1La3wNcmZm3zrK8XcAugOHh4cvHxsY6ekFTU1MMDQ11NM+gHJs8zfC5cOqVxfXfvG51fwuaxXLefmB9TVlfMyulvi1bthzNzJFul9PosdsR8SHgDHB3k+W0y8x9wD6AkZGRHB0d7Wj+8fFxOp1nUHaWx27feWxxm/34TaP9LWgWy3n7gfU1ZX3NvF7q6zoYImIn8A5ga/7tYccksKGt2/rSxjztkqRlpKvbVSNiG/AbwDsz8+W2SQeBGyPinIi4GNgE/DnwFWBTRFwcEWfTukB9sFnpkqR+WPCIISI+DYwCF0bECeAOWnchnQMcighoXVd4f2Y+FRGfAZ6mdYrplsz8m7KcW4HPA2cBd2XmU314PZKkhhYMhsx89yzN++fp/2Hgw7O0Pwg82FF1kqSB85vPkqRKo7uStPxs3PNAR/2P772uT5VIWqk8YpAkVQwGSVLFYJAkVQwGSVLFYJAkVQwGSVLFYJAkVQwGSVLFYJAkVQwGSVLFYJAkVQwGSVLFYJAkVQwGSVLFYJAkVQwGSVLFYJAkVQwGSVLFYJAkVQwGSVLFYJAkVRYMhoi4KyJeiIgn29rOj4hDEfFs+b22tEdEfDwiJiLiiYi4rG2eHaX/sxGxoz8vR5LU1GKOGD4JbJvRtgc4nJmbgMNlHOAaYFP52QV8AlpBAtwBXAlcAdwxHSaSpOVlwWDIzC8BL85o3g4cKMMHgOvb2j+VLUeANRFxEfB24FBmvpiZLwGHeHXYSJKWgcjMhTtFbATuz8xLy/j3MnNNGQ7gpcxcExH3A3sz88tl2mHgdmAUeGNm/vvS/m+BVzLzP82yrl20jjYYHh6+fGxsrKMXNDU1xdDQUEfzDMqxydMMnwunXllc/83rVne1jk7MXMdy3n5gfU1ZXzMrpb4tW7YczcyRbpezqmkhmZkRsXC6LH55+4B9ACMjIzk6OtrR/OPj43Q6z6Ds3PMAuzef4c5ji9vsx28a7WodnZi5juW8/cD6mrK+Zl4v9XV7V9KpcoqI8vuF0j4JbGjrt760zdUuSVpmug2Gg8D0nUU7gPva2t9b7k66CjidmSeBzwNXR8TactH56tImSVpmFjynERGfpnWN4MKIOEHr7qK9wGci4mbgm8ANpfuDwLXABPAy8D6AzHwxIv4d8JXS77czc+YFbUnSMrBgMGTmu+eYtHWWvgncMsdy7gLu6qg6SdLA+c1nSVKl8V1JWtk2zriLaffmM/Pe2XR873X9LknSEvOIQZJUMRgkSRWDQZJU8RpDAzPPz0vSa4FHDJKkisEgSaoYDJKkisEgSaoYDJKkisEgSaoYDJKkisEgSaoYDJKkisEgSaoYDJKkisEgSaoYDJKkisEgSaoYDJKkisEgSaoYDJKkSqNgiIh/HRFPRcSTEfHpiHhjRFwcEY9ExERE3BMRZ5e+55TxiTJ9Y09egSSpp7oOhohYB/wrYCQzLwXOAm4EPgJ8NDN/AngJuLnMcjPwUmn/aOknSVpmmp5KWgWcGxGrgB8DTgK/ANxbph8Ari/D28s4ZfrWiIiG65ck9VhkZvczR9wGfBh4BfgCcBtwpBwVEBEbgIcy89KIeBLYlpknyrTngCsz8zszlrkL2AUwPDx8+djYWEc1TU1NMTQ01PVr6sSxydMdzzN8Lpx6ZXF9N69b3fHyu6mp3UL1dVNTLw3y/e2G9TVjfc1M17dly5ajmTnS7XJWdTtjRKyldRRwMfA94I+Abd0ub1pm7gP2AYyMjOTo6GhH84+Pj9PpPN3aueeBjufZvfkMdx5b3GY/ftNox8vvpqZ2C9XXTU29NMj3txvW14z1NdOr+pqcSvpF4C8y8/9k5l8DnwXeCqwpp5YA1gOTZXgS2ABQpq8Gvttg/ZKkPmgSDP8buCoifqxcK9gKPA08DLyr9NkB3FeGD5ZxyvQvZpPzWJKkvug6GDLzEVoXkR8DjpVl7QNuBz4QERPABcD+Mst+4ILS/gFgT4O6JUl90vU1BoDMvAO4Y0bz88AVs/T9K+CXmqxPS29jh9cwju+9rk+VSOoXv/ksSaoYDJKkisEgSaoYDJKkisEgSaoYDJKkisEgSaoYDJKkisEgSaoYDJKkisEgSaoYDJKkSqOH6Km/On1gnST1gkcMkqSKwSBJqhgMkqSKwSBJqhgMkqSKwSBJqhgMkqSKwSBJqhgMkqSKwSBJqjQKhohYExH3RsTXIuKZiPi5iDg/Ig5FxLPl99rSNyLi4xExERFPRMRlvXkJkqReanrE8DHgTzPzJ4GfBp4B9gCHM3MTcLiMA1wDbCo/u4BPNFy3JKkPug6GiFgNvA3YD5CZP8zM7wHbgQOl2wHg+jK8HfhUthwB1kTERd2uX5LUH5GZ3c0Y8TPAPuBpWkcLR4HbgMnMXFP6BPBSZq6JiPuBvZn55TLtMHB7Zj46Y7m7aB1RMDw8fPnY2FhHdU1NTTE0NNTVa+rUscnTHc8zfC6ceqUPxfRIr+vbvG517xbGYN/fblhfM9bXzHR9W7ZsOZqZI90up8ljt1cBlwG/mpmPRMTH+NvTRgBkZkZER8mTmftoBQ4jIyM5OjraUVHj4+N0Ok+3dnbxWOzdm89w57Hl+7TzXtd3/KbRni0LBvv+dsP6mrG+ZnpVX5NrDCeAE5n5SBm/l1ZQnJo+RVR+v1CmTwIb2uZfX9okSctI18GQmd8GvhURbylNW2mdVjoI7ChtO4D7yvBB4L3l7qSrgNOZebLb9UuS+qPpOYNfBe6OiLOB54H30Qqbz0TEzcA3gRtK3weBa4EJ4OXSV5K0zDQKhsx8HJjtAsfWWfomcEuT9UmS+s9vPkuSKgaDJKliMEiSKgaDJKliMEiSKgaDJKliMEiSKgaDJKliMEiSKgaDJKliMEiSKgaDJKliMEiSKgaDJKliMEiSKgaDJKliMEiSKgaDJKliMEiSKgaDJKliMEiSKgaDJKliMEiSKo2DISLOioj/FRH3l/GLI+KRiJiIiHsi4uzSfk4ZnyjTNzZdtySp93pxxHAb8Ezb+EeAj2bmTwAvATeX9puBl0r7R0s/SdIy0ygYImI9cB3wB2U8gF8A7i1dDgDXl+HtZZwyfWvpL0laRiIzu5854l7gPwBvAn4d2AkcKUcFRMQG4KHMvDQingS2ZeaJMu054MrM/M6MZe4CdgEMDw9fPjY21lFNU1NTDA0Ndf2aOnFs8nTH8wyfC6de6UMxPdLr+javW927hTHY97cb1teM9TUzXd+WLVuOZuZIt8tZ1e2MEfEO4IXMPBoRo90uZ6bM3AfsAxgZGcnR0c4WPT4+TqfzdGvnngc6nmf35jPceazrzd53va7v+E2jPVsWDPb97Yb1NWN9zfSqviafAG8F3hkR1wJvBP4u8DFgTUSsyswzwHpgsvSfBDYAJyJiFbAa+G6D9UuS+qDrawyZ+cHMXJ+ZG4EbgS9m5k3Aw8C7SrcdwH1l+GAZp0z/YjY5jyVJ6ot+fI/hduADETEBXADsL+37gQtK+weAPX1YtySpoZ6cTM7McWC8DD8PXDFLn78CfqkX65Mk9Y/ffJYkVQwGSVJl+d43qdeEjR3e0nt873V9qkTSYnnEIEmqGAySpIrBIEmqGAySpIrBIEmqGAySpIrBIEmqGAySpIrBIEmqGAySpIrBIEmq+KykNp0+10eSXos8YpAkVQwGSVLFYJAkVQwGSVLFYJAkVbwrSVpAJ3er7d58htH+lSINhMGgFa2bW4z970Ol+XkqSZJU6ToYImJDRDwcEU9HxFMRcVtpPz8iDkXEs+X32tIeEfHxiJiIiCci4rJevQhJUu80OWI4A+zOzEuAq4BbIuISYA9wODM3AYfLOMA1wKbyswv4RIN1S5L6pOtgyMyTmflYGf5L4BlgHbAdOFC6HQCuL8PbgU9lyxFgTURc1O36JUn90ZNrDBGxEfhZ4BFgODNPlknfBobL8DrgW22znShtkqRlJDKz2QIihoD/CXw4Mz8bEd/LzDVt01/KzLURcT+wNzO/XNoPA7dn5qMzlreL1qkmhoeHLx8bG+uonqmpKYaGhrp6LccmT3c1XyeGz4VTr/R9NV1b6vo2r1s97/SZ728379lC65ipk3UMnwt/7/zOlj9ITf4+BsH6mpmub8uWLUczc6Tb5TS6XTUi3gD8MXB3Zn62NJ+KiIsy82Q5VfRCaZ8ENrTNvr60VTJzH7APYGRkJEdHRzuqaXx8nE7nmbZzAE9X3b35DHceW753CS91fcdvGp13+sz3t5v3bKF1zNTJOnZvPsMNXe5/g9Dk72MQrK+ZXtXX5K6kAPYDz2Tm77ZNOgjsKMM7gPva2t9b7k66CjjddspJkrRMNPmn4VuB9wDHIuLx0vZvgL3AZyLiZuCbwA1l2oPAtcAE8DLwvgbrliT1SdfBUK4VxByTt87SP4Fbul2fJGkwlu/Jbr0uLfSIi92bzwzkWpD0euYjMSRJFYNBklQxGCRJFYNBklQxGCRJFYNBklQxGCRJFb/HIK0wnf53pv5XpuqURwySpIrBIEmqGAySpIrBIEmqePFZrzudXryVXm88YpAkVQwGSVLFYJAkVbzGIGng5rrOM9d/xOSX9AbLIwZJUsVgkCRVDAZJUsVrDNJrXPv5/LnO4c/U6Tn9fn83xAcHDtZrOhj8IpMkde41HQzSSuA/YLTcDDwYImIb8DHgLOAPMnPvoGuQND/DamGv5dNbAw2GiDgL+H3gnwEngK9ExMHMfHqQdUj95Ifq0uvmPVhJH9z9NugjhiuAicx8HiAixoDtgMEgaUktJkwWe/G+2+W3W8qgiswc3Moi3gVsy8xfKePvAa7MzFvb+uwCdpXRtwBf73A1FwLf6UG5/WJ9zVhfM9bXzEqp7x9m5o93u5Bld/E5M/cB+7qdPyIezcyRHpbUU9bXjPU1Y33NvF7qG/QX3CaBDW3j60ubJGmZGHQwfAXYFBEXR8TZwI3AwQHXIEmax0BPJWXmmYi4Ffg8rdtV78rMp3q8mq5PQw2I9TVjfc1YXzOvi/oGevFZkrT8+RA9SVLFYJAkVVZsMETEtoj4ekRMRMSeWaafExH3lOmPRMTGAda2ISIejoinI+KpiLhtlj6jEXE6Ih4vP785qPrK+o9HxLGy7kdnmR4R8fGy/Z6IiMsGWNtb2rbL4xHx/Yj4tRl9Brr9IuKuiHghIp5sazs/Ig5FxLPl99o55t1R+jwbETsGWN/vRMTXyvv3uYhYM8e88+4LfazvtyJisu09vHaOeef9W+9jffe01XY8Ih6fY95BbL9ZP1P6tg9m5or7oXXh+jngzcDZwFeBS2b0+ZfAfy3DNwL3DLC+i4DLyvCbgG/MUt8ocP8SbsPjwIXzTL8WeAgI4CrgkSV8r79N6ws7S7b9gLcBlwFPtrX9R2BPGd4DfGSW+c4Hni+/15bhtQOq72pgVRn+yGz1LWZf6GN9vwX8+iLe/3n/1vtV34zpdwK/uYTbb9bPlH7tgyv1iOFHj9bIzB8C04/WaLcdOFCG7wW2RkQMorjMPJmZj5XhvwSeAdYNYt09tB34VLYcAdZExEVLUMdW4LnM/OYSrPtHMvNLwIszmtv3sQPA9bPM+nbgUGa+mJkvAYeAbYOoLzO/kJlnyugRWt8bWhJzbL/FWMzfemPz1Vc+N24APt3r9S7WPJ8pfdkHV2owrAO+1TZ+gld/8P6oT/njOA1cMJDq2pRTWD8LPDLL5J+LiK9GxEMR8VODrYwEvhARR6P1GJKZFrONB+FG5v6DXMrtBzCcmSfL8LeB4Vn6LJft+Mu0jgBns9C+0E+3llNdd81xGmQ5bL9/CpzKzGfnmD7Q7TfjM6Uv++BKDYYVISKGgD8Gfi0zvz9j8mO0To/8NPB7wP8YcHk/n5mXAdcAt0TE2wa8/gVF60uQ7wT+aJbJS739Ktk6Zl+W935HxIeAM8Ddc3RZqn3hE8A/An4GOEnrdM1y9G7mP1oY2Pab7zOll/vgSg2GxTxa40d9ImIVsBr47kCqa63zDbTewLsz87Mzp2fm9zNzqgw/CLwhIi4cVH2ZOVl+vwB8jtYhe7vl8PiSa4DHMvPUzAlLvf2KU9On18rvF2bps6TbMSJ2Au8AbiofHK+yiH2hLzLzVGb+TWb+P+C/zbHepd5+q4B/AdwzV59Bbb85PlP6sg+u1GBYzKM1DgLTV9/fBXxxrj+MXivnJPcDz2Tm787R5+9PX/OIiCtovRcDCa6IOC8i3jQ9TOsi5ZMzuh0E3hstVwGn2w5ZB2XOf6kt5fZr076P7QDum6XP54GrI2JtOVVydWnru2j9p1i/AbwzM1+eo89i9oV+1dd+zeqfz7HepX6Mzi8CX8vME7NNHNT2m+czpT/7YD+vpPfzh9ZdM9+gdcfCh0rbb9P6IwB4I61TEBPAnwNvHmBtP0/rkO4J4PHycy3wfuD9pc+twFO07rI4AvyTAdb35rLer5Yaprdfe31B6z9Veg44BowM+P09j9YH/eq2tiXbfrQC6iTw17TO0d5M65rVYeBZ4M+A80vfEVr/O+H0vL9c9sMJ4H0DrG+C1rnl6X1w+i69fwA8ON++MKD6/nvZt56g9QF30cz6yvir/tYHUV9p/+T0PtfWdym231yfKX3ZB30khiSpslJPJUmS+sRgkCRVDAZJUsVgkCRVDAZJUsVgkCRVDAZJUuX/A7zU+O5qfyxrAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_contracts[df_contracts.annual_salary < 20].annual_salary.hist(bins=25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAATlElEQVR4nO3df6zd9X3f8edrEEjKzTCM1iKGzVRyKhG8ZuEWkNpu14kKhvwBUasIygDnh1xNMLUa3eK0q8JKM3lVaKb8KJojrJKF5Yq1SbDAKXKseizVWMCMYgxhOMHJuEP2UlNTJyyt0/f+OF9bp+xe3+vre+45Pp/nQzo63/P5fs73+32f7zmv8z3f7/d+b6oKSVIb/s6wF0CStHwMfUlqiKEvSQ0x9CWpIYa+JDXkzGEvwIlccMEFtXr16oFN//vf/z7nnHPOwKY/bONeH4x/jeNeH4x/jcOob/fu3d+rqh+fbdxIh/7q1at58sknBzb9Xbt2MTU1NbDpD9u41wfjX+O41wfjX+Mw6kvynbnGuXtHkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaMtJ/kXu6Wr3pkaHNe//m9w5t3pJG37xb+kkuTvInSZ5LsjfJr3btdyWZSfJ0d7uu7zkfTbIvyQtJrulrX9+17UuyaTAlSZLmspAt/aPAnVX1VJK3AruT7OjGfbKqPtHfOcmlwI3AO4C3AV9L8vZu9GeBXwBeBp5Isq2qnluKQiRJ85s39KvqFeCVbvgvkzwPrDrBU64Hpqvqh8BLSfYBV3Tj9lXVtwGSTHd9DX1JWiY5mX+MnmQ18BhwGfAvgA3Aa8CT9H4NvJrkM8DjVfWF7jn3AV/tJrG+qj7ctd8CXFlVd7xhHhuBjQArV668fHp6etHFzefIkSNMTEws+XT3zBxe8mku1NpV5x4fHlR9o2Tcaxz3+mD8axxGfevWrdtdVZOzjVvwgdwkE8AfAb9WVa8luRe4G6ju/h7gg6e6sFW1BdgCMDk5WYO8JOmgLnm6YZgHcm+eOj487peshfGvcdzrg/GvcdTqW1DoJ3kTvcB/oKq+BFBVB/rGfw54uHs4A1zc9/SLujZO0C5JWgYLOXsnwH3A81X1e33tF/Z1ex/wbDe8DbgxydlJLgHWAN8AngDWJLkkyVn0DvZuW5oyJEkLsZAt/Z8FbgH2JHm6a/sN4KYk76S3e2c/8CsAVbU3yYP0DtAeBW6vqh8BJLkDeBQ4A9haVXuXrBJJ0rwWcvbO14HMMmr7CZ7zceDjs7RvP9HzJEmD5WUYJKkhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ2ZN/STXJzkT5I8l2Rvkl/t2s9PsiPJi939eV17knwqyb4kzyR5V9+0buv6v5jktsGVJUmazZkL6HMUuLOqnkryVmB3kh3ABmBnVW1OsgnYBHwEuBZY092uBO4FrkxyPvAxYBKobjrbqurVpS5K7dgzc5gNmx4Zyrz3b37vUOYrnYp5t/Sr6pWqeqob/kvgeWAVcD1wf9ftfuCGbvh64PPV8ziwIsmFwDXAjqo61AX9DmD9UhYjSTqxVNXCOyergceAy4DvVtWKrj3Aq1W1IsnDwOaq+no3bie9XwBTwJur6ne69t8CXq+qT7xhHhuBjQArV668fHp6+lTqO6EjR44wMTGx5NPdM3N4yae5UGtXnXt8eFD1jZKDhw5z4PXhzLv/tR6UFtbhuNc4jPrWrVu3u6omZxu3kN07ACSZAP4I+LWqeq2X8z1VVUkW/u1xAlW1BdgCMDk5WVNTU0sx2Vnt2rWLQUx/WLsbAPbfPHV8eFD1jZJPP/AQ9+xZ8Nt4SfW/1oPSwjoc9xpHrb4Fnb2T5E30Av+BqvpS13yg221Dd3+wa58BLu57+kVd21ztkqRlspCzdwLcBzxfVb/XN2obcOwMnNuAh/rab+3O4rkKOFxVrwCPAlcnOa870+fqrk2StEwW8rv4Z4FbgD1Jnu7afgPYDDyY5EPAd4D3d+O2A9cB+4AfAB8AqKpDSe4Gnuj6/XZVHVqKIiRJCzNv6HcHZDPH6PfM0r+A2+eY1lZg68ksoCRp6fgXuZLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqyHD++4SkRVs9zH/S4/8FPu25pS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSHzhn6SrUkOJnm2r+2uJDNJnu5u1/WN+2iSfUleSHJNX/v6rm1fkk1LX4okaT4L2dL/A2D9LO2frKp3drftAEkuBW4E3tE95/eTnJHkDOCzwLXApcBNXV9J0jKa9x+jV9VjSVYvcHrXA9NV9UPgpST7gCu6cfuq6tsASaa7vs+d/CJLkhYrVTV/p17oP1xVl3WP7wI2AK8BTwJ3VtWrST4DPF5VX+j63Qd8tZvM+qr6cNd+C3BlVd0xy7w2AhsBVq5cefn09PSp1HdCR44cYWJiYsmnu2fm8JJPc6HWrjr3+PCg6hslBw8d5sDrw5l3/2s9KLOtw1F5fy2VcX+fDqO+devW7a6qydnGzbulP4d7gbuB6u7vAT64yGn9LVW1BdgCMDk5WVNTU0sx2Vnt2rWLQUx/w6ZHlnyaC7X/5qnjw4Oqb5R8+oGHuGfPYt/Gp6b/tR6U2dbhqLy/lsq4v09Hrb5FfVqq6sCx4SSfAx7uHs4AF/d1vahr4wTtkqRlsqhTNpNc2PfwfcCxM3u2ATcmOTvJJcAa4BvAE8CaJJckOYvewd5ti19sSdJizLuln+SLwBRwQZKXgY8BU0neSW/3zn7gVwCqam+SB+kdoD0K3F5VP+qmcwfwKHAGsLWq9i51MZKkE1vI2Ts3zdJ83wn6fxz4+Czt24HtJ7V0kqQl5V/kSlJDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIfP+Y3RJOmb1pkeWfJp3rj3Khnmmu3/ze5d8vq1yS1+SGjLWW/rzbZUsZAtDksaJW/qS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktSQeUM/ydYkB5M829d2fpIdSV7s7s/r2pPkU0n2JXkmybv6nnNb1//FJLcNphxJ0oksZEv/D4D1b2jbBOysqjXAzu4xwLXAmu62EbgXel8SwMeAK4ErgI8d+6KQJC2feUO/qh4DDr2h+Xrg/m74fuCGvvbPV8/jwIokFwLXADuq6lBVvQrs4P//IpEkDViqav5OyWrg4aq6rHv8F1W1ohsO8GpVrUjyMLC5qr7ejdsJfASYAt5cVb/Ttf8W8HpVfWKWeW2k9yuBlStXXj49Pb3o4vbMHD7h+JVvgQOvL3ryI2ntqnOPDx85coSJiYkhLs3gHTx0eGjrsP+1HpTZ1uF87+vTzUI+h8vxWg/KMD6H69at211Vk7ONO+WrbFZVJZn/m2Ph09sCbAGYnJysqampRU9rvito3rn2KPfsGa8Lje6/eer48K5duziV1+908OkHHhraOux/rQdltnU4bleGXcjncDle60EZtc/hYs/eOdDttqG7P9i1zwAX9/W7qGubq12StIwWG/rbgGNn4NwGPNTXfmt3Fs9VwOGqegV4FLg6yXndAdyruzZJ0jKa93dxki/S2yd/QZKX6Z2Fsxl4MMmHgO8A7++6bweuA/YBPwA+AFBVh5LcDTzR9fvtqnrjwWFJ0oDNG/pVddMco94zS98Cbp9jOluBrSe1dJKkJeVf5EpSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIacOewFkKT5rN70yNDmvX/ze4c270FwS1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUkFMK/ST7k+xJ8nSSJ7u285PsSPJid39e154kn0qyL8kzSd61FAVIkhZuKbb011XVO6tqsnu8CdhZVWuAnd1jgGuBNd1tI3DvEsxbknQSBrF753rg/m74fuCGvvbPV8/jwIokFw5g/pKkOaSqFv/k5CXgVaCA/1BVW5L8RVWt6MYHeLWqViR5GNhcVV/vxu0EPlJVT75hmhvp/RJg5cqVl09PTy96+fbMHD7h+JVvgQOvL3ryI2ntqnOPDx85coSJiYkhLs3gHTx0eGjrsP+1HpTZ1uF87+vTzah/Dk91PQ/jc7hu3brdfXtf/pZTveDaz1XVTJKfAHYk+Wb/yKqqJCf1rVJVW4AtAJOTkzU1NbXohdswz0Wa7lx7lHv2jNc15/bfPHV8eNeuXZzK63c6+PQDDw1tHfa/1oMy2zqc7319uhn1z+GprudR+xye0u6dqprp7g8CXwauAA4c223T3R/sus8AF/c9/aKuTZK0TBYd+knOSfLWY8PA1cCzwDbgtq7bbcBD3fA24NbuLJ6rgMNV9cqil1ySdNJO5TfVSuDLvd32nAn8p6r64yRPAA8m+RDwHeD9Xf/twHXAPuAHwAdOYd6SpEVYdOhX1beBn56l/c+B98zSXsDti52fJOnU+Re5ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqyKL/MbrUutWbHhn4PO5ce5QNyzAftcMtfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSHLHvpJ1id5Icm+JJuWe/6S1LJlDf0kZwCfBa4FLgVuSnLpci6DJLVsubf0rwD2VdW3q+qvgGng+mVeBklqVqpq+WaW/BKwvqo+3D2+Bbiyqu7o67MR2Ng9/CnghQEu0gXA9wY4/WEb9/pg/Gsc9/pg/GscRn3/oKp+fLYRI3c9/araAmxZjnklebKqJpdjXsMw7vXB+Nc47vXB+Nc4avUt9+6dGeDivscXdW2SpGWw3KH/BLAmySVJzgJuBLYt8zJIUrOWdfdOVR1NcgfwKHAGsLWq9i7nMrzBsuxGGqJxrw/Gv8Zxrw/Gv8aRqm9ZD+RKkobLv8iVpIYY+pLUkLEL/SRbkxxM8mxf211JZpI83d2u69rflOT+JHuSPJ/ko33PGcnLRcxWX9f+z5N8M8neJL/b1/7RroYXklzT137a15fkF5Ls7tbf7iTv7ut/ede+L8mnkmS5a5nLya7DbtzfT3Ikya/3tZ3267Br/4dJ/lvXvifJm7v2kVyHJ/keHb2MqaqxugH/GHgX8Gxf213Ar8/S95eB6W74x4D9wGp6B5m/BfwkcBbwZ8Clw67tBPWtA74GnN09/onu/tJu2c8GLulqOmOM6vtHwNu64cuAmb7nfAO4CgjwVeDaYde2mBr7xv8h8J+PvY/HaB2eCTwD/HT3+O8BZ4zyOjzJ+kYuY8ZuS7+qHgMOLbQ7cE6SM4G3AH8FvMYIXy5ijvr+GbC5qn7Y9TnYtV9P7w33w6p6CdhHr7axqK+q/kdV/e+uz17gLUnOTnIh8Her6vHqfdo+D9ywLAUswEmuQ5LcALxEr8ZjxmIdAlcDz1TVn3Xtf15VPxrldXiS9Y1cxoxd6J/AHUme6X6ande1/SHwfeAV4LvAJ6rqELAK+F99z325axtVbwd+Psl/T/JfkvxM1z5XHeNSX79fBJ7qPnSr6NV0zKjXB3PUmGQC+Ajwb97Qf1zW4duBSvJokqeS/Kuu/XRbh3PVN3IZM3KXYRiQe4G76X3r3g3cA3yQ3rftj4C3AecB/zXJ14a1kKfgTOB8ej+FfwZ4MMlPDneRltSs9XVbgCR5B/Dv6G01nq7mWod3AZ+sqiMjskt7seaq70zg57q2HwA7k+wGDg9rQRdprvpGLmOaCP2qOnBsOMnngIe7h78M/HFV/TVwMMmfApP0voFPp8tFvAx8qQvBbyT5G3oXeTrRZS/Gob7/k+Qi4MvArVX1ra7/DL2ajhn1+mDuGq8Efqk7MLgC+Jsk/xfYzXisw5eBx6rqewBJttPbX/4FTq91OFd9I5cxTeze6fYPHvM+4NhR9+8C7+76nEPvW/qbnH6Xi/gKvQNJJHk7vQND36O3zDd2+7kvAdbQOzg2FvUlWQE8Amyqqj891rmqXgFeS3JVd8bHrcBDy73QJ+krzFJjVf18Va2uqtXAvwf+bVV9hjFZh/T+On9tkh/r9nv/E+C503AdfoXZ6xu9jBnmUfBB3IAv0tt/9tf0vn0/BPxHYA+9swS2ARd2fSfonRGxF3gO+Jd907kO+J/0jrD/5rDrmqe+s+htGT0LPAW8u6//b3Y1vEDf2Q/jUB/wr+ntL32673bsrInJrv+3gM/Q/fX5KNxOdh32Pe8u+s5CG4d12PX/p91n8Fngd/vaR3IdnuR7dOQyxsswSFJDmti9I0nqMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQ/4fkkQcB3xhV/gAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's plot the distribution of contract start dates\n",
    "df_contracts.startY.hist(bins=10)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Question 2: calculate proportions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0    0.987293\n",
       "0.0    0.012707\n",
       "Name: a_gender, dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_contracts.a_gender.value_counts(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9401.0"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_contracts.a_gender.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9401.0"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_contracts.a_gender.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum([True,False,False,True,True])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.026105873821609893"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1-(df_contracts.a_gender.sum()/df_contracts.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.6105873821609893"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# proportion of female apprentices\n",
    "(1-(df_contracts.a_gender.sum()/df_contracts.shape[0]))*100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.3723194861701047"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# proportion of female masters\n",
    "(1-(df_contracts.m_gender.sum()/df_contracts.shape[0]))*100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "73.10924369747899"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# prop female apprentices with male master\n",
    "(df_contracts[(df_contracts.a_gender == 0) & (df_contracts.startY < 1800)].m_gender.sum()\\\n",
    "    /df_contracts[(df_contracts.a_gender == 0) & (df_contracts.startY < 1800)].shape[0])*100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "98.10528582193993 %\n"
     ]
    }
   ],
   "source": [
    "# prop male apprentices with male master\n",
    "print((df_contracts[(df_contracts.a_gender == 1) & (df_contracts.startY < 1800)].m_gender.sum()\\\n",
    "    /df_contracts[(df_contracts.a_gender == 1) & (df_contracts.startY < 1800)].shape[0])*100,\"%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>m_gender</th>\n",
       "      <th>0.0</th>\n",
       "      <th>1.0</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>a_gender</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0.0</th>\n",
       "      <td>0.00339</td>\n",
       "      <td>0.00933</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1.0</th>\n",
       "      <td>0.00997</td>\n",
       "      <td>0.97730</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "m_gender      0.0      1.0\n",
       "a_gender                  \n",
       "0.0       0.00339  0.00933\n",
       "1.0       0.00997  0.97730"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This is another elegant solution to obtain the solution\n",
    "pd.crosstab(df_contracts['a_gender'],df_contracts['m_gender'], normalize=True).round(5)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ⏰ ✏️ Exercise\n",
    "\n",
    "* Plot the distribution (histogram) of contract length\n",
    "* How likely it is for a female apprentice to have a female master? And for a male apprentice?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Looking at empirical distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAT5ElEQVR4nO3df4xd5Z3f8fenJslGsFnIQkdem9REdVJB2HWXEaHa7WrYdBMD0UKqVYpFA07SdaIFKVFdrcx2JdJESLTdbLZRtqycYEG0KQ4Km2AF0qyXZkQqlQScUMyPUAwxwq5jKyELOySiNfn2j3umc2PGM3Pnju+Yed4v6WrOec45z3nuF/yZM+ece26qCklSG/7ecg9AkjQ6hr4kNcTQl6SGGPqS1BBDX5IacspyD2A+Z555Zq1bt27g7V588UVOPfXUpR/Qq4x16LEOM6xFz0quw549e35YVWfNtuykD/1169bx4IMPDrzd5OQkExMTSz+gVxnr0GMdZliLnpVchyTPHG+Zp3ckqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0JakhJ/0ncpfLum13z7l8/02XjWgkkrR0PNKXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDZk39JPsSHIkySN9bV9M8lD32p/koa59XZKf9i37i75tLkiyN8m+JJ9OkhPyjiRJx7WQ+/RvBT4DfH66oar+xfR0kk8Cz/et/1RVbZiln5uB3we+BdwDbAS+NvCIJUmLNu+RflXdBzw327LuaP29wO1z9ZFkNfCGqrq/qoreL5ArBh6tJGkow34i958Ch6vqyb62c5J8F3gB+OOq+iawBjjQt86Brm1WSbYAWwDGxsaYnJwceGBTU1OL2m7a1vOPzrl8mL5Hadg6rBTWYYa16Gm1DsOG/iZ+/ij/EPCmqvpRkguAryQ5b9BOq2o7sB1gfHy8FvPlxcN+6fHm+R7DcNXi+x6llfzlz4OwDjOsRU+rdVh06Cc5BfjnwAXTbVX1EvBSN70nyVPAW4CDwNq+zdd2bZKkERrmls1/Bnyvqv7/aZskZyVZ1U2/GVgPPF1Vh4AXklzUXQe4GrhriH1LkhZhIbds3g78D+CtSQ4k+WC36EpeeQH3t4CHu1s4vwR8uKqmLwL/AfA5YB/wFN65I0kjN+/pnaradJz2zbO03QnceZz1HwTeNuD4JElLyE/kSlJDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ1ZyBej70hyJMkjfW0fS3IwyUPd69K+Zdcn2ZfkiSTv6mvf2LXtS7Jt6d+KJGk+CznSvxXYOEv7p6pqQ/e6ByDJucCVwHndNv85yaokq4A/By4BzgU2detKkkbolPlWqKr7kqxbYH+XAzur6iXg+0n2ARd2y/ZV1dMASXZ26z42+JAlSYs1b+jP4bokVwMPAlur6sfAGuD+vnUOdG0Azx7T/vbjdZxkC7AFYGxsjMnJyYEHNzU1tajtpm09/+icy4fpe5SGrcNKYR1mWIueVuuw2NC/GfgEUN3PTwIfWKpBVdV2YDvA+Ph4TUxMDNzH5OQki9lu2uZtd8+5fP9Vi+97lIatw0phHWZYi55W67Co0K+qw9PTST4LfLWbPQic3bfq2q6NOdolSSOyqFs2k6zum30PMH1nzy7gyiSvS3IOsB74NvAAsD7JOUleS+9i767FD1uStBjzHuknuR2YAM5McgC4AZhIsoHe6Z39wIcAqurRJHfQu0B7FLi2ql7u+rkO+DqwCthRVY8u9ZuRJM1tIXfvbJql+ZY51r8RuHGW9nuAewYanSRpSfmJXElqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDZn36xI1u3Xb7j7usv03XTbCkUjSws17pJ9kR5IjSR7pa/uPSb6X5OEkX05yete+LslPkzzUvf6ib5sLkuxNsi/Jp5PkhLwjSdJxLeT0zq3AxmPadgNvq6pfBf4XcH3fsqeqakP3+nBf+83A7wPru9exfUqSTrB5Q7+q7gOeO6btr6vqaDd7P7B2rj6SrAbeUFX3V1UBnweuWNSIJUmLthTn9D8AfLFv/pwk3wVeAP64qr4JrAEO9K1zoGubVZItwBaAsbExJicnBx7U1NTUorabtvX8o/OvdBzD7HepDVuHlcI6zLAWPa3WYajQT/JvgaPAF7qmQ8CbqupHSS4AvpLkvEH7rartwHaA8fHxmpiYGHhsk5OTLGa7aZvnuFA7n/1XLX6/S23YOqwU1mGGtehptQ6LDv0km4F3A+/oTtlQVS8BL3XTe5I8BbwFOMjPnwJa27VJkkZoUffpJ9kI/CHwu1X1k772s5Ks6qbfTO+C7dNVdQh4IclF3V07VwN3DT16SdJA5j3ST3I7MAGcmeQAcAO9u3VeB+zu7ry8v7tT57eAjyf5v8DPgA9X1fRF4D+gdyfQ64GvdS9J0gjNG/pVtWmW5luOs+6dwJ3HWfYg8LaBRidJWlI+hkGSGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQ5r9jty5vuNWklYqj/QlqSGGviQ1pNnTOyfSfKeO9t902YhGIkk/zyN9SWqIoS9JDTH0Jakhhr4kNcTQl6SGLCj0k+xIciTJI31tb0yyO8mT3c8zuvYk+XSSfUkeTvLrfdtc063/ZJJrlv7tSJLmstAj/VuBjce0bQPurar1wL3dPMAlwPrutQW4GXq/JIAbgLcDFwI3TP+ikCSNxoJCv6ruA547pvly4LZu+jbgir72z1fP/cDpSVYD7wJ2V9VzVfVjYDev/EUiSTqBhvlw1lhVHeqmfwCMddNrgGf71jvQtR2v/RWSbKH3VwJjY2NMTk4OPLipqak5t9t6/tGB+1wqi3k/izVfHVphHWZYi55W67Akn8itqkpSS9FX1992YDvA+Ph4TUxMDNzH5OQkc223eTkfuLb3xeMuWupP685Xh1ZYhxnWoqfVOgxz987h7rQN3c8jXftB4Oy+9dZ2bcdrlySNyDChvwuYvgPnGuCuvvaru7t4LgKe704DfR14Z5Izugu47+zaJEkjsqDTO0luByaAM5McoHcXzk3AHUk+CDwDvLdb/R7gUmAf8BPg/QBV9VySTwAPdOt9vKqOvTgsSTqBFhT6VbXpOIveMcu6BVx7nH52ADsWPDpJ0pLyE7mS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQxYd+knemuShvtcLST6a5GNJDva1X9q3zfVJ9iV5Ism7luYtSJIWakFfjD6bqnoC2ACQZBVwEPgy8H7gU1X1J/3rJzkXuBI4D/gV4G+SvKWqXl7sGCRJg1mq0zvvAJ6qqmfmWOdyYGdVvVRV3wf2ARcu0f4lSQuwVKF/JXB73/x1SR5OsiPJGV3bGuDZvnUOdG2SpBFJVQ3XQfJa4H8D51XV4SRjwA+BAj4BrK6qDyT5DHB/Vf1lt90twNeq6kuz9LkF2AIwNjZ2wc6dOwce19TUFKeddtpxl+89+PzAfY7C+Wt+aUn7m68OrbAOM6xFz0quw8UXX7ynqsZnW7boc/p9LgG+U1WHAaZ/AiT5LPDVbvYgcHbfdmu7tleoqu3AdoDx8fGamJgYeFCTk5PMtd3mbXcP3Oco7L9qYkn7m68OrbAOM6xFT6t1WIrTO5voO7WTZHXfsvcAj3TTu4Ark7wuyTnAeuDbS7B/SdICDXWkn+RU4HeAD/U1/4ckG+id3tk/vayqHk1yB/AYcBS41jt3JGm0hgr9qnoR+OVj2t43x/o3AjcOs09J0uL5iVxJaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIUOHfpL9SfYmeSjJg13bG5PsTvJk9/OMrj1JPp1kX5KHk/z6sPuXJC3cUh3pX1xVG6pqvJvfBtxbVeuBe7t5gEuA9d1rC3DzEu1fkrQAJ+r0zuXAbd30bcAVfe2fr577gdOTrD5BY5AkHWMpQr+Av06yJ8mWrm2sqg510z8AxrrpNcCzfdse6NokSSNwyhL08ZtVdTDJ3wd2J/le/8KqqiQ1SIfdL48tAGNjY0xOTg48qKmpqTm323r+0YH7HIXFvNe5zFeHVliHGdaip9U6DB36VXWw+3kkyZeBC4HDSVZX1aHu9M2RbvWDwNl9m6/t2o7tczuwHWB8fLwmJiYGHtfk5CRzbbd5290D9zkK+6+aWNL+5qtDK6zDDGvR02odhjq9k+TUJL84PQ28E3gE2AVc0612DXBXN70LuLq7i+ci4Pm+00CSpBNs2CP9MeDLSab7+i9V9V+TPADckeSDwDPAe7v17wEuBfYBPwHeP+T+JUkDGCr0q+pp4Ndmaf8R8I5Z2gu4dph9SpIWz0/kSlJDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWrIUjxaWUto3TxP/9x/02UjGomklcgjfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JasiiQz/J2Um+keSxJI8m+UjX/rEkB5M81L0u7dvm+iT7kjyR5F1L8QYkSQs3zGMYjgJbq+o7SX4R2JNkd7fsU1X1J/0rJzkXuBI4D/gV4G+SvKWqXh5iDJKkASz6SL+qDlXVd7rpvwMeB9bMscnlwM6qeqmqvg/sAy5c7P4lSYNLVQ3fSbIOuA94G/Cvgc3AC8CD9P4a+HGSzwD3V9VfdtvcAnytqr40S39bgC0AY2NjF+zcuXPgMU1NTXHaaacdd/neg88P3OfJ4Pw1vzTn8mPf19jr4fBPF7btSjbf/w8tsRY9K7kOF1988Z6qGp9t2dBP2UxyGnAn8NGqeiHJzcAngOp+fhL4wCB9VtV2YDvA+Ph4TUxMDDyuyclJ5tpu8zxPszxZ7b9qYs7lx76vrecf5ZN7T1nQtivZfP8/tMRa9LRah6Hu3knyGnqB/4Wq+iuAqjpcVS9X1c+AzzJzCucgcHbf5mu7NknSiAxz906AW4DHq+pP+9pX9632HuCRbnoXcGWS1yU5B1gPfHux+5ckDW6Y0zu/AbwP2Jvkoa7tj4BNSTbQO72zH/gQQFU9muQO4DF6d/5c6507kjRaiw79qvrvQGZZdM8c29wI3LjYfUqShuMnciWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDhn4Mg1aOdXM8mmL/TZeNcCSSTpQVHfpzhdir1Up8T5JGx9M7ktQQQ1+SGmLoS1JDVvQ5fY3OfNcavBAsnRw80pekhnikrxVt+i+QrecfnfXb0vwLRK3xSF+SGmLoS1JDDH1Jaojn9KXG7D34/KzXN8BrHC0w9DUSLT7Xp8X3fLKa7b/F9MX91v5bjDz0k2wE/hOwCvhcVd006jG0apjn9iznM39O1s8A+BwkvRqNNPSTrAL+HPgd4ADwQJJdVfXYKMehk8vJGp4n67ikYYz6SP9CYF9VPQ2QZCdwOWDoa9FejX/BnMj9DvOXzzDjmm+/J2vfy2k5TgGmqk5Ix7PuLPk9YGNV/atu/n3A26vqumPW2wJs6WbfCjyxiN2dCfxwiOGuFNahxzrMsBY9K7kO/6CqzpptwUl5IbeqtgPbh+kjyYNVNb5EQ3rVsg491mGGtehptQ6jvk//IHB23/zark2SNAKjDv0HgPVJzknyWuBKYNeIxyBJzRrp6Z2qOprkOuDr9G7Z3FFVj56g3Q11emgFsQ491mGGtehpsg4jvZArSVpePntHkhpi6EtSQ1Zc6CfZmOSJJPuSbFvu8YxSkh1JjiR5pK/tjUl2J3my+3nGco5xFJKcneQbSR5L8miSj3TtTdUiyS8k+XaS/9nV4d917eck+Vb3b+SL3U0VTUiyKsl3k3y1m2+uFisq9Pse83AJcC6wKcm5yzuqkboV2HhM2zbg3qpaD9zbza90R4GtVXUucBFwbff/QWu1eAn47ar6NWADsDHJRcC/Bz5VVf8Q+DHwweUb4sh9BHi8b765Wqyo0KfvMQ9V9X+A6cc8NKGq7gOeO6b5cuC2bvo24IpRjmk5VNWhqvpON/139P6Rr6GxWlTPVDf7mu5VwG8DX+raV3wdpiVZC1wGfK6bDw3WYqWF/hrg2b75A11by8aq6lA3/QNgbDkHM2pJ1gH/GPgWDdaiO53xEHAE2A08BfxtVR3tVmnp38ifAX8I/Kyb/2UarMVKC33NoXr35zZzj26S04A7gY9W1Qv9y1qpRVW9XFUb6H36/ULgHy3viJZHkncDR6pqz3KPZbmdlM/eGYKPeXilw0lWV9WhJKvpHfGteEleQy/wv1BVf9U1N1kLgKr62yTfAP4JcHqSU7oj3Fb+jfwG8LtJLgV+AXgDve/1aK4WK+1I38c8vNIu4Jpu+hrgrmUcy0h052pvAR6vqj/tW9RULZKcleT0bvr19L7H4nHgG8Dvdaut+DoAVNX1VbW2qtbRy4X/VlVX0WAtVtwncrvf5H/GzGMeblzeEY1OktuBCXqPjD0M3AB8BbgDeBPwDPDeqjr2Yu+KkuQ3gW8Ce5k5f/tH9M7rN1OLJL9K7+LkKnoHeHdU1ceTvJneTQ5vBL4L/Muqemn5RjpaSSaAf1NV726xFisu9CVJx7fSTu9IkuZg6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SG/D/CpIrwm9HJbAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_contracts[df_contracts.annual_salary < 50].annual_salary.hist(bins=40)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD4CAYAAAAEhuazAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAQoUlEQVR4nO3df4xlZX3H8fenoNWwhoVgJwRohzabNpRtKUyApqaZrSku8AeYGCKhuljN+gckmvIHWxMD9UeyadS2tpZ2LRsxVaekatkALd1snFD+QNm1yPKjlq0uLZPtbuzC6qixWf32j3u2nZ3O751f9z7vVzK55z7nuec+35ydzz3znHPPpqqQJLXlp9Z6AJKk1Wf4S1KDDH9JapDhL0kNMvwlqUFnr/UA5nLBBRfU8PDwaW3f//73Oeecc9ZmQCvIuvrPoNY2qHXB4NY2va4DBw58p6reONdr1nX4Dw8Ps3///tPaxsfHGR0dXZsBrSDr6j+DWtug1gWDW9v0upK8NN9rnPaRpAYZ/pLUIMNfkhpk+EtSgwx/SWqQ4S9JDTL8JalBhr8kNcjwl6QGretv+EorYXjHI4vqf3jnjSs0EmnteOQvSQ0y/CWpQYa/JDXI8JekBhn+ktQgw1+SGmT4S1KDDH9JapDhL0kNMvwlqUGGvyQ1yPCXpAYZ/pLUIMNfkhpk+EtSg+YN/ySXJPlKkueTPJfkfV37+Un2Jnmxezyva0+STyY5lOSZJFdO2da2rv+LSbatXFmSpLks5Mj/JHBXVV0GXAvckeQyYAewr6o2Afu65wDXA5u6n+3AfdD7sADuAa4BrgbuOfWBIUlaXfOGf1Udqaqvd8vfA14ALgJuAh7ouj0A3Nwt3wR8tnqeBDYmuRB4C7C3qo5X1SvAXmDrchYjSVqYVNXCOyfDwOPA5cC/V9XGrj3AK1W1McnDwM6qeqJbtw+4GxgFXldVH+naPwj8sKo+Nu09ttP7i4GhoaGrxsbGThvD5OQkGzZsWHSh6511rZ6DEycW1X/zRefO2L4ea1sOg1oXDG5t0+vasmXLgaoames1C/4/fJNsAL4IvL+qvtvL+56qqiQL/xSZQ1XtAnYBjIyM1Ojo6Gnrx8fHmd42CKxr9dy+2P/D97bRGdvXY23LYVDrgsGtbSl1LehqnySvoRf8n6uqL3XNR7vpHLrHY137BHDJlJdf3LXN1i5JWmULudonwP3AC1X1iSmr9gCnrtjZBjw0pf2d3VU/1wInquoI8BhwXZLzuhO913VtkqRVtpBpn98A3gEcTPJ01/YBYCfwYJJ3Ay8Bt3TrHgVuAA4BPwDeBVBVx5N8GHiq6/ehqjq+HEVIkhZn3vDvTtxmltVvnqF/AXfMsq3dwO7FDFCStPz8hq8kNcjwl6QGGf6S1CDDX5IaZPhLUoMMf0lqkOEvSQ0y/CWpQYa/JDXI8JekBhn+ktQgw1+SGmT4S1KDDH9JapDhL0kNMvwlqUGGvyQ1yPCXpAYZ/pLUIMNfkhpk+EtSgwx/SWqQ4S9JDTL8JalBhr8kNcjwl6QGGf6S1CDDX5IaZPhLUoMMf0lqkOEvSQ0y/CWpQYa/JDXI8JekBhn+ktQgw1+SGmT4S1KDDH9JatC84Z9kd5JjSZ6d0nZvkokkT3c/N0xZ9/tJDiX5ZpK3TGnf2rUdSrJj+UuRJC3UQo78PwNsnaH9j6rqiu7nUYAklwFvB365e82fJzkryVnAp4DrgcuAW7u+kqQ1cPZ8Harq8STDC9zeTcBYVf0I+HaSQ8DV3bpDVfUtgCRjXd/nFz9kSdKZSlXN36kX/g9X1eXd83uB24HvAvuBu6rqlSR/BjxZVX/d9bsf+PtuM1ur6j1d+zuAa6rqzhneazuwHWBoaOiqsbGx09ZPTk6yYcOGRRe63lnX6jk4cWJR/TdfdO6M7euxtuUwqHXB4NY2va4tW7YcqKqRuV4z75H/LO4DPgxU9/hx4HeXuK3TVNUuYBfAyMhIjY6OnrZ+fHyc6W2DwLpWz+07HllU/8O3jc7Yvh5rWw6DWhcMbm1LqWtJ4V9VR08tJ/k08HD3dAK4ZErXi7s25miXJK2yJV3qmeTCKU/fCpy6EmgP8PYkP53kUmAT8DXgKWBTkkuTvJbeSeE9Sx+2JOlMzHvkn+QLwChwQZKXgXuA0SRX0Jv2OQy8F6CqnkvyIL0TuSeBO6rqx9127gQeA84CdlfVc8tdjCRpYRZytc+tMzTfP0f/jwIfnaH9UeDRRY1OkrQi/IavJDXI8JekBhn+ktQgw1+SGmT4S1KDDH9JapDhL0kNMvwlqUGGvyQ1yPCXpAYZ/pLUIMNfkhpk+EtSgwx/SWqQ4S9JDTL8JalBhr8kNWhJ/4G7tFKGdzyyqP6Hd964QiORBptH/pLUIMNfkhrktI/62mKniST1eOQvSQ0y/CWpQYa/JDXI8JekBhn+ktQgw1+SGmT4S1KDDH9JapDhL0kNMvwlqUGGvyQ1yPCXpAYZ/pLUIMNfkhpk+EtSgwx/SWqQ4S9JDZo3/JPsTnIsybNT2s5PsjfJi93jeV17knwyyaEkzyS5csprtnX9X0yybWXKkSQtxEKO/D8DbJ3WtgPYV1WbgH3dc4DrgU3dz3bgPuh9WAD3ANcAVwP3nPrAkCStvnnDv6oeB45Pa74JeKBbfgC4eUr7Z6vnSWBjkguBtwB7q+p4Vb0C7OX/f6BIklZJqmr+Tskw8HBVXd49f7WqNnbLAV6pqo1JHgZ2VtUT3bp9wN3AKPC6qvpI1/5B4IdV9bEZ3ms7vb8aGBoaumpsbOy09ZOTk2zYsGFJxa5n1tVzcOLECo5maTZfdO6M7e6z/jOotU2va8uWLQeqamSu15x9pm9aVZVk/k+QhW9vF7ALYGRkpEZHR09bPz4+zvS2QWBdPbfveGTlBrNEh28bnbHdfdZ/BrW2pdS11PA/muTCqjrSTesc69ongEum9Lu4a5ugd/Q/tX18ie8trarhWT6Q7tp8csYPq8M7b1zpIUlnbKmXeu4BTl2xsw14aEr7O7urfq4FTlTVEeAx4Lok53Uneq/r2iRJa2DeI/8kX6B31H5BkpfpXbWzE3gwybuBl4Bbuu6PAjcAh4AfAO8CqKrjST4MPNX1+1BVTT+JLElaJfOGf1XdOsuqN8/Qt4A7ZtnObmD3okYnSVoRZ3zCV9LpZjtHMBvPEWgteHsHSWqQ4S9JDTL8JalBzvlrRR2cOLEuv7gltc4jf0lqkOEvSQ0y/CWpQYa/JDXI8JekBhn+ktQgw1+SGmT4S1KDDH9JapDhL0kNMvwlqUGGvyQ1yPCXpAYZ/pLUIMNfkhpk+EtSgwx/SWqQ4S9JDTL8JalBhr8kNcjwl6QGGf6S1CDDX5IadPZaD0DS4gzveGRR/Q/vvHGFRqJ+5pG/JDXI8JekBhn+ktQgw1+SGmT4S1KDDH9JapDhL0kNMvwlqUGGvyQ16IzCP8nhJAeTPJ1kf9d2fpK9SV7sHs/r2pPkk0kOJXkmyZXLUYAkafGW48h/S1VdUVUj3fMdwL6q2gTs654DXA9s6n62A/ctw3tLkpZgJaZ9bgIe6JYfAG6e0v7Z6nkS2JjkwhV4f0nSPFJVS39x8m3gFaCAv6yqXUleraqN3foAr1TVxiQPAzur6olu3T7g7qraP22b2+n9ZcDQ0NBVY2Njp73n5OQkGzZsWPKY16tBrevY8RMc/eFaj2JlDL2eZalt80XnLqr/wYkTK7r9Qf23CINb2/S6tmzZcmDKbMyMzvSunm+qqokkPwPsTfIvU1dWVSVZ1KdLVe0CdgGMjIzU6OjoaevHx8eZ3jYIBrWuP/3cQ3z84GDePPauzSeXpbbDt40uqv/ti72r5yK3P6j/FmFwa1tKXWc07VNVE93jMeDLwNXA0VPTOd3jsa77BHDJlJdf3LVJklbZksM/yTlJ3nBqGbgOeBbYA2zrum0DHuqW9wDv7K76uRY4UVVHljxySdKSncnfrEPAl3vT+pwNfL6q/iHJU8CDSd4NvATc0vV/FLgBOAT8AHjXGby3JOkMLDn8q+pbwK/O0P5fwJtnaC/gjqW+nyRp+fgNX0lqkOEvSQ0y/CWpQYa/JDXI8JekBhn+ktSgwfzevdRHhhd5u4aV3v5dm08yujJD0Trikb8kNcjwl6QGGf6S1CDDX5IaZPhLUoMMf0lqkOEvSQ0y/CWpQYa/JDXI8JekBhn+ktQgw1+SGmT4S1KDDH9JapDhL0kNMvwlqUGGvyQ1yPCXpAYZ/pLUIMNfkhpk+EtSgwx/SWrQ2Ws9APWX4R2PLKr/XZtXaCCSzohH/pLUIMNfkhpk+EtSgwx/SWqQ4S9JDfJqn8Yt9uodSYPB8Jd0xhZ7EHF4540rNBItlOEvad3zw2X5rXr4J9kK/AlwFvBXVbVztccgaW5OBw6+VQ3/JGcBnwJ+G3gZeCrJnqp6fjXH0S/8BZS0Ulb7yP9q4FBVfQsgyRhwE2D4Sw1Z6QOb2bZ/1+aT3L5M773YqaX1NnWVqlrRNzjtzZK3AVur6j3d83cA11TVnVP6bAe2d09/EfjmtM1cAHxnFYa72qyr/wxqbYNaFwxubdPr+rmqeuNcL1h3J3yrahewa7b1SfZX1cgqDmlVWFf/GdTaBrUuGNzallLXan/JawK4ZMrzi7s2SdIqWu3wfwrYlOTSJK8F3g7sWeUxSFLzVnXap6pOJrkTeIzepZ67q+q5RW5m1imhPmdd/WdQaxvUumBwa1t0Xat6wleStD54YzdJapDhL0kN6pvwT7I1yTeTHEqyY63Hs5ySHE5yMMnTSfav9XiWKsnuJMeSPDul7fwke5O82D2et5ZjXKpZars3yUS3355OcsNajnEpklyS5CtJnk/yXJL3de19vd/mqKuv91mS1yX5WpJvdHX9Qdd+aZKvdvn4N90FNXNvqx/m/LvbQvwrU24LAdw6KLeFSHIYGKmqvv7ySZLfBCaBz1bV5V3bHwLHq2pn96F9XlXdvZbjXIpZarsXmKyqj63l2M5EkguBC6vq60neABwAbgZup4/32xx13UIf77MkAc6pqskkrwGeAN4H/B7wpaoaS/IXwDeq6r65ttUvR/7/e1uIqvpv4NRtIbSOVNXjwPFpzTcBD3TLD9D7Bew7s9TW96rqSFV9vVv+HvACcBF9vt/mqKuvVc9k9/Q13U8BvwX8bde+oP3VL+F/EfAfU56/zADsyCkK+MckB7rbWwySoao60i3/JzC0loNZAXcmeaabFuqrqZHpkgwDvwZ8lQHab9Pqgj7fZ0nOSvI0cAzYC/wb8GpVney6LCgf+yX8B92bqupK4Hrgjm6KYeBUb45x/c8zLtx9wC8AVwBHgI+v6WjOQJINwBeB91fVd6eu6+f9NkNdfb/PqurHVXUFvTskXA380lK20y/hP9C3haiqie7xGPBlejt0UBzt5l9PzcMeW+PxLJuqOtr9Iv4E+DR9ut+6ueMvAp+rqi91zX2/32aqa1D2GUBVvQp8Bfh1YGOSU1/aXVA+9kv4D+xtIZKc052QIsk5wHXAs3O/qq/sAbZ1y9uAh9ZwLMvqVDh23kof7rfuBOL9wAtV9Ykpq/p6v81WV7/vsyRvTLKxW349vYtgXqD3IfC2rtuC9ldfXO0D0F2S9cf8320hPrq2I1oeSX6e3tE+9G638fl+rS3JF4BRereXPQrcA/wd8CDws8BLwC1V1XcnTmepbZTe9EEBh4H3Tpkn7wtJ3gT8E3AQ+EnX/AF68+N9u9/mqOtW+nifJfkVeid0z6J38P5gVX2oy5Ex4Hzgn4Hfqaofzbmtfgl/SdLy6ZdpH0nSMjL8JalBhr8kNcjwl6QGGf6S1CDDX5IaZPhLUoP+BweMU0oHLqrWAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_contracts[df_contracts.a_age < 30].a_age.hist(bins=25)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the characteristics of a normal distribution (age of apprentice)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's store the *mean* (μ) and *standard deviation* (σ) of the distribution of apprentice age:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "a_age_std = df_contracts.a_age.describe()['std']\n",
    "a_age_mean = df_contracts.a_age.describe()['mean']"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "μ + 2σ captures 95% of the population (if the distribution is normal):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "sigma_2_upper = (a_age_mean + (a_age_std * 2)).round(2)\n",
    "sigma_2_lower = (a_age_mean - (a_age_std * 2)).round(2)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "μ + 3σ captures 99.7% of the population (if the distribution is normal):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "sigma_3_upper = (a_age_mean + (a_age_std * 3)).round(2)\n",
    "sigma_3_lower = (a_age_mean - (a_age_std * 3)).round(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'In 95% of the contracts, the age of apprentice should be comprised between 8.46-20.07 y.o.'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f'In 95% of the contracts, the age of apprentice should be comprised between {sigma_2_lower}-{sigma_2_upper} y.o.'"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's verify whether this holds true for apprentice age:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19.0"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_contracts.a_age.quantile(.95)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'In 99.7% of the contracts, the age of apprentice will be comprised between 5.56-22.97 y.o.'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f'In 99.7% of the contracts, the age of apprentice will be comprised between {sigma_3_lower}-{sigma_3_upper} y.o.'"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's verify whether this holds true for apprentice age:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.0"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_contracts.a_age.quantile(.997)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAU4UlEQVR4nO3df4xdZ53f8feHJPwQXuGEZEep49apcLsKpBgYJVnBH5MgEiesmqxEUVAKDpuVt1UigZq2GKQ2S9hIWWkXWlQ2qnfjYloWbwSksUK2qWsyovwRkhhCEiegeMGUWCbWrp3AQJvK9Ns/7uPdW3c8cz0zd2zP835JV/ec5zz33Oebe/O5x88990yqCklSH151qgcgSVo+hr4kdcTQl6SOGPqS1BFDX5I6cvapHsBczj///Fq3bt2cfX7+85/z+te/fnkGdJrptXbr7ot1n7w9e/b8ZVVdMNu20zr0161bxxNPPDFnn+npaaamppZnQKeZXmu37r5Y98lL8qMTbXN6R5I6YuhLUkfmDf0kr03yWJLvJtmb5JOt/fNJfpjkyXbb0NqT5LNJ9iV5Ksnbh/a1Kcnz7bZpbFVJkmY1ypz+K8BVVTWT5Bzgm0n+vG37F1X15eP6Xwusb7fLgXuAy5OcB9wBTAIF7Emys6qOLEUhkqT5zXukXwMzbfWcdpvrgj3XA19oj3sUWJ3kQuAaYFdVHW5BvwvYuLjhS5JOxkhn7yQ5C9gDvAn4XFV9K8k/Be5K8q+B3cCWqnoFWAP8eOjhL7S2E7Uf/1ybgc0AExMTTE9Pzzm2mZmZefusVL3Wbt19se6lNVLoV9UvgQ1JVgP3J3kL8HHgJ8Crga3Ax4A7Fzugqtra9sfk5GTNd8pSr6dzQb+1W3dfrHtpndTZO1X1EvAIsLGqDrYpnFeA/wBc1rodANYOPeyi1naidknSMhnl7J0L2hE+SV4HvAf4XpunJ0mAG4Bn2kN2Ah9qZ/FcAbxcVQeBh4Grk5yb5Fzg6tYmSVomo0zvXAhsb/P6rwLuq6oHk3w9yQVAgCeBf9L6PwRcB+wDfgF8GKCqDif5FPB463dnVR1eskp0UtZt+dpI/fbf/d4xj0TScpo39KvqKeBts7RfdYL+Bdx6gm3bgG0nOUZJ0hLxF7mS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHZk39JO8NsljSb6bZG+ST7b2i5N8K8m+JH+W5NWt/TVtfV/bvm5oXx9v7d9Pcs3YqpIkzWqUI/1XgKuq6q3ABmBjkiuA3wc+U1VvAo4At7T+twBHWvtnWj+SXALcCLwZ2Aj8UZKzlrAWSdI85g39Gphpq+e0WwFXAV9u7duBG9ry9W2dtv3dSdLad1TVK1X1Q2AfcNlSFCFJGs3Zo3RqR+R7gDcBnwP+Anipqo62Li8Aa9ryGuDHAFV1NMnLwBtb+6NDux1+zPBzbQY2A0xMTDA9PT3n2GZmZubts1ItpvbbLz06fyc4Lf/b9vqaW3dfxlX3SKFfVb8ENiRZDdwP/NqSj+RvnmsrsBVgcnKypqam5uw/PT3NfH1WqsXUfvOWr43Ub/9NC9v/OPX6mlt3X8ZV90mdvVNVLwGPAL8OrE5y7EPjIuBAWz4ArAVo298A/NVw+yyPkSQtg1HO3rmgHeGT5HXAe4DnGIT/+1q3TcADbXlnW6dt/3pVVWu/sZ3dczGwHnhsieqQJI1glOmdC4HtbV7/VcB9VfVgkmeBHUl+D/gOcG/rfy/wH5PsAw4zOGOHqtqb5D7gWeAocGubNpIkLZN5Q7+qngLeNkv7D5jl7Juq+l/APzrBvu4C7jr5YUqSloK/yJWkjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqyLyhn2RtkkeSPJtkb5KPtPbfTXIgyZPtdt3QYz6eZF+S7ye5Zqh9Y2vbl2TLeEqSJJ3I2SP0OQrcXlXfTvIrwJ4ku9q2z1TVHwx3TnIJcCPwZuBvAf8tyd9rmz8HvAd4AXg8yc6qenYpCpEkzW/e0K+qg8DBtvyzJM8Ba+Z4yPXAjqp6Bfhhkn3AZW3bvqr6AUCSHa2voS9JyyRVNXrnZB3wDeAtwD8DbgZ+CjzB4F8DR5L8O+DRqvpP7TH3An/edrGxqn67tX8QuLyqbjvuOTYDmwEmJibesWPHjjnHNDMzw6pVq0auYSVZTO1PH3h5pH6XrnnDgvY/Tr2+5tbdl8XUfeWVV+6pqsnZto0yvQNAklXAV4CPVtVPk9wDfAqodv+HwG8taIRDqmorsBVgcnKypqam5uw/PT3NfH1WqsXUfvOWr43Ub/9NC9v/OPX6mlt3X8ZV90ihn+QcBoH/xar6KkBVvTi0/Y+BB9vqAWDt0MMvam3M0S5JWgajnL0T4F7guar69FD7hUPdfhN4pi3vBG5M8pokFwPrgceAx4H1SS5O8moGX/buXJoyJEmjGOVI/53AB4GnkzzZ2j4BfCDJBgbTO/uB3wGoqr1J7mPwBe1R4Naq+iVAktuAh4GzgG1VtXfJKpEkzWuUs3e+CWSWTQ/N8Zi7gLtmaX9orsdJksbLX+RKUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdWTe0E+yNskjSZ5NsjfJR1r7eUl2JXm+3Z/b2pPks0n2JXkqyduH9rWp9X8+yabxlSVJms0oR/pHgdur6hLgCuDWJJcAW4DdVbUe2N3WAa4F1rfbZuAeGHxIAHcAlwOXAXcc+6CQJC2PeUO/qg5W1bfb8s+A54A1wPXA9tZtO3BDW74e+EINPAqsTnIhcA2wq6oOV9URYBewcSmLkSTNLVU1eudkHfAN4C3A/6iq1a09wJGqWp3kQeDuqvpm27Yb+BgwBby2qn6vtf8r4H9W1R8c9xybGfwLgYmJiXfs2LFjzjHNzMywatWqkWtYSRZT+9MHXh6p36Vr3rCg/Y9Tr6+5dfdlMXVfeeWVe6pqcrZtZ4+6kySrgK8AH62qnw5yfqCqKsnonx5zqKqtwFaAycnJmpqamrP/9PQ08/VZqRZT+81bvjZSv/03LWz/49Tra27dfRlX3SOdvZPkHAaB/8Wq+mprfrFN29DuD7X2A8DaoYdf1NpO1C5JWiajnL0T4F7guar69NCmncCxM3A2AQ8MtX+oncVzBfByVR0EHgauTnJu+wL36tYmSVomo0zvvBP4IPB0kidb2yeAu4H7ktwC/Ah4f9v2EHAdsA/4BfBhgKo6nORTwOOt351VdXgpipAkjWbe0G9fyOYEm989S/8Cbj3BvrYB205mgJKkpeMvciWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUkZH/iIrODOtG/OMokvrkkb4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI/OGfpJtSQ4leWao7XeTHEjyZLtdN7Tt40n2Jfl+kmuG2je2tn1Jtix9KZKk+YxypP95YOMs7Z+pqg3t9hBAkkuAG4E3t8f8UZKzkpwFfA64FrgE+EDrK0laRvNehqGqvpFk3Yj7ux7YUVWvAD9Msg+4rG3bV1U/AEiyo/V99uSHLElaqMVce+e2JB8CngBur6ojwBrg0aE+L7Q2gB8f1375bDtNshnYDDAxMcH09PScg5iZmZm3z0o1W+23X3p0SZ/jdPxv2+trbt19GVfdCw39e4BPAdXu/xD4raUYUFVtBbYCTE5O1tTU1Jz9p6enma/PSjVb7Tcv8QXX9t80NW+f5dbra27dfRlX3QsK/ap68dhykj8GHmyrB4C1Q10vam3M0S5JWiYLOmUzyYVDq78JHDuzZydwY5LXJLkYWA88BjwOrE9ycZJXM/iyd+fChy1JWoh5j/STfAmYAs5P8gJwBzCVZAOD6Z39wO8AVNXeJPcx+IL2KHBrVf2y7ec24GHgLGBbVe1d6mIkSXMb5eydD8zSfO8c/e8C7pql/SHgoZManSRpSfmLXEnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktSRxVx7R/pr60a8/MP+u9875pFImotH+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI/OGfpJtSQ4leWao7bwku5I83+7Pbe1J8tkk+5I8leTtQ4/Z1Po/n2TTeMqRJM1llCP9zwMbj2vbAuyuqvXA7rYOcC2wvt02A/fA4EMCuAO4HLgMuOPYB4UkafnMG/pV9Q3g8HHN1wPb2/J24Iah9i/UwKPA6iQXAtcAu6rqcFUdAXbx/3+QSJLGbKFz+hNVdbAt/wSYaMtrgB8P9XuhtZ2oXZK0jBb95xKrqpLUUgwGIMlmBlNDTExMMD09PWf/mZmZefusVLPVfvulR5f0OUb9bzvq8y7Fa9Xra27dfRlX3QsN/ReTXFhVB9v0zaHWfgBYO9TvotZ2AJg6rn16th1X1VZgK8Dk5GRNTU3N1u2vTU9PM1+flWq22m8e8W/Vjuzpn4/YcbS30v6bphY8lGN6fc2tuy/jqnuh0zs7gWNn4GwCHhhq/1A7i+cK4OU2DfQwcHWSc9sXuFe3NknSMpr38CzJlxgcpZ+f5AUGZ+HcDdyX5BbgR8D7W/eHgOuAfcAvgA8DVNXhJJ8CHm/97qyq478cliSN2byhX1UfOMGmd8/St4BbT7CfbcC2kxqdJGlJ+YtcSeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHFv2H0aWTsW7Ev+G7/+73jnkkUp880pekjhj6ktQRQ1+SOmLoS1JHDH1J6siiQj/J/iRPJ3kyyROt7bwku5I83+7Pbe1J8tkk+5I8leTtS1GAJGl0S3Gkf2VVbaiqyba+BdhdVeuB3W0d4FpgfbttBu5ZgueWJJ2EcUzvXA9sb8vbgRuG2r9QA48Cq5NcOIbnlySdQKpq4Q9OfggcAQr491W1NclLVbW6bQ9wpKpWJ3kQuLuqvtm27QY+VlVPHLfPzQz+JcDExMQ7duzYMecYZmZmWLVq1YJrOJPNVvvTB14+RaNZWpeuecMJt/X6mlt3XxZT95VXXrlnaPbl/7HYX+S+q6oOJPlVYFeS7w1vrKpKclKfKlW1FdgKMDk5WVNTU3P2n56eZr4+K9Vstd884i9eT3f7b5o64bZeX3Pr7su46l7U9E5VHWj3h4D7gcuAF49N27T7Q637AWDt0MMvam2SpGWy4CP9JK8HXlVVP2vLVwN3AjuBTcDd7f6B9pCdwG1JdgCXAy9X1cHFDL4XJ7peze2XHl0xR/aSlsdipncmgPsH0/acDfxpVf2XJI8D9yW5BfgR8P7W/yHgOmAf8Avgw4t4bknSAiw49KvqB8BbZ2n/K+Dds7QXcOtCn0+StHj+IleSOuL19HVamuu6+8PfZXjdfenkeKQvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSNehuEUmutSA5I0Dh7pS1JHDH1J6oihL0kdMfQlqSOGviR1xLN3dEY7mTOg/IMrkkf6ktQVQ1+SOrLs0ztJNgL/FjgL+JOqunu5xyDNZdQpI6eLdCZa1tBPchbwOeA9wAvA40l2VtWzyzmOcfOXtpJOV8t9pH8ZsK+qfgCQZAdwPbCiQl+np6X+MF7uD/fbLz3KzVu+5r8wtCipquV7suR9wMaq+u22/kHg8qq6bajPZmBzW/37wPfn2e35wF+OYbhngl5rt+6+WPfJ+ztVdcFsG067UzaraiuwddT+SZ6oqskxDum01Wvt1t0X615ay332zgFg7dD6Ra1NkrQMljv0HwfWJ7k4yauBG4GdyzwGSerWsk7vVNXRJLcBDzM4ZXNbVe1d5G5HngpagXqt3br7Yt1LaFm/yJUknVr+IleSOmLoS1JHzujQT7IxyfeT7Euy5VSPZ1ySbEtyKMkzQ23nJdmV5Pl2f+6pHOM4JFmb5JEkzybZm+QjrX1F157ktUkeS/LdVvcnW/vFSb7V3u9/1k6GWHGSnJXkO0kebOu91L0/ydNJnkzyRGtb8vf6GRv6Q5d0uBa4BPhAkktO7ajG5vPAxuPatgC7q2o9sLutrzRHgdur6hLgCuDW9hqv9NpfAa6qqrcCG4CNSa4Afh/4TFW9CTgC3HLqhjhWHwGeG1rvpW6AK6tqw9D5+Uv+Xj9jQ5+hSzpU1f8Gjl3SYcWpqm8Ah49rvh7Y3pa3Azcs55iWQ1UdrKpvt+WfMQiCNazw2mtgpq2e024FXAV8ubWvuLoBklwEvBf4k7YeOqh7Dkv+Xj+TQ38N8OOh9RdaWy8mqupgW/4JMHEqBzNuSdYBbwO+RQe1tymOJ4FDwC7gL4CXqupo67JS3+//BviXwP9p62+kj7ph8MH+X5PsaZejgTG810+7yzDo5FVVJVmx594mWQV8BfhoVf10cPA3sFJrr6pfAhuSrAbuB37t1I5o/JL8BnCoqvYkmTrFwzkV3lVVB5L8KrAryfeGNy7Ve/1MPtLv/ZIOLya5EKDdHzrF4xmLJOcwCPwvVtVXW3MXtQNU1UvAI8CvA6uTHDtQW4nv93cC/zDJfgbTtVcx+NsbK71uAKrqQLs/xOCD/jLG8F4/k0O/90s67AQ2teVNwAOncCxj0eZz7wWeq6pPD21a0bUnuaAd4ZPkdQz+/sRzDML/fa3biqu7qj5eVRdV1ToG/z9/vapuYoXXDZDk9Ul+5dgycDXwDGN4r5/Rv8hNch2DOcBjl3S469SOaDySfAmYYnCp1ReBO4D/DNwH/G3gR8D7q+r4L3vPaEneBfx34Gn+Zo73Ewzm9Vds7Un+AYMv7c5icGB2X1XdmeTvMjgCPg/4DvCPq+qVUzfS8WnTO/+8qn6jh7pbjfe31bOBP62qu5K8kSV+r5/RoS9JOjln8vSOJOkkGfqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI/8XPx8/NkAzKbwAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_contracts.a_age.hist(bins=30)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "~99.7% of apprentice age in the contracts in our dataset follow a normal distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAARB0lEQVR4nO3df6hfd33H8edrrbrRiG2pu3RttnQjG1TLar20gjJuEfvzjyiM0lJsqo74RwMK+cMojBa7jjCsDsGVRRpsQQ0FdQ1tWJcFL51/VJuW0vTHuoaargkxwaWrXhVH9L0/vifx2/Te3F/f3Jt7Ps8HXL7nfD7nnO/nzUle33M/3+/33FQVkqQ2/N5yD0CStHQMfUlqiKEvSQ0x9CWpIYa+JDXk7OUewKlccMEFtWbNmhPrv/jFLzjnnHOWb0BLoO81Wt/K1/ca+1DfU0899dOqevd0fWd06K9Zs4Y9e/acWJ+cnGRiYmL5BrQE+l6j9a18fa+xD/UleXWmPqd3JKkhhr4kNcTQl6SGGPqS1BBDX5IaMmvoJ1md5PtJXkjyfJLPdO13JTmY5Jnu54ahfT6fZF+Sl5JcO9R+Xde2L8nm01OSJGkmc/nI5jFgU1U9neSdwFNJdnV9X6mqLw1vnORS4GbgPcAfAf+e5M+77q8BHwEOAE8m2VFVL4yiEEnS7GYN/ao6BBzqln+e5EXgolPssg7YXlW/Bn6cZB9wZde3r6peAUiyvdvW0JekJTKvL2clWQO8D/gh8EFgY5LbgD0Mfht4ncELwhNDux3gdy8Sr53UftU0z7EB2AAwNjbG5OTkib6pqak3rfdR32u0vpWv7zX2vb45h36SVcB3gM9W1c+S3AfcDVT3eC/wycUOqKq2AlsBxsfHa/ibcX34ptxs+l7jqOtbs/nRRe2/f8uNIxrJQN/PH/S/xr7XN6fQT/I2BoH/zar6LkBVHR7q/zrwSLd6EFg9tPvFXRunaJckLYG5fHonwP3Ai1X15aH2C4c2+xjwXLe8A7g5yTuSXAKsBX4EPAmsTXJJkrczeLN3x2jKkCTNxVyu9D8IfBzYm+SZru0LwC1JLmcwvbMf+DRAVT2f5CEGb9AeA+6oqt8AJNkIPAacBWyrqudHVokkaVZz+fTOD4BM07XzFPvcA9wzTfvOU+0nSTq9/EauJDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGzOVv5Eq9tWbzowved/+WG0c4EmlpeKUvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUkFlDP8nqJN9P8kKS55N8pms/P8muJC93j+d17Uny1ST7kjyb5IqhY63vtn85yfrTV5YkaTpzudI/BmyqqkuBDwB3JLkU2Azsrqq1wO5uHeB6YG33swG4DwYvEsCdwFXAlcCdx18oJElLY9bQr6pDVfV0t/xz4EXgImAd8EC32QPAR7vldcCDNfAEcG6SC4FrgV1VdbSqXgd2AdeNshhJ0qmlqua+cbIGeBx4L/DfVXVu1x7g9ao6N8kjwJaq+kHXtxv4HDAB/H5V/V3X/rfAr6rqSyc9xwYGvyEwNjb2/u3bt5/om5qaYtWqVQsqdKXoe42jrm/vwTdGdqz5uuyid72lre/nD/pfYx/qu/rqq5+qqvHp+ub8l7OSrAK+A3y2qn42yPmBqqokc3/1OIWq2gpsBRgfH6+JiYkTfZOTkwyv91Hfaxx1fbcv4i9fLdb+Wyfe0tb38wf9r7Hv9c3p0ztJ3sYg8L9ZVd/tmg930zZ0j0e69oPA6qHdL+7aZmqXJC2RuXx6J8D9wItV9eWhrh3A8U/grAceHmq/rfsUzweAN6rqEPAYcE2S87o3cK/p2iRJS2Qu0zsfBD4O7E3yTNf2BWAL8FCSTwGvAjd1fTuBG4B9wC+BTwBU1dEkdwNPdtt9saqOjqIISdLczBr63RuymaH7w9NsX8AdMxxrG7BtPgOUJI2O38iVpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGzBr6SbYlOZLkuaG2u5IcTPJM93PDUN/nk+xL8lKSa4far+va9iXZPPpSJEmzmcuV/jeA66Zp/0pVXd797ARIcilwM/Cebp9/SnJWkrOArwHXA5cCt3TbSpKW0NmzbVBVjydZM8fjrQO2V9WvgR8n2Qdc2fXtq6pXAJJs77Z9Yf5DliQt1Kyhfwobk9wG7AE2VdXrwEXAE0PbHOjaAF47qf2q6Q6aZAOwAWBsbIzJyckTfVNTU29a76O+1zjq+jZddmxkx5qv6ero+/mD/tfY9/oWGvr3AXcD1T3eC3xyFAOqqq3AVoDx8fGamJg40Tc5Ocnweh/1vcZR13f75kdHdqz52n/rxFva+n7+oP819r2+BYV+VR0+vpzk68Aj3epBYPXQphd3bZyiXZK0RBb0kc0kFw6tfgw4/smeHcDNSd6R5BJgLfAj4ElgbZJLkrydwZu9OxY+bEnSQsx6pZ/k28AEcEGSA8CdwESSyxlM7+wHPg1QVc8neYjBG7THgDuq6jfdcTYCjwFnAduq6vlRFyNJOrW5fHrnlmma7z/F9vcA90zTvhPYOa/RSZJGym/kSlJDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWrIYu6nL2mB1iziltD7t9w4wpGoNV7pS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDVk1tBPsi3JkSTPDbWdn2RXkpe7x/O69iT5apJ9SZ5NcsXQPuu77V9Osv70lCNJOpW5XOl/A7jupLbNwO6qWgvs7tYBrgfWdj8bgPtg8CIB3AlcBVwJ3Hn8hUKStHRmDf2qehw4elLzOuCBbvkB4KND7Q/WwBPAuUkuBK4FdlXV0ap6HdjFW19IJEmn2ULn9Meq6lC3/BNgrFu+CHhtaLsDXdtM7ZKkJXT2Yg9QVZWkRjEYgCQbGEwNMTY2xuTk5Im+qampN633Ud9rHHV9my47NrJjzdd0dcy1vsWMe7n/ffhvdGVbaOgfTnJhVR3qpm+OdO0HgdVD213ctR0EJk5qn5zuwFW1FdgKMD4+XhMTv9ttcnKS4fU+6nuNo67v9s2PjuxY87X/1om3tM21vsWMe7rnXUr+G13ZFhr6O4D1wJbu8eGh9o1JtjN40/aN7oXhMeDvh968vQb4/MKHLS2/NdME96bLji3rC5E0m1lDP8m3GVylX5DkAINP4WwBHkryKeBV4KZu853ADcA+4JfAJwCq6miSu4Enu+2+WFUnvzksSTrNZg39qrplhq4PT7NtAXfMcJxtwLZ5jU6SNFJ+I1eSGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SGGPqS1BBDX5IaYuhLUkMMfUlqiKEvSQ0x9CWpIYa+JDXE0Jekhhj6ktSQs5d7AJLmZ83mRxe1//4tN45oJFqJvNKXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNWRRoZ9kf5K9SZ5JsqdrOz/JriQvd4/nde1J8tUk+5I8m+SKURQgSZq7UVzpX11Vl1fVeLe+GdhdVWuB3d06wPXA2u5nA3DfCJ5bkjQPp2N6Zx3wQLf8APDRofYHa+AJ4NwkF56G55ckzSBVtfCdkx8DrwMF/HNVbU3yv1V1btcf4PWqOjfJI8CWqvpB17cb+FxV7TnpmBsY/CbA2NjY+7dv336ib2pqilWrVi14vCtB32scdX17D74xsmONwtgfwOFfLfcoTu2yi961qP39N3rmu/rqq58amn15k8XecO1DVXUwyR8Cu5L853BnVVWSeb2qVNVWYCvA+Ph4TUxMnOibnJxkeL2P+l7jqOu7fZE3Hxu1TZcd4969Z/Z9DPffOrGo/f03urItanqnqg52j0eA7wFXAoePT9t0j0e6zQ8Cq4d2v7hrkyQtkQWHfpJzkrzz+DJwDfAcsANY3222Hni4W94B3NZ9iucDwBtVdWjBI5ckzdtifg8dA743mLbnbOBbVfWvSZ4EHkryKeBV4KZu+53ADcA+4JfAJxbx3OqRxd4fXtLcLTj0q+oV4C+naf8f4MPTtBdwx0KfT2e2uQb3psuOnXHz8FJL/EauJDXE0Jekhhj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSGGviQ1xNCXpIYY+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0JakhZ/ZfcJZ0Rlmz+dEF/yGc/VtuPA0j0nx5pS9JDTH0Jakhhr4kNcQ5fZ0w1z9uLmnl8kpfkhrilb7UGH+ja5tX+pLUEENfkhpi6EtSQwx9SWqIoS9JDTH0Jakhhr4kNcTQl6SG+OUsSUtiMV8K87bMo2PoS+q1+b7YnPz3Avr2grPk0ztJrkvyUpJ9STYv9fNLUsuW9Eo/yVnA14CPAAeAJ5PsqKoXlnIcfeZ9VSSdylJP71wJ7KuqVwCSbAfWAYZ+Z+/BNxb0p+ikPlvOi5nleu7TNa2UqjotB572yZK/Bq6rqr/p1j8OXFVVG4e22QBs6Fb/Anhp6BAXAD9douEul77XaH0rX99r7EN9f1JV756u44x7I7eqtgJbp+tLsqeqxpd4SEuq7zVa38rX9xr7Xt9Sv5F7EFg9tH5x1yZJWgJLHfpPAmuTXJLk7cDNwI4lHoMkNWtJp3eq6liSjcBjwFnAtqp6fh6HmHbap2f6XqP1rXx9r7HX9S3pG7mSpOXlvXckqSGGviQ1ZMWEft9v35Bkf5K9SZ5Jsme5xzMKSbYlOZLkuaG285PsSvJy93jeco5xMWao764kB7vz+EySG5ZzjIuRZHWS7yd5IcnzST7TtffpHM5UY2/O48lWxJx+d/uG/2Lo9g3ALX26fUOS/cB4Va30L4WckOSvgCngwap6b9f2D8DRqtrSvXifV1WfW85xLtQM9d0FTFXVl5ZzbKOQ5ELgwqp6Osk7gaeAjwK3059zOFONN9GT83iylXKlf+L2DVX1f8Dx2zfoDFZVjwNHT2peBzzQLT/A4D/YijRDfb1RVYeq6ulu+efAi8BF9OsczlRjb62U0L8IeG1o/QD9OzEF/FuSp7pbUfTVWFUd6pZ/Aowt52BOk41Jnu2mf1bs1MewJGuA9wE/pKfn8KQaoYfnEVZO6LfgQ1V1BXA9cEc3ddBrNZhbPPPnF+fnPuDPgMuBQ8C9yzqaEUiyCvgO8Nmq+tlwX1/O4TQ19u48HrdSQr/3t2+oqoPd4xHgewymtProcDePenw+9cgyj2ekqupwVf2mqn4LfJ0Vfh6TvI1BGH6zqr7bNffqHE5XY9/O47CVEvq9vn1DknO6N5FIcg5wDfDcqfdasXYA67vl9cDDyziWkTsehp2PsYLPY5IA9wMvVtWXh7p6cw5nqrFP5/FkK+LTOwDdR6b+kd/dvuGe5R3R6CT5UwZX9zC4Nca3+lBfkm8DEwxuVXsYuBP4F+Ah4I+BV4GbqmpFvhk6Q30TDKYECtgPfHpo/ntFSfIh4D+AvcBvu+YvMJjz7ss5nKnGW+jJeTzZigl9SdLirZTpHUnSCBj6ktQQQ1+SGmLoS1JDDH1JaoihL0kNMfQlqSH/DxDIlhKHvc86AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_contracts.a_age[df_contracts.a_age <= df_contracts.a_age.quantile(.997)].hist(bins=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAASlElEQVR4nO3df2zcd33H8dfbl3AXm65OqRVVLZmLqMCdV+h6ZUVELAFCYEVLqyFGNFCkWo5M2MGUTHMS/zFlW6z4n8AWQY6URETqZkBsXaoW0UTZRZOn5YdN25Biunb9obUqjYE4K3bPOPF7f/gbY7t2/HV8d19/fM+HFN19P/e9+74tOS9/9Pl+Pp8zdxcAIDw1SRcAALg+BDgABIoAB4BAEeAAECgCHAACtaySF7v55pu9sbGxkpcEgOD19fX9wt0bprdXNMAbGxvV29tbyUsCQPDM7JWZ2hlCAYBAEeAAECgCHAACRYADQKAIcAAIVKwAN7OXzewnZva0mfVGbTeZ2XEzez56XFneUoHSy+VyymQyMjNlMhnlcrmkSwJim08PfJ27f9Dds9HxDkkn3P0OSSeiYyAYuVxO+XxenZ2dGhoaUmdnp/L5PCGOYFic7WTN7GVJWXf/xaS25yStdffXzewWSSfd/X3X+pxsNuvMA8dikclk1NnZqW3btk207du3T7t27VKxWEywMmAqM+ub1Hn+bXvMAH9J0kVJLulb7n7QzAbdvT563SRdvHo87b1bJG2RpNWrV9/zyiszzkcHKs7MNDQ0pNra2om24eFh1dXViX3ysZjMFuBxh1DWuPsfSPq0pC+b2Ucnv+jjv+0z/sa7+0F3z7p7tqHhbStBgcSk02nl8/kpbfl8Xul0OqGKgPmJtZTe3V+LHi+Y2aOSPiTpDTO7ZdIQyoUy1gmUXGtrq9rb2yVJbW1tyufzam9vV1tbW8KVAfHMGeBmViepxt3fjJ5/UtLfSnpM0mZJe6PHo+UsFCi1/fv3S5J27dql7du3K51Oq62tbaIdWOzmHAM3s/dIejQ6XCbpn919j5m9S9L3Ja2W9Iqkz7n7r671WdzEBID5m20MfM4euLu/KOkDM7T/UtLHS1MeAGC+WIkJAIEiwAEgUAQ4AASKAAeAQBHgABAoAhwAAkWAA0CgCHAACBQBDgCBIsABIFAEOAAEigAHgEAR4AAQKAIcAAJFgANAoAhwAAgUAQ4AgSLAASBQBDgABIoAB4BAEeAAECgCHAACRYADQKAIcAAIFAEOAIEiwAEgUAQ4AASKAAeAQBHgABAoAhwAAkWAA0CgYge4maXM7Ckzezw6vt3MTpvZC2b2PTN7R/nKBABMN58e+Fcl9U867pL0NXd/r6SLklpKWRhQCblcTplMRmamTCajXC6XdElAbLEC3Mxuk3S/pG9HxybpY5J+EJ1yRNIDZagPKJtcLqd8Pq/Ozk4NDQ2ps7NT+XyeEEcw4vbAvy7pryWNRcfvkjTo7pej41cl3Vra0oDyevjhh9XV1aVt27aptrZW27ZtU1dXlx5++OGkSwNimTPAzewzki64e9/1XMDMtphZr5n1DgwMXM9HAGUxMjKitra2KW1tbW0aGRlJqCJgfuL0wD8i6U/M7GVJ39X40Mk/SKo3s2XRObdJem2mN7v7QXfPunu2oaGhBCUDpZFOp5XP56e05fN5pdPphCoC5mfOAHf3ne5+m7s3Svq8pH939z+XVJD02ei0zZKOlq1KoAxaW1vV3t6uffv2aXh4WPv27VN7e7taW1uTLg2IZdncp8yqXdJ3zezvJT0l6VBpSgIqY//+/ZKkXbt2afv27Uqn02pra5toBxY7c/eKXSybzXpvb2/FrgcAS4GZ9bl7dno7KzEBIFAEOAAEigAHgEAR4AAQKAIcAAJFgANAoAhwAAgUAQ4AgSLAUdU2bNigmpoamZlqamq0YcOGpEsCYiPAUbU2bNigY8eOqa2tTYODg2pra9OxY8cIcQRjIXuhAEE7fvy4vvSlL+mb3/ymJE08Tt+hEFis2AsFVcvMNDg4qBtvvHGi7dKlS6qvr1cl/18Ac2EvFGAaM9POnTuntO3cuVPj3xgILH4EOKrW+vXrdeDAAW3dulWXLl3S1q1bdeDAAa1fvz7p0oBYGEJBVduwYYOOHz8ud5eZaf369XryySeTLguYYrYhFG5ioqoR1ggZQyioat3d3WpublYqlVJzc7O6u7uTLgmIjR44qlZ3d7c6Ojp06NAhrVmzRj09PWppaZEkbdq0KeHqgLkxBo6q1dzcrP3792vdunUTbYVCQblcTufPn0+wMmCq2cbACXBUrVQqpWKxqOXLl0+0jY6OKpPJ6MqVKwlWBkzFPHBgmqamJu3evXvKGPju3bvV1NSUdGlALAQ4qta6devU1dWlhx56SG+++aYeeughdXV1TRlSARYzAhxVq1AoqL29XYcPH9YNN9ygw4cPq729XYVCIenSgFgYA0fVYgwcoWAMHJimqalJPT09U9p6enoYA0cwCHBUrY6ODrW0tKhQKGh0dFSFQkEtLS3q6OhIujQgFhbyoGpdXayTy+XU39+vpqYm7dmzh0U8CAZj4ACwyDEGDswgl8spk8nIzJTJZJTL5ZIuCYiNAEfVyuVyyufz6uzs1NDQkDo7O5XP5wlxBIMhFFStTCajzs5Obdu2baJt37592rVrl4rFYoKVAVOxFwowjZlpaGhItbW1E23Dw8Oqq6vjOzGxqFz3GLiZZczsjJk9Y2bPmtnuqP12MzttZi+Y2ffM7B3lKBwol3Q6rbq6OpnZxL+6ujql0+mkSwNiiTMGPiLpY+7+AUkflPQpM7tPUpekr7n7eyVdlNRStiqBMhgZGZEkLV++XD09PRMrMq+2A4vdnAHu434dHS6P/rmkj0n6QdR+RNID5SgQKKdUKqWamhqtWbNGNTU1SqVSSZcExBZrFoqZpczsaUkXJB2X9D+SBt39cnTKq5JuneW9W8ys18x6BwYGSlAyUDpnzpxRsViUu6tYLOrMmTNJlwTEFmslprtfkfRBM6uX9Kik98e9gLsflHRQGr+JeR01AmVz7733yt0nvpXezJIuCYhtXvPA3X1QUkHShyXVm9nVPwC3SXqttKUB5Tc2NqaamhqdPHlSNTU1GhsbS7okILY4s1Aaop63zGyFpPWS+jUe5J+NTtss6WiZagTK4mpv+8qVK1q7du3EFrL0whGKOD3wWyQVzOycpLOSjrv745LaJW0zsxckvUvSofKVCZSeu2twcHBiCGXyMRCCOcfA3f2cpLtnaH9R0ofKURRQCWamVatWTZk2mE6n6YEjGOyFgqqVSqU0MjKilStX6ty5c1q5cqVGRkaYSohgsB84qtbly5dVW1urwcFB3XXXXTIz1dbWanh4OOnSgFjogaOq9ff3a2xsTO6usbEx9ff3J10SEBs9cFS1pqYmvfXWWxPzwFesWJF0SUBs9MBRtZYtW6bh4WHV19fr3Llzqq+v1/DwsJYto1+DMPCbiqp15coVpdNpXbx4UXfddZek8Vkov/nNbxKuDIiHHjiqlrvrjTfemDIP/OoxEAJ64KhaZqb6+voZ24EQ0ANH1bra006lUjp58uTE/G964AgFAY6qdnUDq7Vr105sbAWEgt9WVLWzZ89OmQd+9uzZpEsCYmMMHFXtnnvuSboE4LrRAwckPfLII0mXAMwbAQ5I+sIXvpB0CcC8EeCoan19fVPmgff19SVdEhAbY+CoaoyBI2T0wAFJDz74YNIlAPNGgAOSHn300aRLAOaNAEdVa21tnTIG3tramnRJQGxWyWXD2WzWe3t7K3Y94FqutecJy+mxmJhZn7tnp7fTAwckbdy4MekSgHkjwAFJR48eTboEYN4IcFS1U6dOTRkDP3XqVNIlAbER4Khqa9euveYxsJgR4KhaZqZisagVK1bo9OnTWrFihYrFIl/ogGCwEhNV6+r+38ViUffdd5+k8VAfGxtLuDIgHqYRYklq3PFERa7z8t77K3IdVLfZphHSA8eSNN9gbdzxBGGM4DAGDgCBIsABIFAEOAAEas4AN7N3m1nBzH5qZs+a2Vej9pvM7LiZPR89rix/uQCAq+L0wC9L2u7ud0q6T9KXzexOSTsknXD3OySdiI4BABUyZ4C7++vu/uPo+ZuS+iXdKmmjpCPRaUckPVCmGgEAM5jXGLiZNUq6W9JpSavc/fXopZ9LWjXLe7aYWa+Z9Q4MDCykVgDAJLED3MzeKelfJP2lu//f5Nd8fDXQjCuC3P2gu2fdPdvQ0LCgYgEAvxUrwM1sucbD+5/c/V+j5jfM7Jbo9VskXShPiQCAmcSZhWKSDknqd/d9k156TNLm6PlmSWyoDAAVFGcp/UckfVHST8zs6ahtl6S9kr5vZi2SXpH0ubJUCACY0ZwB7u49kmbbX/PjpS0HABAXKzEBIFAEOAAEigAHgEAR4AAQKAIcAAJFgANAoAhwAAgUAQ4AgSLAASBQBDgABCrOXihAoj6w+5guvTVa9us07niirJ9/44rleuZvPlnWa6C6EOBY9C69NaqX996fdBkLVu4/EKg+DKEAQKAIcAAIFAEOAIEiwAEgUAQ4AASKAAeAQDGNEIveDU079PtHdiRdxoLd0CRJ4U+HxOJBgGPRe7N/L/PAgRkwhAIAgSLAASBQBDgABIoAB4BAEeAAECgCHAACRYADQKCYB44gLIU51DeuWJ50CVhiCHAsepVYxNO444klsVgI1YUhFAAIFAEOAIGaM8DN7LCZXTCz85PabjKz42b2fPS4srxlAgCmi9MD/46kT01r2yHphLvfIelEdAwAqKA5A9zd/0PSr6Y1b5R0JHp+RNIDpS0LADCX6x0DX+Xur0fPfy5pVYnqAQDEtOCbmO7ukny2181si5n1mlnvwMDAQi8HAIhcb4C/YWa3SFL0eGG2E939oLtn3T3b0NBwnZcDAEx3vQH+mKTN0fPNko6WphwAQFxxphF2S/ovSe8zs1fNrEXSXknrzex5SZ+IjgEAFTTnUnp33zTLSx8vcS0AgHlgJSYABIoAB4BAEeAAECgCHAACRYADQKAIcAAIFAEOAIEiwAEgUAQ4AASKAAeAQBHgABCoOfdCAULUuOOJirzn5b33z/s9QKkQ4FiSCFZUA4ZQACBQBDgABIoAB4BAEeCoat3d3WpublYqlVJzc7O6u7uTLgmIjZuYqFrd3d3q6OjQoUOHtGbNGvX09KilpUWStGnTbF9EBSwe5u4Vu1g2m/Xe3t6KXQ+4lubmZu3fv1/r1q2baCsUCsrlcjp//nyClQFTmVmfu2ff1k6Ao1qlUikVi0UtX758om10dFSZTEZXrlxJsDJgqtkCnDFwVK2mpib19PRMaevp6VFTU1NCFQHzQ4CjanV0dKilpUWFQkGjo6MqFApqaWlRR0dH0qUBsXATE1Xr6o3KXC6n/v5+NTU1ac+ePdzARDAYAweARY4xcABYYghwAAgUAQ4AgSLAASBQBDiqGnuhIGRMI0TVYi8UhI5phKha7IWCULAXCjANe6EgFGWZB25mnzKz58zsBTPbsZDPAiqNvVAQuusOcDNLSfqGpE9LulPSJjO7s1SFAeXGXigI3UJuYn5I0gvu/qIkmdl3JW2U9NNSFAaUG3uhIHQLCfBbJf3vpONXJf3h9JPMbIukLZK0evXqBVwOKL1NmzYR2AhW2eeBu/tBd8+6e7ahoaHclwOAqrGQAH9N0rsnHd8WtQEAKmAhAX5W0h1mdruZvUPS5yU9VpqyAABzue4xcHe/bGZ/IelJSSlJh9392ZJVBgC4pgUtpXf3H0r6YYlqAQDMQ0VXYprZgKRXKnZBIL6bJf0i6SKAWfyuu79tFkhFAxxYrMysd6alysBixnayABAoAhwAAkWAA+MOJl0AMF+MgQNAoOiBA0CgCHAACBQBDgCBIsABIFAEOJYcM/s3M+szs2ej/ehnO++AmfVG5+2e1P7HZvaz6DP+0cwej9rrzOywmZ0xs6fMbGMlfh5gNsxCwZJjZje5+6/MbIXGd838I3f/5TXOS0k6Iekrkv5b0vOSPuruL5lZt6Qb3P0zZtYp6afu/oiZ1Us6I+ludx+q1M8GTEYPHEvRV8zsGUmnNL5n/R2znPc5M/uxpKck/Z7Gv9v1/ZJedPeXonO6J53/SUk7zOxpSSclZSTxNVNIzIJ2IwQWGzNbK+kTkj7s7sNmdlLjQTv9vNsl/ZWke939opl9Z6bzpr9N0p+6+3OlrBm4XvTAsdTcKOliFN7vl3TfLOf9jqQhSZfMbJWkT0ftz0l6j5k1Rsd/Nuk9T0rKmZlJkpndXerigfmgB46l5keS2sysX+NhfGqmk9z9GTN7StLPNP7l3P8Ztb9lZlsl/cjMhjQ+hn7V30n6uqRzZlYj6SVJnynXDwLMhZuYwDRm9k53/3XU0/6GpOfd/WtJ1wVMxxAK8Hat0Y3KZzU+JPOtZMsBZkYPHEuemZ2WlJ7W/EV3/0kS9QClQoADQKAYQgGAQBHgABAoAhwAAkWAA0Cg/h9hB+jLQw1e5AAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_contracts.a_age.plot(kind='box')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "24"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_contracts.a_age[df_contracts.a_age > 28].shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9653"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_contracts.a_age.shape[0]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Two very important distributions"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normal\n",
    "\n",
    "Also known as Gaussian, is a bell-shaped distribution with mass around the mean and exponentially decaying on the sides. It is fully characterized by the mean (center of mass) and standard deviation (spread).\n",
    "\n",
    "https://en.wikipedia.org/wiki/Normal_distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.FacetGrid at 0x1254ef220>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAFgCAYAAACFYaNMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAVuklEQVR4nO3db7Bd5XXf8e/ij2xjYglhVZYlxKWYSWBSGbs3NjaZTgNOBztucDIOJk1txkOqF5Udu84kwel02rR9QWZC/KfVkGogDbSuCSF4UFJK4gBOp0NNLGz3yBzoWMYCpAgEGMsk4F4LVl+creujg6R7JO4+6/z5fmY0d+9n73tZB65+s3n2ftaOzESSNHonVRcgSbPKAJakIgawJBUxgCWpiAEsSUVOqS7glbjsssvyrrvuqi5DkpYSRxqc6Cvgp59+uroESTphEx3AkjTJDGBJKmIAS1IRA1iSihjAklTEAJakIgawJBUxgCWpiAEsSUUMYEkqYgBLUhEDWJKKGMCSVGSi21FKJ2JhYYFOp3PY2KZNm1ixYkVRRZpVBrBmTqfTYcvW7axcNwfAgX272boF5ufnawvTzDGANZNWrptj9dz51WVoxjkHLElFDGBJKmIAS1IRA1iSihjAklTEAJakIgawJBUxgCWpiAEsSUVaDeCI2B0ROyPi6xGxoxlbHRFfjIhvNl/PaMYjIj4bEbsiohMRb22zNkmqNoor4J/KzAsz89BC+2uAuzPzPODuZh/g3cB5zZ/NwPUjqE2SylRMQVwO3NRs3wS8r2/85uz5MrAqItYV1CdJI9F2M54E/jwiEvhPmbkNWJuZ+5rjTwBrm+31wON937unGdvXN0ZEbKZ3hczGjRtbLF3TpL8FZbfbJTMXj7304kG63e5h59ueUqPQdgD/ZGbujYi/A3wxIh7uP5iZ2YTz0JoQ3wYwPz9/XN+r2dXfgnLvzvtYde6FnNkce27/Hq577AXWPnwQsD2lRqfVAM7Mvc3X/RHxBeBtwJMRsS4z9zVTDPub0/cCZ/V9+4ZmTFoWh1pQHti3+2XHTl97tu0pNXKtBXBEvBY4KTOfa7b/EfBvge3AVcC1zdc7mm/ZDnwkIm4B3g4c6JuqkI7Jt1xoErV5BbwW+EJEHPrn/LfMvCsivgLcGhFXA48CVzTn3wm8B9gFPA98uMXaNGV8y4UmUWsBnJmPAG8+wvgzwKVHGE9gS1v1aPr5lgtNGlfCSVIRA1iSihjAklTEAJakIgawJBUxgCWpiAEsSUUMYEkq0nYzHmniucxZbTGApSW4zFltMYClIbjMWW1wDliSihjAklTEKQhNpcHXDA2+hkgaBwawptLga4YGX0MkjQMDWFOr/zVDR3oNkVTNOWBJKmIAS1IRA1iSijgHLA3wCQqNigEsDVjqCYrBgAZ7Q+jEGMDSERzrCYrBgLY3hE6UASydgP6Alk6UN+EkqYgBLElFDGBJKmIAS1IRA1iSihjAklTEAJakIgawJBUxgCWpiAEsSUUMYEkqYi8ITaSFhQU6nc7ivi0jNYkMYE2kTqfDlq3bWbluDvClm5pMBrAm1sp1c750UxPNOWBJKmIAS1IRA1iSihjAklTEAJakIgawJBUxgCWpiAEsSUUMYEkqYgBLUhEDWJKKGMCSVKT1ZjwRcTKwA9ibme+NiHOAW4AzgQeAD2bmQkS8CrgZ+PvAM8AHMnN32/VJr9RLLx6k2+0eNrZp0yZWrFhRVJEmxSi6oX0MeAh4XbP/28CnMvOWiPg94Grg+ubrs5n5poi4sjnvAyOoT3pFntu/h+see4G1Dx8Eep3Ztm6B+fn54so07lqdgoiIDcDPADc0+wFcAtzWnHIT8L5m+/Jmn+b4pc350tg7fe3ZrJ47n9Vz5y/2KJaW0vYc8KeBXwdeavbPBL6bmQeb/T3A+mZ7PfA4QHP8QHO+JE2l1gI4It4L7M/MB5b5526OiB0RseOpp55azh8tSSPV5hXwxcDPRsRuejfdLgE+A6yKiENzzxuAvc32XuAsgOb4Sno34w6Tmdsycz4z59esWdNi+ZLUrtYCODM/mZkbMnMOuBK4JzN/CbgXeH9z2lXAHc329maf5vg96VsWJU2xiueAfwP4RETsojfHe2MzfiNwZjP+CeCagtokaWRG8lLOzPwS8KVm+xHgbUc45/vAL4yiHkkaB66Ek6QiBrAkFTGAJanISOaApeO1sLBAp9M5bMz+Cpo2BrDGUqfTYcvW7YvLeu2voGlkAGtsrVw3x+q586vLkFrjHLAkFTGAJamIUxCaCINNz7vdLq5U16QzgDURBpue7915H6vOvdB+pZpoBrAmxqGm59B7KkKadM4BS1IRA1iSijgFIS0z35KsYRnA0jLzLckalgEstaD/hqF0NM4BS1IRA1iSihjAklTEOWCpZT4VoaMxgKWW+VSEjsYAlkbApyJ0JM4BS1IRA1iSihjAklTEAJakIgawJBUxgCWpiAEsSUUMYEkqYgBLUhEDWJKKGMCSVMQAlqQiBrAkFTGAJamIASxJRQxgSSpiAEtSEQNYkor4SiKNhYWFBTqdzuJ+t9slMwsrktpnAGssdDodtmzdzsp1cwDs3Xkfq869kDNry5JaZQBrbKxcN7f44soD+3bXFiONgHPAklTEAJakIgawJBUxgCWpiAEsSUUMYEkqYgBLUpHWAjgiXh0RfxUR/yciHoyI32rGz4mI+yNiV0T8YUSsaMZf1ezvao7PtVWbJI2DNq+A/x9wSWa+GbgQuCwiLgJ+G/hUZr4JeBa4ujn/auDZZvxTzXmSNLVaWwmXvYX8f9Psntr8SeAS4J804zcB/wa4Hri82Qa4DfiPERFpQ4CpZO8HqeWlyBFxMvAA8CZgK/At4LuZebA5ZQ+wvtleDzwOkJkHI+IAcCbw9MDP3AxsBti4cWOb5atF9n6QWr4Jl5kvZuaFwAbgbcCPLcPP3JaZ85k5v2bNmlf641ToUO+H1XPnc/rr31hdjjRyQwVwRFw8zNjRZOZ3gXuBdwCrIuLQlfcGYG+zvRc4q/nZpwArgWeG/WdI0qQZ9gr4Pww5tigi1kTEqmb7NcBPAw/RC+L3N6ddBdzRbG9v9mmO3+P8r6Rpdsw54Ih4B/BOYE1EfKLv0OuAk5f42euAm5p54JOAWzPzTyOiC9wSEf8e+BpwY3P+jcB/iYhdwHeAK4/700gT4KUXD9Ltdg8b27RpEytWrCiqSFWWugm3Aji9Oe9H+sa/xw+vYo8oMzvAW44w/gi9+eDB8e8Dv7BEPdLEe27/Hq577AXWPty7F31g3262boH5+fniyjRqxwzgzPxL4C8j4g8y89ER1SRNvdPXnr3YfF6za9jH0F4VEduAuf7vycxL2ihKkmbBsAH8R8DvATcAL7ZXjiTNjmED+GBmXt9qJZI0Y4YN4D+JiH8OfIFejwcAMvM7rVQlzRCfiphdwwbwoedzf61vLIG/u7zlSLPHpyJm11ABnJnntF2INMt8KmI2DRXAEfGhI41n5s3LW44kzY5hpyB+om/71cClwFcBA1iSTtCwUxAf7d9vejzc0kZBmk72/5Ve7kT7Af8t4Lywhmb/X+nlhp0D/hN6Tz1ArwnP+cCtbRWl6XSo/y/07vRLs27YK+Df6ds+CDyamXtaqEeSZsZQ/YCbpjwP0+uIdgaw0GZRkjQLhn0jxhXAX9FrF3kFcH9EHLMdpSTp2IadgviXwE9k5n7ove0C+At6by+WJJ2AYV9JdNKh8G08cxzfK0k6gmGvgO+KiD8DPt/sfwC4s52SJGk2LPVOuDcBazPz1yLi54GfbA79b+BzbRcnSdNsqSvgTwOfBMjM24HbASLi7zXH/nGLtUnSVFtqHndtZu4cHGzG5lqpSJJmxFIBvOoYx16zjHVI0sxZKoB3RMQ/GxyMiF8GHminJEmaDUvNAX8c+EJE/BI/DNx5YAXwcy3WJUlT75gBnJlPAu+MiJ8CfrwZ/u+ZeU/rlUnSlBu2H/C9wL0t1yJJM8XVbJJUxACWpCIGsCQVMYAlqYgBLElFDGBJKmIAS1IRA1iSihjAklTEAJakIgawJBUxgCWpiAEsSUUMYEkqYgBLUhEDWJKKDNWQXTpeCwsLdDqdxf1ut0tmFlYkjR8DWK3odDps2bqdlevmANi78z5WnXshZ9aWJY0VA1itWblujtVz5wNwYN/u2mKkMeQcsCQVMYAlqYhTEFoW3nSTjp8BrGXhTTfp+LU2BRERZ0XEvRHRjYgHI+JjzfjqiPhiRHyz+XpGMx4R8dmI2BURnYh4a1u1qR2Hbrqtnjuf01//xupypLHX5hzwQeBXM/MC4CJgS0RcAFwD3J2Z5wF3N/sA7wbOa/5sBq5vsTZJKtdaAGfmvsz8arP9HPAQsB64HLipOe0m4H3N9uXAzdnzZWBVRKxrqz5JqjaSOeCImAPeAtwPrM3Mfc2hJ4C1zfZ64PG+b9vTjO3rGyMiNtO7Qmbjxo3tFS0VeenFg3S73cPGNm3axIoVK4oqUltaD+CIOB34Y+Djmfm9iFg8lpkZEcd1qzwztwHbAObn573Nrqnz3P49XPfYC6x9+CDQW8SydQvMz88XV6bl1moAR8Sp9ML3c5l5ezP8ZESsy8x9zRTD/mZ8L3BW37dvaMakmXP62rMXVxFqerUWwNG71L0ReCgzf7fv0HbgKuDa5usdfeMfiYhbgLcDB/qmKqSZ5ZTE9GrzCvhi4IPAzoj4ejP2m/SC99aIuBp4FLiiOXYn8B5gF/A88OEWa5MmhlMS06u1AM7M/wXEUQ5feoTzE9jSVj3SJHNKYjrZC0KSihjAklTEAJakIgawJBUxgCWpiAEsSUUMYEkqYgBLUhEDWJKKGMCSVMQAlqQiBrAkFTGAJamIASxJRQxgSSpiAEtSEQNYkoqM5LX0mj4LCwt0Op3F/W63S++lJpKGZQDrhHQ6HbZs3c7KdXMA7N15H6vOvZAza8uSJooBrBO2ct3c4nvKDuzbXVuMNIEMYGnC+Jr66WEASxPG19RPDwNYmkC+pn46+BiaJBUxgCWpiAEsSUUMYEkqYgBLUhEDWJKKGMCSVMQAlqQiBrAkFTGAJamIASxJRQxgSSpiMx4NxTdgSMvPANZQfAPG+LI/8OQygDU034AxnuwPPLkMYGkK2B94MnkTTpKKGMCSVMQAlqQiBrAkFTGAJamIT0HoiFx4IbXPANYRufBCap8BrKNy4YXULueAJamIASxJRVoL4Ij4/YjYHxHf6BtbHRFfjIhvNl/PaMYjIj4bEbsiohMRb22rLkkaF21eAf8BcNnA2DXA3Zl5HnB3sw/wbuC85s9m4PoW65Km2qHuaDt27GDHjh0sLCxUl6SjaO0mXGb+z4iYGxi+HPiHzfZNwJeA32jGb87ec05fjohVEbEuM/e1VZ80rfq7o9kZbbyN+imItX2h+gSwttleDzzed96eZuxlARwRm+ldJbNx48b2KpUmmN3RJkPZTbjmave4n+zPzG2ZOZ+Z82vWrGmhMkkajVEH8JMRsQ6g+bq/Gd8LnNV33oZmTJKm1qgDeDtwVbN9FXBH3/iHmqchLgIOOP8radq1NgccEZ+nd8Pt9RGxB/jXwLXArRFxNfAocEVz+p3Ae4BdwPPAh9uqS0dm74fp5PvixlubT0H84lEOXXqEcxPY0lYtWpq9H6aT74sbb/aC0CJ7P0wnn4gYXy5FlqQiBrAkFTGAJamIASxJRQxgSSpiAEtSEQNYkooYwJJUxIUYM2RwuTG4LFWqZADPkMHlxi5LlWoZwDOmf7mxpFrOAUtSEa+AZ1h/q0LbT0qjZwDPsP5WhbafnA32Bx4vBvCMO9Sq0PaTs8H+wOPFAJZmjP2Bx4c34SSpiFfA0gxzTriWASzNMOeEaxnA0oxzTriOc8CSVMQAlqQiBrAkFTGAJamIASxJRQxgSSriY2iSFrkwY7QM4CniK4f0SrkwY7QM4Cky+MqhZ/d8i4++q8sFF1wA2PNXw3FhxugYwFOm/5VDB/bt5rr/sXPxasaev9J4MYCnXP/VjD1/pfHiUxCSVMQAlqQiBrAkFTGAJamIASxJRQxgSSriY2gTbHDlmwsttNxcmtwuA3iCDa58c6GFltvg0uTB1ZVgIL8SBvCEG1z5Ji23wcU8/asr7RXxyhjAko6LvSKWjwEs6YQ5R/zKGMATxJtuGje2r3xlDOAxdqTA3XrPN1n5xnMAb7ppPBxrSsIe1cdmAI+RYQPXm24aV4NTEoO/w14hH84AHiNHe6zMwNWkGJySGPwd1uEM4DHjY2WadPagHp4BXMibapp1sz5HbACPkDfVNOucIz7cWAVwRFwGfAY4GbghM68tLmlZOcerWXe8c8TTfoU8NgEcEScDW4GfBvYAX4mI7ZnZPfZ3Hp/B/6A/+MEPADj11FMXx472H3ipX4alfna32+V1b3B+TLPteOaIl3rT9/H8/Vvq/OM9vhzGJoCBtwG7MvMRgIi4BbgcWNYA7nQ6fOhffZbTVr8BgGe+/SAnveZHOOMNGwF4/jtPcM0vvuuwZiOHdLtdrv38Xyx+7+C5g8cHf/Yz336Q1519AREBwN88/dec8v0X+M5pp5Xvj1MtftbZ/awH9u2m2/1hLA2usnv+2Sf5rZu+xRlv+MZx//1b6vxhjt/8735lWadHYlxu+kTE+4HLMvOXm/0PAm/PzI8MnLcZ2Nzs/ijwf0daaHteDzxdXUSRWf3ss/q5YfY++9OZedng4DhdAQ8lM7cB26rrWG4RsSMzZ+POw4BZ/eyz+rlhtj97v3F6I8Ze4Ky+/Q3NmCRNpXEK4K8A50XEORGxArgS2F5ckyS1ZmymIDLzYER8BPgzeo+h/X5mPlhc1ihN3bTKcZjVzz6rnxtm+7MvGpubcJI0a8ZpCkKSZooBLElFDOBiEXFWRNwbEd2IeDAiPlZd0yhFxMkR8bWI+NPqWkYpIlZFxG0R8XBEPBQR76iuaVQi4l80v+vfiIjPR8Srq2uqYgDXOwj8amZeAFwEbImIly/Dm14fAx6qLqLAZ4C7MvPHgDczI/8OImI98CvAfGb+OL0b7lfWVlXHAC6Wmfsy86vN9nP0/iKur61qNCJiA/AzwA3VtYxSRKwE/gFwI0BmLmTmd0uLGq1TgNdExCnAacBfF9dTxgAeIxExB7wFuL+4lFH5NPDrwEvFdYzaOcBTwH9upl9uiIjXVhc1Cpm5F/gd4DFgH3AgM/+8tqo6BvCYiIjTgT8GPp6Z36uup20R8V5gf2Y+UF1LgVOAtwLXZ+ZbgL8FrqktaTQi4gx6TbbOAd4IvDYi/mltVXUM4DEQEafSC9/PZebt1fWMyMXAz0bEbuAW4JKI+K+1JY3MHmBPZh76P53b6AXyLHgX8O3MfCozfwDcDryzuKYyBnCx6PWmvBF4KDN/t7qeUcnMT2bmhsyco3cT5p7MnIkrocx8Ang8In60GbqUZW67OsYeAy6KiNOa3/1LmZEbkEcyNkuRZ9jFwAeBnRHx9WbsNzPzzrqSNAIfBT7X9D15BPhwcT0jkZn3R8RtwFfpPQH0NWZ4WbJLkSWpiFMQklTEAJakIgawJBUxgCWpiAEsSUUMYEkqYgBLUpH/D8UsCiyYbvgdAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "s1 = np.random.normal(5, 1, 10000)\n",
    "sns.displot(s1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAD4CAYAAADSIzzWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAALs0lEQVR4nO3dX4jdZX7H8c9jpuYvtl2zKo3FqUyoSKVsm4ttF8pEo4mm296IeNEqFfTCElMplDYEMkLwqpRKLgqypUa6tBd2r0oIVdLrhcnuglsVM2jcbrr+i3az+auJTy+SSZNxkpwZZvI9ia/X1eQ5J898czzPe37zmwRb7z0AXH03VA8A8FUlwABFBBigiAADFBFggCIjc3ny6tWr++jo6CKNAnB92r9//8e996/PXJ9TgEdHRzM5OblwUwF8BbTW3ptt3S0IgCICDFBEgAGKCDBAEQEGKCLAAEUEGKCIAAMUEWCAIgIMUESAAYoIMEARAQYoIsAARQQYoIgAAxQRYIAiAgxQRIABiszp/wkHg9i1a1empqYWZe9Dhw4lSdasWbMo+yfJ2NhYtmzZsmj7wzQBZsFNTU3lRz9+M2dWfG3B915y/OdJkvdPLc5bd8nxTxZlX5iNALMozqz4Wk7c9dCC77v8rT1Jsih7X7g/XA3uAQMUEWCAIgIMUESAAYoIMEARAQYoIsAARQQYoIgAAxQRYIAiAgxQRIABiggwQBEBBigiwABFBBigiAADFBFggCICDFBEgAGKCDBAEQEGKCLAAEUEGKCIAAMUEWCAIgIMUESAAYoIMEARAQYoIsAARQQYoIgAAxQRYIAiAgxQRIABiggwQBEBBigiwABFBBigiAADFBHgedi1a1d27dpVPQZcVd73C2+keoBr0dTUVPUIcNV53y88V8AARQQYoIgAAxQRYIAiAgxQRIABiggwQBEBBigiwABFBBigiAADFBFggCICDFBEgAGKCDBAEQEGKCLAAEUEGKCIAAMUEWCAIgIMUESAAYoIMEARAQYoIsAARQQYoIgAAxQRYIAiAgxQRIABiggwQBEBBigiwABFBBigiAADFBFggCICDFBEgAGKXJUAHz58OM8880wOHz48r8fnuu9s6zPXpqamsnnz5uzbty+bNm3K+vXr88QTT2T//v158MEHs3HjxmzcuDEbNmzI+Ph47rvvvoyPj+eRRx7J22+/nc8//3yOrwJc+44cOZLx8fGMj4/n0UcfzcaNG8//enx8PA8//HCefPLJPP3005mcnMxDDz2UJ598Mvv27cv69evzwAMPZNOmTZmamvrS3tNndHJyMps3b87+/fuv2IVBzvrlDPLc+fZpEFclwLt3787rr7+el19+eV6Pz3Xf2dZnru3cuTPHjh3L888/n5MnT6b3nnfeeSc7duzIiRMncurUqZw6dSqnT59Okpw5cyZJ8uGHH+bEiRP54IMP5jQrXA/ee++98x+///77OXXq1EWPf/zxxzlw4EDeeOONTExM5Pjx4zlw4ECef/759N7z2Wef5eTJk9m5c+eX9p4+oxMTEzl27Fh27NhxxS4MctYvZ5DnzrdPg1j0AB8+fDh79+5N7z179+6d9Wr1co/Pdd/Z1meuTU5O5uDBg0lyPrDTjh49OvDnX4yviDCsjhw5ki+++GLg5194lmaes4MHD150FXzhGZ3+fUePHr1sFwY561e6sr3Sc+fbp0GNLOhus9i9e/f5/2hnzpzJyy+/nGeffXbgx+e672zrvfeL1iYmJhbkz/bUU0/l9ttvX5C9ridTU1O54bNePca83HDySKamfpGtW7dWjzJ03n333QXdb+fOnXnppZeSXHyeZ7pUFwY565frySDtmW+fBnXFK+DW2lOttcnW2uRHH30050/w2muvnf/qd/r06bz66qtzenyu+862PnNt0KvcK/n0008XZB/4Kpr+LjS5+NzOdKkuDHLWL9eTQZ473z4N6opXwL33F5O8mCTr1q2b82XNhg0bsmfPnpw+fTojIyO5//775/T4XPedbb33ftHasmXLFiTC3/72txf0q+H1YuvWrdn/zrV5j/yLZTdl7M5b88ILL1SPMnTuvffeOd2CuJLR0dHzH194bme6VBcGOeuX68kg7Zlvnwa16PeAH3/88dxww9lPs2TJkjz22GNzenyu+862PnNtoW5BDDorXA/uuOOOBd1v+/bt5z++8IzOdKkuDHLWL3dGB3nufPs0qEUP8M0335xNmzaltZZNmzbl5ptvntPjc913tvWZa+vWrTv/1Xdk5OJvAlatWjXw5x90Vrge3HTTTZeM5GwuPEszz9no6GjGxsbO//rCMzr9+1atWnXZLgxy1i93Rgd57nz7NKir8tfQHn/88dxzzz2X/Opxpcfnuu9s6zPXtm/fnpUrV2bbtm1ZtmxZWmu5884789xzz2X58uVZunRpli5dev6Ns2TJkiTJLbfckuXLl+fWW2+d06xwPbjwKvi2227L0qVLL3p89erVWbt2be6+++5MTExkxYoVWbt2bbZt25bWWm688cYsW7bsoqvfadNndGJiIitXrsxzzz13xS4MctYvZ5DnzrdPg2i9D35bd926dX1ycnLBh7jWTP+E3H3C2U3fAz5x10MLvvfyt/YkyaLsPb3/77oHPCvv+/lrre3vva+bue6fIgMUEWCAIgIMUESAAYoIMEARAQYoIsAARQQYoIgAAxQRYIAiAgxQRIABiggwQBEBBigiwABFBBigiAADFBFggCICDFBEgAGKCDBAEQEGKCLAAEUEGKCIAAMUEWCAIgIMUESAAYoIMEARAQYoIsAARQQYoIgAAxQRYIAiAgxQRIABiggwQBEBBigyUj3AtWhsbKx6BLjqvO8XngDPw5YtW6pHgKvO+37huQUBUESAAYoIMEARAQYoIsAARQQYoIgAAxQRYIAiAgxQRIABiggwQBEBBigiwABFBBigiAADFBFggCICDFBEgAGKCDBAEQEGKCLAAEUEGKCIAAMUEWCAIgIMUESAAYoIMEARAQYoIsAARQQYoIgAAxQRYIAiAgxQRIABiggwQBEBBigiwABFBBigiAADFBFggCIj1QNwfVpy/JMsf2vPIux7OEkWZe+z+3+S5NZF2RtmEmAW3NjY2KLtfejQ6STJmjWLFclbF3V+uJAAs+C2bNlSPQJcE9wDBigiwABFBBigiAADFBFggCICDFBEgAGKCDBAEQEGKCLAAEUEGKCIAAMUEWCAIgIMUESAAYoIMEARAQYoIsAARQQYoIgAAxRpvffBn9zaR0neW7xxhsLqJB9XDzGEvC6z87pcmtfm/93Re//6zMU5BfiroLU22XtfVz3HsPG6zM7rcmlemytzCwKgiAADFBHgL3uxeoAh5XWZndfl0rw2V+AeMEARV8AARQQYoIgAJ2mt/Xpr7T9ba2+01v6rtba1eqZh0lpb0lr7YWvt36tnGSattV9prb3SWnurtfZma+33qmcaBq21Z8+dox+31v6ltbaseqZhJcBnnU7yl733u5N8M8mft9buLp5pmGxN8mb1EEPohSR7e+93JfnteI3SWluT5Jkk63rvv5VkSZJHa6caXgKcpPf+s977D859/IucPUhraqcaDq2125NsTvKd6lmGSWvtl5P8QZJ/TJLe+2e99/8tHWp4jCRZ3lobSbIiyf8UzzO0BHiG1tpokm8k+X7xKMPi75P8VZIviucYNr+R5KMk/3Tu9sx3Wmsrq4eq1ns/lORvk/wkyc+S/Lz3/h+1Uw0vAb5Aa21Vkn9L8he99yPV81Rrrf1hkg977/urZxlCI0l+J8k/9N6/keRYkr+uHalea+1Xk/xxzn6B+rUkK1trf1I71fAS4HNaa7+Us/H9bu/9e9XzDIlvJfmj1trBJP+a5N7W2j/XjjQ0fprkp7336e+UXsnZIH/VbUjybu/9o97750m+l+T3i2caWgKcpLXWcvZe3pu997+rnmdY9N7/pvd+e+99NGd/kLKv9+5qJknv/f0k/91a+81zS/cleaNwpGHxkyTfbK2tOHeu7osfTl7SSPUAQ+JbSf40yeuttR+dW9vWe99TNxLXgC1JvttauzHJO0n+rHiecr3377fWXknyg5z920U/jH+SfEn+KTJAEbcgAIoIMEARAQYoIsAARQQYoIgAAxQRYIAi/wf1moUDMV/SHQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# for boxplots see https://en.wikipedia.org/wiki/Interquartile_range (or ask!)\n",
    "sns.boxplot(x=s1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Heavy-tailed\n",
    "Distributions with a small but non-negligible amount of observations with high values. Several probability distributions follow this pattern: https://en.wikipedia.org/wiki/Heavy-tailed_distribution#Common_heavy-tailed_distributions.\n",
    "\n",
    "We pick the lognormal here: https://en.wikipedia.org/wiki/Log-normal_distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.FacetGrid at 0x1253ec2b0>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAFgCAYAAABqo8hyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAXQUlEQVR4nO3df5Bd5X3f8fdXu9JKKxwERMNQYIo8ZpwyaRszig3G8bSmtoGmxk0x4KEgHLA84Ng4tEmgnmnavxq3mQCObX4YSMSE4h8EB+zaYIKADrUjImNsfpmgGBvEILTyGIzuhWW1+vaP++xytdaP1eqefS5736+ZO/d5nnPuOV90mc+e+9xzzo3MRJJUx6LaBUjSIDOEJakiQ1iSKjKEJakiQ1iSKhquXcCBOOWUU/LOO++sXYYkzRSzXfENfSS8bdu22iVI0gF5Q4ewJL3RGcKSVJEhLEkVGcKSVJEhLEkVGcKSVJEhLEkVGcKSVJEhLEkVGcKSVJEhLEkVGcKSVJEhLEkVDWQIZyatVgt/5FRSbQMZwu12m7OuupN2u127FEkDbiBDGGB4ydLaJUjS4IawJPUDQ1iSKjKEJakiQ1iSKjKEJakiQ1iSKjKEJakiQ1iSKjKEJakiQ1iSKjKEJakiQ1iSKjKEJakiQ1iSKjKEJakiQ1iSKjKEJakiQ1iSKjKEJakiQ1iSKjKEJakiQ1iSKjKEJakiQ1iSKjKEJakiQ1iSKjKEJakiQ1iSKjKEJamigQvhzKTVatUuQ5KAAQzhdrvNeV+4mx2TO2m1WmRm7ZIkDbCBC2GA4SVLmZwY5/xr76fdbtcuR9IAazSEI+L3I+KxiHg0Im6JiKURsSoiNkTEpoj4ckQsKeuOlP6msvyYJmsDGBpZ1vQuJGmvGgvhiDgS+CSwOjN/HRgCzgY+A1yRmW8Bfg5cUF5yAfDzMn5FWU+SFrSmpyOGgWURMQyMAs8D7wFuLcvXAR8s7dNLn7L85IiIhuuTpKoaC+HMfA74U+AZOuH7EvA94MXM3FFW2wwcWdpHAs+W1+4o6x82c7sRsTYiNkbExrGxsabKl6R50eR0xCF0jm5XAf8EWA6ccqDbzczrMnN1Zq5euXLlgW5Okqpqcjri3wBPZ+ZYZk4AtwEnASvK9ATAUcBzpf0ccDRAWX4w8LMG65Ok6poM4WeAEyJitMztngw8DtwLnFHWWQPcXtp3lD5l+fr0JF5JC1yTc8Ib6HzB9hDwSNnXdcAfAZdGxCY6c743lJfcABxWxi8FLmuqNknqF8P7XmXuMvOPgT+eMfxj4O27WfdV4ENN1iNJ/WYgr5iTpH5hCEtSRYawJFVkCEtSRYawJFVkCEtSRQMdwjtee9Vf2ZBU1UCHsCTVZghLUkWGsCRVZAhLUkWGsCRVZAhLUkWGsCRVZAhLUkWGsCRVZAhLUkWGsCRVZAhLUkWGsCRVZAhLUkWGsCRVZAhLUkWGsCRVZAhLUkWGsCRVZAhLUkWGsCRVZAhLUkWGsCRVZAhLUkUDHcKZSavVIjNrlyJpQA10CE9OjHPRugdpt9u1S5E0oAY6hAGGRpbWLkHSABv4EJakmgxhSarIEJakigxhSarIEJakigxhSarIEJakigxhSarIEJakigxhSarIEJakigxhSarIEJakigxhSarIEJakigxhSarIEJakigxhSarIEJakigxhSarIEJakigxhSarIEJakigxhSaqo0RCOiBURcWtE/CginoiIEyPi0Ii4OyKeKs+HlHUjIj4bEZsi4ocRcXyTtUlSP2j6SPgq4M7M/DXgXwJPAJcB92TmscA9pQ9wKnBseawFrm64NkmqrrEQjoiDgXcDNwBk5muZ+SJwOrCurLYO+GBpnw7clB1/B6yIiCOaqk+S+kGTR8KrgDHgLyLi+xFxfUQsBw7PzOfLOluAw0v7SODZrtdvLmO7iIi1EbExIjaOjY01WL4kNa/JEB4Gjgeuzsy3AS1en3oAIDMTyP3ZaGZel5mrM3P1ypUre1asJNXQZAhvBjZn5obSv5VOKL8wNc1QnreW5c8BR3e9/qgyJkkLVmMhnJlbgGcj4q1l6GTgceAOYE0ZWwPcXtp3AOeVsyROAF7qmraQpAVpuOHtfwK4OSKWAD8GPkIn+L8SERcAPwXOLOt+EzgN2AS0y7qStKA1GsKZ+TCwejeLTt7Nugl8vMl6JKnfDPwVc5lJq9Wi8zdAkubXwIfw5MQ45197P+12u3YpkgbQwIcwwNDIstolSBpQhrAkVWQIS1JFhrAkVWQIS1JFhrAkVWQIS1JFhrAkVWQIS1JFhrAkVWQI4/0jJNVjCOP9IyTVYwgX3j9CUg2GsCRVZAhLUkWGsCRVZAhLUkWGsCRVZAhLUkWGsCRVZAhLUkWzCuGIOGk2Y5Kk/TPbI+E/n+WYJGk/DO9tYUScCLwTWBkRl3Yt+hVgqMnCJGkQ7DWEgSXAQWW9N3WN/wI4o6miJGlQ7DWEM/N+4P6I+MvM/Ok81SRJA2NfR8JTRiLiOuCY7tdk5nuaKEqSBsVsQ/irwDXA9cBkc+VI0mCZbQjvyMyrG61EkgbQbE9R+3pEXBwRR0TEoVOPRiuTpAEw2yPhNeX5D7rGEnhzb8uRpMEyqxDOzFVNFyJJg2hWIRwR5+1uPDNv6m05kjRYZjsd8Ztd7aXAycBDgCEsSQdgttMRn+juR8QK4EtNFFRLZtJqtRgdHSUiapcjaUDM9VaWLWBBzRNPToxz/rX30263a5ciaYDMdk7463TOhoDOjXv+GfCVpoqqZWhkWe0SJA2Y2c4J/2lXewfw08zc3EA9kjRQZjUdUW7k8yM6d1I7BHityaIkaVDM9pc1zgQeBD4EnAlsiAhvZSlJB2i20xGfBn4zM7cCRMRK4G+BW5sqTJIGwWzPjlg0FcDFz/bjtZKkPZjtkfCdEXEXcEvpnwV8s5mSJGlw7Os35t4CHJ6ZfxARvwO8qyz6LnBz08VJ0kK3ryPhK4HLATLzNuA2gIj452XZv2uwNkla8PY1r3t4Zj4yc7CMHdNIRZI0QPYVwiv2sszLyyTpAO0rhDdGxEdnDkbEhcD3milJkgbHvuaEPwV8LSLO4fXQXQ0sAf59g3VJ0kDYawhn5gvAOyPiXwO/Xob/T2aub7wySRoAs72f8L3AvQ3XIkkDx6veJKkiQ1iSKjKEJakiQ1iSKjKEJamixkM4IoYi4vsR8Y3SXxURGyJiU0R8OSKWlPGR0t9Ulh/TdG2SVNt8HAlfAjzR1f8McEVmvgX4OXBBGb8A+HkZv6KsJ0kLWqMhHBFHAf8WuL70A3gPr/8ixzrgg6V9eulTlp9c1pekBavpI+ErgT8Edpb+YcCLmbmj9DcDR5b2kcCzAGX5S2X9XUTE2ojYGBEbx8bGelrsjtdepdVq9XSbkrQ3jYVwRPw2sDUze3qjn8y8LjNXZ+bqlStX9nLTkjTvZvvzRnNxEvCBiDgNWAr8CnAVsCIihsvR7lHAc2X954Cjgc0RMQwcTOe37CRpwWrsSDgzL8/MozLzGOBsYH1mnkPnHhRnlNXWALeX9h2lT1m+PjOzqfokqR/UOE/4j4BLI2ITnTnfG8r4DcBhZfxS4LIKtUnSvGpyOmJaZt4H3FfaPwbevpt1XgU+NB/1SFK/8Io5SarIEJakigxhSarIEJakigxhSarIEJakigxhSarIEJakigxhSarIEJakigxhSarIEJakigzhLplJq9XCO2hKmi+GcJfJiXEuWvcg7Xa7dimSBoQhPMPQyNLaJUgaIIawJFVkCEtSRQMVwlNfvElSvxioEG6325z3hbuZ3OnZD5L6w0CFMMDwEr94k9Q/Bi6EJamfGMKSVJEhLEkVGcKSVJEhLEkVGcKSVJEhLEkVGcKSVJEhLEkVGcKSVJEhLEkVGcKSVJEhLEkVGcKSVJEhLEkVGcKSVJEhLEkVGcK7MfVbdJn+DJKkZhnCM2QmY2NjnHXVnbTb7drlSFrgDOEZJifG+diNDxDDI7VLkTQADOHdGBpZVrsESQPCEJakigxhSarIEJakigxhSarIEJakigxhSarIEJakigxhSarIEJakigzhPdjx2qu0Wq3aZUha4AxhSarIEJakigzhPfCewpLmgyG8B5MT41y07kHvKSypUYbwXgyNLK1dgqQFzhCWpIoMYUmqqLEQjoijI+LeiHg8Ih6LiEvK+KERcXdEPFWeDynjERGfjYhNEfHDiDi+qdokqV80eSS8A/hPmXkccALw8Yg4DrgMuCczjwXuKX2AU4Fjy2MtcHWDtUlSX2gshDPz+cx8qLRfBp4AjgROB9aV1dYBHyzt04GbsuPvgBURcURT9UlSP5iXOeGIOAZ4G7ABODwzny+LtgCHl/aRwLNdL9tcxiRpwWo8hCPiIOCvgU9l5i+6l2XnSoj9uhoiItZGxMaI2Dg2NtbDSiVp/jUawhGxmE4A35yZt5XhF6amGcrz1jL+HHB018uPKmO7yMzrMnN1Zq5euXJlc8XjVXOSmtfk2REB3AA8kZl/1rXoDmBNaa8Bbu8aP6+cJXEC8FLXtEUVkxPjnH/t/V41J6kxww1u+yTgXOCRiHi4jP0X4E+Ar0TEBcBPgTPLsm8CpwGbgDbwkQZrm7WhkWW1S5C0gDUWwpn5ABB7WHzybtZP4ONN1SNJ/cgr5iSpIkN4H/xyTlKTDOF98Ms5SU0yhGfBL+ckNcUQlqSKDGFJqsgQlqSKDGFJqsgQngVPU5PUFEN4FjxNTVJTDOFZ8jQ1SU0whCWpIkNYkioyhCWpIkNYkioyhCWpIkNYkioyhCWpIkN4lna89iqtVqt2GZIWGEN4P3j5sqReM4T3Q7vd5qyr7vTyZUk9Ywjvp+ElS2uXIGkBMYQlqSJDeD85LyyplwzhWcpM2u22t7WU1FOG8CxNToxzyc0bmdyZ3tZSUs8YwvthaMQv5ST1liEsSRUZwpJUkSEsSRUZwpJUkSEsSRUZwpJUkSE8R9u3b2fr1q1eOSfpgBjCc5CZbNu2jTXX3MvY2JhBLGnODOE5mJwY55M3fZediZcwSzoghvAcLSpXz3kJs6QDYQgfIO+qJulAGMIHaOquamNjY/4GnaT9Zgj3gFMSkubKEJakioZrF7AQZCbbt28nMxkdHSUiapck6Q3CI+EemJwY56PXrmfN1fd5upqk/WII98iikaXe9F3SfjOEJakiQ1iSKjKEe8yLNyTtD0O4x9rtNmdddSetVsswlrRPhnADhpcspd1uc+aV3/Iua5L2yhDuoampiCkR4V3WJO2VIdxDkxPjfOzGBxh/9ZXpMJ66pNm5Ykm7Ywj32NDIMjLzl45+p+aKPSqW1M3LlhswOTHOJTdvZPHocobKJc3QmSsGpo+Sly9fXq1GSf3BEG7I1NVzkxPjnPu5bwMw8qZDapYkqQ85HTEPhkaWTU9TtFotdu7cyfbt29m+fTs7d+50rlgaYIbwPNjx2qvsnJxkcmKcNdfcxzPPPMO5n7uLM6/4Ftu2bXOuWBpghvB8W7SIT970XWJ4ZPrMiaHFIx4NSwPKEK5g6kdCp6Ynpo6Q93Zhh6e4SQuTIVzR5MQ4F15zDxMTO2DRItZccx9PP/00W7Zs4eWXX+bll19my5YtbNmyha1bt+522qI7nA1q6Y2nr86OiIhTgKuAIeD6zPyTyiU1blH3PYgXLeKj164nhhazaKjz93HytfHp/pKDVtBqtVi2bBntdnv6fOQLbniAL33y/bTbbT7yxf/LjRf+FsuXL58+Ba7dbvfkFz+m9uevh0i90zchHBFDwOeB9wKbgb+PiDsy8/G6lc2vRSNLWTS0mEVDQwBkxHR/vPUS535+Pdec/w4uWvcdJl55hRhazNKDD2VsbIyPfnE9QyMHce7nvs3Q4hE+/x9XA/CJWx7ixgt/i9HR0TnVFBGMjo6ybdu26ZCfCuLR0dHpo/OpdmZOL3vllVd2Ce2ZQT7VX7ZsGa+88souz7vb1szlU/uduZ9ue/rj0as/Knv6b9pTfza1zXZf6r3u2w8sX7688X/n6JePrhFxIvDfMvP9pX85QGb+jz29ZvXq1blx48ZZ76PVavEf/tffsHNyJ7DrUebM/t6W9fK1c1k3JycYWjLyS8uAXV772vZfsGh4McNLl023Y9Hr291bv9vQkhGu/d13sfaL64mhJbtst3t8qr1zMqdfc/G677Du4vdOH5W3Wi3WfOFu1l383un+xeu+wxfWvJO1X1zPleecwH/+6sPT/altXXH28fz+lx6a3ubU8qn9ztxPt61bt/KxGx/grz7x/l2Wd9dyIBfOzNzOvvp7e+2etg/sc1vqjVarxTlXfp0YWsLXLvuduf47zzq5+ymEzwBOycwLS/9c4B2Z+Xsz1lsLrC3dtwJP7ueufhXYdoDlNsXa5sba5sba5mY2tW3LzFNms7G+mY6Yrcy8Drhurq+PiI2ZubqHJfWMtc2Ntc2Ntc1Nr2vrp7MjngOO7uofVcYkacHqpxD+e+DYiFgVEUuAs4E7KtckSY3qm+mIzNwREb8H3EXnFLUbM/OxBnY156mMeWBtc2Ntc2Ntc9PT2vrmizlJGkT9NB0hSQPHEJakigYqhCPilIh4MiI2RcRl87TPGyNia0Q82jV2aETcHRFPledDynhExGdLfT+MiOO7XrOmrP9URKzpUW1HR8S9EfF4RDwWEZf0S30RsTQiHoyIH5Ta/nsZXxURG0oNXy5f4hIRI6W/qSw/pmtbl5fxJyPi/QdaW9nmUER8PyK+0U91le3+JCIeiYiHI2JjGav+npZtroiIWyPiRxHxRESc2A+1RcRby7/X1OMXEfGpealt6sYvC/1B58u+fwTeDCwBfgAcNw/7fTdwPPBo19j/BC4r7cuAz5T2acC36FxtcwKwoYwfCvy4PB9S2of0oLYjgONL+03APwDH9UN9ZR8HlfZiYEPZ51eAs8v4NcBFpX0xcE1pnw18ubSPK+/1CLCq/D8w1IN/u0uB/w18o/T7oq6y7Z8AvzpjrPp7Wra7DriwtJcAK/qltq4ah4AtwD+dj9oaDaB+egAnAnd19S8HLp+nfR/DriH8JHBEaR8BPFna1wIfnrke8GHg2q7xXdbrYZ2307l3R1/VB4wCDwHvoHOl0vDM95TOWTUnlvZwWS9mvs/d6x1APUcB9wDvAb5R9lO9rq5t/YRfDuHq7ylwMPA05YSAfqptRj3vA/7ffNU2SNMRRwLPdvU3l7EaDs/M50t7C3B4ae+pxsZrLx+T30bniLMv6isf+R8GtgJ30zlafDEzd+xmP9M1lOUvAYc1VNuVwB8CO0v/sD6pa0oC346I70XnMn/oj/d0FTAG/EWZyrk+Ipb3SW3dzgZuKe3GaxukEO5L2flzWfU8wYg4CPhr4FOZ+YvuZTXry8zJzPwNOkeebwd+rUYd3SLit4Gtmfm92rXsxbsy83jgVODjEfHu7oUV39NhOlNzV2fm24AWnY/4/VAbAGUu/wPAV2cua6q2QQrhfros+oWIOAKgPG8t43uqsbHaI2IxnQC+OTNv67f6ADLzReBeOh/zV0TE1EVG3fuZrqEsPxj4WQO1nQR8ICJ+AnyJzpTEVX1Q17TMfK48bwW+RucPWD+8p5uBzZm5ofRvpRPK/VDblFOBhzLzhdJvvLZBCuF+uiz6DmDqW9M1dOZip8bPK9+8ngC8VD4K3QW8LyIOKd/Ovq+MHZCICOAG4InM/LN+qi8iVkbEitJeRmeu+gk6YXzGHmqbqvkMYH05crkDOLucpbAKOBZ4cK51ZeblmXlUZh5D5/+h9Zl5Tu26pkTE8oh401SbznvxKH3wnmbmFuDZiHhrGToZeLwfauvyYV6fipiqodnaejWZ/UZ40PlG8x/ozC1+ep72eQvwPDBB50jgAjpzgvcATwF/Cxxa1g06N7b/R+ARYHXXdn4X2FQeH+lRbe+i8/Hqh8DD5XFaP9QH/Avg+6W2R4H/WsbfTCesNtH5yDhSxpeW/qay/M1d2/p0qflJ4NQevrf/itfPjuiLukodPyiPx6b+P++H97Rs8zeAjeV9/Rs6ZxD0S23L6XxKObhrrPHavGxZkioapOkISeo7hrAkVWQIS1JFhrAkVWQIS1JFhrAkVWQIS1JF/x84b4FarVTFzQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "s2 = np.random.lognormal(5, 1, 10000)\n",
    "sns.displot(s2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWMAAAD4CAYAAAA5FIfVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAPpElEQVR4nO3db2xV933H8c/XfwCDu8Q2EUoM1LEMxUiJ2gxtjVZV0UYysKYqivog0aRYWyakbQI3RPmLM0ziPMiURAM0rY32R0baum4d7UhkyHDbR3uQzrQJUAzmlrkUpwn0kpIlOMbGvz24v3tzfbF9fYnv8dfk/ZKse865v3N+H+ecfHzvuTeKhRAEAJhfFfMdAABAGQOAC5QxADhAGQOAA5QxADhQVcrg5cuXh6ampjJFAYAb05EjR34dQrhlpjEllXFTU5P6+/s/XSoA+Iwxs18UG8NtCgBwgDIGAAcoYwBwgDIGAAcoYwBwgDIGAAcoYwBwgDIGAAcoYwBwgDIGAAcoYwBwgDIGAAcoYwBwgDIGAAcoYwBwgDIGAAcoYwBwgDIGAAcoYwBwoKT/B95c2Lt3r1KplCRpeHhYktTY2Djl2JaWFm3dujWxbAAwXxIv41QqpbeOD+jq0npVXr4kSXp39NoYlZcvJh0NAOZN4mUsSVeX1mtkXZtqTvZKkkbWtV0zJvscAHwWcM8YABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABygjAHAAcoYABxIpIz37t2rvXv3JjFVonMBwFypSmKSVCqVxDSJzwUAc4XbFADgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5QxgDgAGUMAA5UzXeAuTY4OKiRkRHdc8898x1lVioqKjQxMaHq6mqNjY2pvr5eFy9eVF1dnS5duqT7779f+/fvl5nptttuUzqd1urVq7Vlyxbt2LFDo6Ojam5u1jPPPKMXX3xRZ8+e1ZUrV1RVVSUz05UrV7R69WrV1tZq+/bteuWVVxRC0GOPPaY9e/Zo27Ztkx537typhoYGSVI6ndaTTz6pd955R48//rheeuklPffcc+rp6dG2bdsmHSu7/Mgjj+jZZ5/VqlWr9MQTT+jll1/W1atXVVlZqe7ubklSZ2enzEzbt2+/Zs5C6XQ6N/7555+fNC6dTmvXrl0z7l+qwmMWW5/NMa5nHsy/7DmZ6t+NcrjhXhmPjIzMd4SSTExMSJLGxsYkSRcvXpQkvf/++5qYmND+/fslSSEEDQ8P6+OPP9bg4KC6uro0OjoqSTpz5oy6u7t1+vRpjY6OKoSgsbExXblyRZJ09uxZnThxQt3d3Tpx4oQGBgbU3d2tY8eOXfO4b9++XLaenh6lUildvnxZL7zwgj766CPt3LkzNz7/WNnlrq4ujYyMaHBwUN3d3RoYGNDg4KAGBga0b98+9fT0aGBgIJencM5C+eMLx/X09BTdv1SFxyy2PptjXM88mH/ZczKb63Qu3FBlvG3btvmOkJgPP/xw0vrQ0FDRffLHDA0NKYRwzeOhQ4eUTqeVTqfV29ubGz8+Pp6bNzt+quPm5yrM1NvbO+mYhXMWSqfTOnjwYG794MGDuXHpdFqHDh2acf9SFR4zlUrNuD5d5lLHzOa4SFb+OSp2nc6VRG5TDA8Pa2RkRB0dHUqlUqq4EoruU/HxB0ql/k8dHR2znufo0aOfJiYkXb16Vfv27VMIIVfAcyX76n+6OR999NFJ23t6eiZlGBsby43r6enJvauYbv9SFR6zu7t7xvXpMpc6ZjbHRbLyz1FWuc9N0VfGZrbFzPrNrP/ChQtlCQE/xsfHdfjwYfX19SU+Z6G+vj6F8Mkf7hBCblxfX1+uqKfbv1SFxxwaGppxfbrMpY6ZzXGRrPxzlFXuc1P0lXEI4VVJr0rShg0bir+knUJjY6Mkaffu3ero6NCRM+8V3WdiyW+ppXmFdu/ePet5FsqHdp5VVVXp3nvvVQhBBw4cSHTOQhs3btRrr72WK2Qzy43buHGjent7NT4+Pu3+pSo85sqVK3Xu3Llp16fLXCxXsXnm4nfBp5N/jrLKfW5uqHvGd95553xHWPAqKyv18MMPq729XVVVc3sXq7q6espjZucsVJihuro6N669vV0VFRUz7l+qwmN2dnbOuD5d5lLHzOa4SFb+Ocoq97m5ocp4z5498x0hMbW1tZPWm5qaiu6TP6apqUlmds3jpk2b1NDQoIaGBrW1teXGZ0uxtrY2N36q4+bnKszU1tY26ZiFcxZqaGjQ5s2bc+ubN2/OjWtoaNCmTZtm3L9UhcdsaWmZcX26zKWOmc1xkaz8c1TsOp0rN1QZS1JNTc18RyhJ9q9vdXW1JKm+vl6SVFdXp4qKCj3wwAOSMm/RGxsbtWTJEq1du1ZdXV1avHixJKm5uVmdnZ1as2aNFi9eLDNTdXW1Fi1aJElavXq11q9fr87OTq1fv16tra3q7OzUHXfccc1j/l/+9vZ2tbS0aOnSpdqxY4eWLVumXbt25cbnHyu73NXVpZqaGq1du1adnZ1qbW3V2rVr1dramnvF3dramstTOGeh/PGF49rb24vuX6rCYxZbn80xrmcezL/sOZnNdToXLP8DkmI2bNgQ+vv7S54k+42I/HvGI+vaVHMy8zWnkXVt1+xTc7JXv13iPePCuQDAAzM7EkLYMNOYG+6VMQAsRJQxADhAGQOAA5QxADhAGQOAA5QxADhAGQOAA5QxADhAGQOAA5QxADhAGQOAA5QxADhAGQOAA5QxADhAGQOAA5QxADhAGQOAA5QxADhAGQOAA5QxADhAGQOAA5QxADhAGQOAA5QxADhAGQOAA5QxADhAGQOAA5QxADhAGQOAA5QxADhAGQOAA5QxADhAGQOAA5QxADhAGQOAA5QxADhAGQOAA5QxADhQlcQkLS0tSUyT+FwAMFcSKeOtW7cmMU3icwHAXOE2BQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgAOUMQA4QBkDgANV8zFp5eWLqjnZq8rLaUlSzcneKcdIKxJOBgDzI/EybmlpyS0PD49LkhobpyrdFZPGAsCNLPEy3rp1a9JTAoB73DMGAAcoYwBwgDIGAAcoYwBwgDIGAAcoYwBwgDIGAAcoYwBwgDIGAAcoYwBwgDIGAAcoYwBwgDIGAAcoYwBwgDIGAAcoYwBwgDIGAAcoYwBwgDIGAAcoYwBwwEIIsx9sdkHSL65zruWSfn2d+84H8pbfQsu80PJKCy/zjZr38yGEW2YaUFIZfxpm1h9C2JDIZHOAvOW30DIvtLzSwsv8Wc7LbQoAcIAyBgAHkizjVxOcay6Qt/wWWuaFlldaeJk/s3kTu2cMAJgetykAwAHKGAAcKHsZm9kmMztlZikze6rc8xXJ8o9mdt7Mjudtqzezw2Z2Oj7Wxe1mZnti7qNmdlfePu1x/Gkzay9j3lVm9iMzO2FmPzOzDs+ZzWyJmf3YzN6OeXfF7beb2Zsx13fMbFHcvjiup+LzTXnHejpuP2Vmf1iOvHlzVZrZT83s9QWSd8jMjpnZW2bWH7e5vCbiPDeb2XfN7KSZDZjZ3V7zmtkX4j/X7M8HZvaNRPKGEMr2I6lS0s8lNUtaJOltSevLOWeRPF+VdJek43nb/lrSU3H5KUkvxuU2SQclmaQvS3ozbq+XdCY+1sXlujLlvVXSXXH5c5IGJa33mjnOWxuXqyW9GXP8m6QH4/ZvSvrzuPwXkr4Zlx+U9J24vD5eK4sl3R6vocoyXhfbJf2LpNfjuve8Q5KWF2xzeU3EuXok/VlcXiTpZs9583JXSnpX0ueTyFu2XyQGulvSG3nrT0t6upxzziJTkyaX8SlJt8blWyWdisvfkvRQ4ThJD0n6Vt72SePKnP0/Jd27EDJLWirpJ5J+V5n/Qqmq8JqQ9Iaku+NyVRxnhddJ/rgy5Fwp6QeSfl/S63F+t3nj8Yd0bRm7vCYk3STpfxW/LOA9b0HG+yT9d1J5y32bolHSL/PWz8VtnqwIIfwqLr8raUVcni77vPxO8S3xl5R5tek2c3zL/5ak85IOK/Mq8TchhPEp5s7lis9fktSQZF5JfyPpCUkTcb3BeV5JCpL+y8yOmNmWuM3rNXG7pAuS/ineCvp7M1vmOG++ByV9Oy6XPS8f4OUJmT9h7r7rZ2a1kv5D0jdCCB/kP+ctcwjhagjhi8q84vwdSevmN9H0zOyPJJ0PIRyZ7ywl+koI4S5JmyX9pZl9Nf9JZ9dElTK3Bv8uhPAlSR8p8zY/x1leSVL8nOBrkv698Lly5S13GQ9LWpW3vjJu8+Q9M7tVkuLj+bh9uuyJ/k5mVq1MEf9zCGH/QsgsSSGE30j6kTJv8282s6op5s7lis/fJCmdYN7fk/Q1MxuS9K/K3KrY7TivJCmEMBwfz0v6njJ/9LxeE+cknQshvBnXv6tMOXvNm7VZ0k9CCO/F9bLnLXcZ/4+kNfHT6UXKvOw/UOY5S3VAUvaTznZl7stmtz8cPy39sqRL8W3KG5LuM7O6+InqfXHbnDMzk/QPkgZCCK94z2xmt5jZzXG5Rpn72wPKlPLXp8mb/T2+LumH8VXHAUkPxm8v3C5pjaQfz3XeEMLTIYSVIYQmZa7NH4YQ/thrXkkys2Vm9rnssjLn8ricXhMhhHcl/dLMvhA3/YGkE17z5nlIn9yiyOYqb95y3gCPN67blPkWwM8l7Sj3fEWyfFvSrySNKfMX+xFl7vn9QNJpSX2S6uNYk/S3MfcxSRvyjvOnklLx50/KmPcryrwdOirprfjT5jWzpDsl/TTmPS7pr+L2ZmXKKaXM277FcfuSuJ6KzzfnHWtH/D1OSdqcwLVxjz75NoXbvDHb2/HnZ9l/p7xeE3GeL0rqj9fF95X5doHnvMuUecdzU962suflP4cGAAf4AA8AHKCMAcAByhgAHKCMAcAByhgAHKCMAcAByhgAHPh/4w+32hcqmX4AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.boxplot(x=s2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.FacetGrid at 0x1255419c0>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAFgCAYAAACFYaNMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAX4klEQVR4nO3df5Dc9X3f8efbkhFYFhLGiricxEitNG481LHdDcY+1+OgOgPENTTjUGdSW/GQqh1wRm4ySXDyRyYz6dSeSWLTUmg1yLVIbX4E8ysuQ0wB26QN2AsoZzBuewEp3FVCAiPJPyqB4N0/9qtjtTpxe9J997O3+3zM3Nx+P/vdvbdAet13P9/Pj8hMJEm994bSBUjSsDKAJakQA1iSCjGAJakQA1iSCllcuoBTcdFFF+W9995bugxJmk3M1Ligr4Cff/750iVI0klb0AEsSQuZASxJhRjAklSIASxJhRjAklSIASxJhRjAklSIASxJhRjAklSIASxJhRjAklSIASxJhRjAklTIgl6OUpoPhw8fptlsHtfeaDRYsmRJgYo0LAxgDb1ms8mW6+5k+ej66bYDUxNccyWMjY0VrEyDzgCWgOWj61m5/h2ly9CQsQ9YkgoxgCWpEANYkgoxgCWpkFpvwkXETuCHwCvAkcxsRMRbgFuAtcBO4PLMfDEiArgGuAT4CfBrmflYnfVJ3ZppqJrD1HSqejEK4uczs3374quB+zPzsxFxdXX8u8DFwIbq6z3A9dV3qbjOoWoOU9N8KDEM7VLgg9Xj7cA3aAXwpcCNmZnAwxGxIiJGMnN3gRql4zhUTfOt7gBO4OsRkcB/zsytwKq2UN0DrKoejwLPtr12smo7JoAjYjOwGeDcc8+tsXQtRPPVVfDqkZcZHx+fPh4fHycz5qVG6ai6A/j9mTkVET8F3BcR329/MjOzCueuVSG+FaDRaMzptRp889VVcHDPLq7deYhznmndp57a8RArNjTmvV4Nt1oDODOnqu97I+IO4HzguaNdCxExAuytTp8C1rS9fHXVJs3JfHUVLBtZN/0++ycnTvn9pE61DUOLiKURsezoY+AXgCeAu4FN1WmbgLuqx3cDn4iWC4AD9v9KGmR1XgGvAu5ojS5jMfCVzLw3Ir4D3BoRVwC7gMur8++hNQRtgtYwtE/WWJskFVdbAGfm08DPztD+ArBxhvYErqqrHknqN66GpqHTOVLCEQ4qxQDW0OkcKeEIB5ViAGsotY+UcISDSnExHkkqxACWpEIMYEkqxD5gDbTONR3AUQ/qHwawBlrnmg7gqAf1DwNYA699TQdw1IP6h33AklSIASxJhRjAklSIASxJhRjAklSIASxJhRjAklSIASxJhRjAklSIASxJhRjAklSIa0FIJ2GmVdYAGo0GS5YsKVCRFiIDWDoJM62ydmBqgmuuhLGxsYKVaSExgKWT1LnKmjRX9gFLUiEGsCQVYgBLUiEGsCQVYgBLUiEGsCQVYgBLUiEGsCQVYgBLUiEGsCQVYgBLUiEGsCQVYgBLUiEGsCQVYgBLUiGuB6wF6/DhwzSbzWPaxsfHyYxCFUlzYwBrwWo2m2y57k6Wj66fbpva8RArNjQKVvWamX5BgNsW6TUGsBa05aPrj9mVYv/kRMFqjjXTLwi3LVI7A1iqUecvCKmdN+EkqRADWJIKMYAlqRADWJIKMYAlqRADWJIKMYAlqRADWJIKqT2AI2JRRDweEV+rjtdFxCMRMRERt0TEaVX7kup4onp+bd21SVJJvbgC3gI81Xb8OeDzmbkeeBG4omq/Anixav98dZ4kDaxaAzgiVgO/CNxQHQdwIXBbdcp24LLq8aXVMdXzG6vzJWkg1b0WxBeA3wGWVcdnA/sz80h1PAmMVo9HgWcBMvNIRByozn++5hq1QHSuLubSk1roagvgiPgwsDczH42ID87j+24GNgOce+658/W2WgA6Vxfrp6UnpZNR5xXwGPCRiLgEOB04E7gGWBERi6ur4NXAVHX+FLAGmIyIxcBy4IXON83MrcBWgEajkTXWrz7UvrpYPy09KZ2M2vqAM/Mzmbk6M9cCHwMeyMxfBR4EPlqdtgm4q3p8d3VM9fwDmWnAShpYJdYD/l3g5oj4I+BxYFvVvg34s4iYAH5AK7SlBePVIy8zPj4+fWwftWbTkwDOzG8A36gePw2cP8M5h4Bf7kU9Uh0O7tnFtTsPcc4zrQ+W9lFrNu6IIc2jZSPr7KNW15yKLEmFGMCSVIhdEFIPdd6oA7epH2YGsNRDnTfq3KZ+uBnAUo+136jTcLMPWJIKMYAlqRADWJIKMYAlqRADWJIKMYAlqRADWJIKMYAlqRADWJIKMYAlqRADWJIKMYAlqRADWJIKMYAlqRADWJIKMYAlqRADWJIKMYAlqRADWJIKMYAlqRADWJIKMYAlqRADWJIKMYAlqZDFpQuQZnL48GGazeYxbePj42RGoYqk+WcAqy81m022XHcny0fXT7dN7XiIFRsaBauS5pcBrL61fHQ9K9e/Y/p4/+REwWqk+WcfsCQV4hWw+kJnn6/9vRoGBrD6Qmefr/29GgYGsPpGe5/vsPT3vnrkZcbHx49rbzQaLFmypEBF6iUDWCro4J5dXLvzEOc889rtmANTE1xzJYyNjRWsTL1gAEuFLRtZd8xoDw0PR0FIUiEGsCQVYgBLUiEGsCQVYgBLUiEGsCQVYgBLUiEGsCQVYgBLUiEGsCQV4lRkFeHyk5IBrEJcflKqsQsiIk6PiG9HxN9ExJMR8YdV+7qIeCQiJiLilog4rWpfUh1PVM+vras29Yejy0+uXP8Olr51tHQ5Us/V2Qd8GLgwM38WeCdwUURcAHwO+HxmrgdeBK6ozr8CeLFq/3x1niQNrNoCOFt+VB2+sfpK4ELgtqp9O3BZ9fjS6pjq+Y0RYaegpIFV6yiIiFgUETuAvcB9wN8C+zPzSHXKJHD0s+co8CxA9fwB4OwZ3nNzRDQjorlv3746y5ekWtUawJn5Sma+E1gNnA/8g3l4z62Z2cjMxsqVK0/17SSpmJ6MA87M/cCDwHuBFRFxdPTFamCqejwFrAGonl8OvNCL+iSphNqGoUXESuDlzNwfEWcAH6J1Y+1B4KPAzcAm4K7qJXdXx39dPf9AZmZd9Un9aqaNOt2kczDVOQ54BNgeEYtoXWnfmplfi4jvATdHxB8BjwPbqvO3AX8WERPAD4CP1Vib1Lc6N+p0k87B1VUAR8RYZv6P2draZeY48K4Z2p+m1R/c2X4I+OVu6tHC0jnrDZz5Nhs36hwO3V4B/wfg3V20ScfpnPUGznyTYJYAjoj3Au8DVkbEb7Y9dSawqM7CNFiOzno7av/kRMFqpP4w2xXwacCbq/OWtbUfpHWjTJJ0kl43gDPzm8A3I+JLmbmrRzVJ0lDotg94SURsBda2vyYzL6yjKEkaBt0G8J8D/wm4AXilvnIkaXh0G8BHMvP6WiuRpCHT7VTkv4iIKyNiJCLecvSr1sokacB1ewW8qfr+221tCfy9+S1HkoZHVwGcmevqLkSShk23U5E/MVN7Zt44v+VI6jTT4jzgAj2DoNsuiJ9re3w6sBF4DDCApZp1Ls4DLtAzKLrtgviN9uOIWEFrOUlJPeDiPIPpZBdk/zFgv7AknYJu+4D/gtaoB2gtwvMzwK11FSVJw6DbPuA/bnt8BNiVmZM11CNJQ6OrLohqUZ7v01oR7SzgpTqLkqRh0FUAR8TlwLdp7VhxOfBIRLgcpSSdgm67IH4f+LnM3AvTG27+d+C2ugqTpEHX7SiINxwN38oLc3itJGkG3V4B3xsRfwncVB3/c+CeekqSpOEw255w64FVmfnbEfFLwPurp/4a+HLdxUnSIJvtCvgLwGcAMvN24HaAiPiH1XP/tMbaJGmgzdaPuyozv9vZWLWtraUiSRoSswXwitd57ox5rEOShs5sAdyMiH/Z2RgRvw48Wk9JkjQcZusD/jRwR0T8Kq8FbgM4DfhnNdYlSQPvdQM4M58D3hcRPw+cVzX/t8x8oPbKJGnAdbse8IPAgzXXIklDxdlsklRItzPhpK4dPnyYZrM5fTw+Pk5mFKxo8My0T5x7xC08BrDmXbPZZMt1d7J8dD0AUzseYsWGRuGqBkvnPnHuEbcwGcCqxfLR9dN7mO2fnChczWByn7iFzz5gSSrEAJakQgxgSSrEAJakQgxgSSrEAJakQgxgSSrEAJakQgxgSSrEAJakQpyKrFPSufAOuPiO1C0DWKekc+EdcPEdqVsGsE5Z+8I74OI7UrfsA5akQgxgSSrEAJakQgxgSSqktgCOiDUR8WBEfC8inoyILVX7WyLivoj4P9X3s6r2iIh/HxETETEeEe+uqzZJ6gd1XgEfAX4rM98OXABcFRFvB64G7s/MDcD91THAxcCG6mszcH2NtUlScbUFcGbuzszHqsc/BJ4CRoFLge3VaduBy6rHlwI3ZsvDwIqIGKmrPkkqrSd9wBGxFngX8AiwKjN3V0/tAVZVj0eBZ9teNlm1db7X5ohoRkRz37599RUtSTWrfSJGRLwZ+Crw6cw8GPHaFNXMzIjIubxfZm4FtgI0Go05vVYaVK8eeZnx8fHj2huNBkuWLClQkbpRawBHxBtphe+XM/P2qvm5iBjJzN1VF8Peqn0KWNP28tVVm6RZHNyzi2t3HuKcZ177UHtgaoJrroSxsbGClen11BbA0brU3QY8lZl/2vbU3cAm4LPV97va2j8VETcD7wEOtHVVSJrFspF1x0wJV/+r8wp4DPg48N2I2FG1/R6t4L01Iq4AdgGXV8/dA1wCTAA/AT5ZY22SVFxtAZyZfwWcaE3CjTOcn8BVddUjSf3GmXCSVIgBLEmFGMCSVIgBLEmFGMCSVIgBLEmFGMCSVIgBLEmFGMCSVIgBLEmFGMCSVIgBLEmFGMCSVEjtO2JosBw+fJhmszl9PD4+TuaJFr2T9HoMYM1Js9lky3V3snx0PQBTOx5ixYZG4ao0k5m2KXKLov5iAGvOlo+un955Yf/kROFqdCKd2xS5RVH/MYClAeY2Rf3Nm3CSVIgBLEmFGMCSVIgBLEmFGMCSVIgBLEmFGMCSVIgBLEmFGMCSVIgBLEmFGMCSVIgBLEmFuBiPTqhz7V9w/V9pPhnAOqHOtX/B9X+l+WQA63W1r/0Lrv8rzSf7gCWpEANYkgoxgCWpEANYkgoxgCWpEANYkgoxgCWpEMcBS0Pi1SMvMz4+fkxbo9FgyZIlhSqSASwNiYN7dnHtzkOc80zrg++BqQmuuRLGxsYKVza8DGBpiCwbWXfMzEaVZR+wJBViAEtSIQawJBViAEtSId6E07TOBdhdfF2qlwGsaZ0LsLv4ulQvA1jHaF+A3cXXB9tMEzPAyRm9ZABLQ6pzYgY4OaPXagvgiPgi8GFgb2aeV7W9BbgFWAvsBC7PzBcjIoBrgEuAnwC/lpmP1VWbpBYnZpRV5yiILwEXdbRdDdyfmRuA+6tjgIuBDdXXZuD6GuuSpL5QWwBn5reAH3Q0Xwpsrx5vBy5ra78xWx4GVkTESF21SVI/6PU44FWZubt6vAdYVT0eBZ5tO2+yajtORGyOiGZENPft21dfpZJUs2ITMTIzgTyJ123NzEZmNlauXFlDZZLUG70O4OeOdi1U3/dW7VPAmrbzVldtkjSweh3AdwObqsebgLva2j8RLRcAB9q6KiRpINU5DO0m4IPAWyNiEvgD4LPArRFxBbALuLw6/R5aQ9AmaA1D+2Rddamlc9oxOPVY6rXaAjgzf+UET22c4dwErqqrFh2vc9oxOPVY6jVnwg2x9mnH4NRjqddcjlKSCjGAJakQA1iSCjGAJakQA1iSCnEUhKRpMy3S7gLt9TGAJU3rXKTdBdrrZQAPCTfcVLdcpL13DOAh4YabUv8xgIeIG25K/cVREJJUiAEsSYUYwJJUiAEsSYUYwJJUiAEsSYUYwJJUiAEsSYUYwJJUiAEsSYU4FVnSCc20PCW4ROV8MYAlnVDn8pTgEpXzyQCW9LpcnrI+BrCkOXHXjPljAEuaE3fNmD8GsKQ5a++W8EbdyTOAJZ0Sb9SdPANY0inzRt3JMYAHUOcGnOAmnFI/MoAHUOcGnOAmnFI/MoAHwExbzp/503//mI+EbsIp9R8DeAC45by0MBnAA8It56WFxwBegGbqcvAGm7TwGMALkF0O0mAwgBcouxzUz1wvojsGsKR553oR3TGAJdXC2XGzc0siSSrEK+A+57RiaXAZwH3OacUaBC5ZOTMDeAFoH/EAjnrQwuOSlTMzgCX1ROdNOYeqGcBFzdS/C8P3l1DDyaFqBnBRM/XvDuNfQg2vYR+qZgD3UDfLRnZ+LHPEg4bVMHxCNIB7qJs1HDo/ljniQcNqGD4hGsA91s0aDu0fyxzxoGHWOQJo0BjANXEChTQ33XS/DdrICQN4nszUv3vDt55mxRonUEjd6Kb7bdBGTvRVAEfERcA1wCLghsz8bOGSunai/l0nUEjd66b7bbaRE50XQy+99BIAp5122jHn9cOVc98EcEQsAv4j8CFgEvhORNydmd+bz5/Tzf+czrbZjuH4EQ2GrVS/mbokOj99Tu34JovefDbnrD9v+pxurpxn6kac79DumwAGzgcmMvNpgIi4GbgUmNcAbjabbPr9P2Hp2SMAPP/0Eyw640zOGjl3+pzOttmOj7atWPcOIlp9Vj9+fopFhw6xb+nS6XM62+o6p+TPHpY/Q7/XNwh/hm5es/uJ/8m/e/iHnDXyxPQ5R/8tzmamtSk6n/+TW+6bzoofv7Cb7f/2t+a1uyMyc97e7FRExEeBizLz16vjjwPvycxPdZy3GdhcHb4N+F8neMu3As/XVO7J6Kd6+qkW6K96+qkW6K96rOXEZqvn+cy8qLOxn66Au5KZW4Gts50XEc3M7Js7Xv1UTz/VAv1VTz/VAv1Vj7Wc2MnW008Lsk8Ba9qOV1dtkjSQ+imAvwNsiIh1EXEa8DHg7sI1SVJt+qYLIjOPRMSngL+kNQzti5n55Cm85azdFD3WT/X0Uy3QX/X0Uy3QX/VYy4mdVD19cxNOkoZNP3VBSNJQMYAlqZCBC+CI+GJE7I2IJ2Y/u/Za1kTEgxHxvYh4MiK2FK7n9Ij4dkT8TVXPH5asp6ppUUQ8HhFf64NadkbEdyNiR0QcvxBtb2tZERG3RcT3I+KpiHhvwVreVv03Ofp1MCI+XbCef1P9/X0iIm6KiNML1rKlquPJk/lvMnB9wBHxAeBHwI2Zed5s59dcywgwkpmPRcQy4FHgsvmeXj2HegJYmpk/iog3An8FbMnMh0vUU9X0m0ADODMzP1yqjqqWnUAjM4sP8I+I7cBDmXlDNSroTZm5v3BZR5cMmKI1SWpXgZ8/Suvv7dsz8/9FxK3APZn5pQK1nAfcTGsW70vAvcC/zsyu1yEYuCvgzPwW8IPSdQBk5u7MfKx6/EPgKWC0YD2ZmT+qDt9YfRX7DRwRq4FfBG4oVUM/iojlwAeAbQCZ+VI/hG9lI/C3JcK3zWLgjIhYDLwJ+L+F6vgZ4JHM/ElmHgG+CfzSXN5g4AK4X0XEWuBdwCOF61gUETuAvcB9mVmyni8AvwO8WrCGdgl8PSIeraa8l7IO2Af8l6p75oaIWDrbi3rkY8BNpX54Zk4Bfwz8HbAbOJCZXy9UzhPAP46IsyPiTcAlHDuZbFYGcA9ExJuBrwKfzsyDJWvJzFcy8520ZhqeX32M6rmI+DCwNzMfLfHzT+D9mflu4GLgqqo7q4TFwLuB6zPzXcCPgasL1TKt6gr5CPDnBWs4i9YiXeuAnwaWRsS/KFFLZj4FfA74Oq3uhx3AK3N5DwO4ZlVf61eBL2fm7aXrOar6SPsgcNwCIT0yBnyk6ne9GbgwIv5roVqA6asrMnMvcAetvr0SJoHJtk8nt9EK5NIuBh7LzOcK1vBPgGcyc19mvgzcDryvVDGZuS0z/1FmfgB4Efjfc3m9AVyj6qbXNuCpzPzTPqhnZUSsqB6fQWvt5e+XqCUzP5OZqzNzLa2PtQ9kZpErGYCIWFrdKKX6uP8LtD5i9lxm7gGejYi3VU0bmedlWU/Sr1Cw+6Hyd8AFEfGm6t/XRlr3VoqIiJ+qvp9Lq//3K3N5fd9MRZ4vEXET8EHgrRExCfxBZm4rVM4Y8HHgu1W/K8DvZeY9heoZAbZXd7LfANyamcWHf/WJVcAd1XrOi4GvZOa9Bev5DeDL1cf+p4FPFqzl6C+lDwH/qmQdmflIRNwGPAYcAR6n7LTkr0bE2cDLwFVzvVk6cMPQJGmhsAtCkgoxgCWpEANYkgoxgCWpEANYkgoxgCWpEANYkgr5/xwOkyKNgfiBAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Why \"lognormal\"?\n",
    "\n",
    "sns.displot(np.log(s2))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Box plots\n",
    "\n",
    "<img src=\"figures/eda-boxplot.png\" width=\"800px\" heigth=\"800px\">"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outliers, missing values\n",
    "\n",
    "An *outlier* is an observation far from the center of mass of the distribution. It might be an error or a genuine observation: this distinction requires domain knowledge. Outliers infuence the outcomes of several statistics and machine learning methods: it is important to decide how to deal with them.\n",
    "\n",
    "A *missing value* is an observation without a value. There can be many reasons for a missing value: the value might not exist (hence its absence is informative and it should be left empty) or might not be known (hence the value is existing but missing in the dataset and it should be marked as NA).\n",
    "\n",
    "*One way to think about the difference is with this Zen-like koan: An explicit missing value is the presence of an absence; an implicit missing value is the absence of a presence.*"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary statistics\n",
    "A statistic is a function of a collection of observations, or otherwise stated a measure over a distribution. \n",
    "\n",
    "A statistic is said to be *robust* if not sensitive to outliers.\n",
    "\n",
    "* Not robust: min, max, mean, standard deviation.\n",
    "* Robust: mode, median, other quartiles.\n",
    "\n",
    "A closer look at the mean:\n",
    "\n",
    "$\\bar{x} = \\frac{1}{n} \\sum_{i}x_i$\n",
    "\n",
    "And variance (the standard deviation is the square root of the variance):\n",
    "\n",
    "$Var(x) = \\frac{1}{n} \\sum_{i}(x_i - \\bar{x})^2$\n",
    "\n",
    "The mean, the median, etc. are measures of location (e.g., the typical value); the variance is a measure of dispersion."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"figures/2560px-Comparison_mean_median_mode.svg.png\" width=\"400px\" heigth=\"400px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.002529505202645\n",
      "243.78623044943689\n"
     ]
    }
   ],
   "source": [
    "# Not robust: min, max, mean, mode, standard deviation\n",
    "\n",
    "print(np.mean(s1)) # should be 5\n",
    "print(np.mean(s2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.002529505202645"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(s1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.0047858671954035\n",
      "148.1832512562982\n"
     ]
    }
   ],
   "source": [
    "# Robust: median, other quartiles\n",
    "\n",
    "print(np.quantile(s1, 0.5)) # should coincide with mean and mode\n",
    "print(np.quantile(s2, 0.5))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions\n",
    "\n",
    "* Calculate the min, max, mode and sd. *hint: explore the numpy documentation!*\n",
    "* Calculate the 90% quantile values.\n",
    "* Consider our normally distributed data in s1. Add an outlier (e.g., value 100). What happens to the mean and mode? Write down your answer and then check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>annual_salary</th>\n",
       "      <th>a_age</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>7870.000000</td>\n",
       "      <td>9303.000000</td>\n",
       "      <td>9645.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>5.916921</td>\n",
       "      <td>14.266688</td>\n",
       "      <td>5.005694</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>6.985214</td>\n",
       "      <td>2.902770</td>\n",
       "      <td>1.462343</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.166667</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.083333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>4.000000</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6.000000</td>\n",
       "      <td>16.000000</td>\n",
       "      <td>6.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>180.000000</td>\n",
       "      <td>50.000000</td>\n",
       "      <td>15.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       annual_salary        a_age       length\n",
       "count    7870.000000  9303.000000  9645.000000\n",
       "mean        5.916921    14.266688     5.005694\n",
       "std         6.985214     2.902770     1.462343\n",
       "min         0.166667     1.000000     0.083333\n",
       "25%         3.000000    12.000000     4.000000\n",
       "50%         4.000000    14.000000     5.000000\n",
       "75%         6.000000    16.000000     6.000000\n",
       "max       180.000000    50.000000    15.000000"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's explore our dataset\n",
    "df_contracts[[\"annual_salary\",\"a_age\",\"length\"]].describe()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Relating two variables\n",
    "\n",
    "### Covariance\n",
    "\n",
    "Measure of association, specifically of the joint linear variability of two variables:\n",
    "\n",
    "<img src=\"figures/covariance.png\" width=\"400px\" heigth=\"400px\">\n",
    "\n",
    "Its normalized version is called the (Pearson's) correlation coefficient:\n",
    "\n",
    "<img src=\"figures/pearson.png\" width=\"400px\" heigth=\"400px\">\n",
    "\n",
    "Correlation is helpful to spot possible relations, but is of tricky interpretation and is not exhaustive:\n",
    "\n",
    "<img src=\"figures/800px-Correlation_examples2.svg.png\" width=\"700px\" heigth=\"&00px\">\n",
    "\n",
    "See: https://en.wikipedia.org/wiki/Covariance and https://en.wikipedia.org/wiki/Pearson_correlation_coefficient.\n",
    "\n",
    "*Note: correlation is not causation!*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>annual_salary</th>\n",
       "      <th>a_age</th>\n",
       "      <th>length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>annual_salary</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.205404</td>\n",
       "      <td>-0.361611</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>a_age</th>\n",
       "      <td>0.205404</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-0.430062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>length</th>\n",
       "      <td>-0.361611</td>\n",
       "      <td>-0.430062</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               annual_salary     a_age    length\n",
       "annual_salary       1.000000  0.205404 -0.361611\n",
       "a_age               0.205404  1.000000 -0.430062\n",
       "length             -0.361611 -0.430062  1.000000"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_contracts[[\"annual_salary\",\"a_age\",\"length\"]].corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='a_age', ylabel='annual_salary'>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA/Y0lEQVR4nO29e3xU5bX//1lzzySZIYSQhGtAbpJEUNOAVqyitXgD2lKt37aoYPn1HC2e3qynvx5RS/3W2tqqv7YWbxWtBasepdZyar33VEHwBhGVi4BgboSQ+9zX74+Z2WQyzx6ykz2ZzGS9Xy9ezDyzZz9rz0z2ep51JWaGIAiCIACAJdMCCIIgCMMHUQqCIAiChigFQRAEQUOUgiAIgqAhSkEQBEHQsGVagMEyZswYrqioyLQYgiAIWcX27duPMHNJ3/GsVwoVFRXYtm1bpsUQBEHIKojogGpczEeCIAiChigFQRAEQUOUgiAIgqAhSkEQBEHQEKUgCIIgaKRVKRDRg0TUREQ7e42NJqLniWh37P+i2DgR0d1EtIeI3iOi09IpmzA4IhHGvuZOvL73CPY1dyISkcKKgpALpHun8AcAi/qM3QjgBWaeDuCF2HMAuBDA9Ni/VQB+l2bZhAESiTA21zXgortfwxX3bcFFd7+GzXUNohgEIQdIq1Jg5lcBHO0zvATAw7HHDwNY2mt8PUd5A8AoIipPp3zCwNjf0oXvPv4OfMEIAMAXjOC7j7+D/S1dGZZMEITBkgmfQikz18ceNwAojT0eD+CTXscdio0lQUSriGgbEW1rbm5On6SCksZ2n6YQ4viCETR1+DIkkSAIZpFRRzNHO/wYtjkw8zpmrmHmmpKSpCxtIc2Uelxw2RN/Oi67BWMLXRmSSBAEs8iEUmiMm4Vi/zfFxg8DmNjruAmxMWGYUVGcjzsvm6spBpfdgjsvm4uK4vwMSyYIwmDJRO2jTQCuBPCz2P/P9Bq/jog2AJgHoK2XmUkYRlgshEWVZZi1egGaOnwYW+hCRXE+LBbKtGiCIAyStCoFIvoTgHMAjCGiQwDWIKoMHieilQAOALgsdvhzAC4CsAdAN4Cr0ymbMDgsFsLUkgJMLSnItCiCIJhIWpUCM1+h89J5imMZwLXplEcQBEFIjWQ0C4IgCBqiFARBEAQNUQqCIAiChigFQRAEQUOUgiAIgqAhSkEQBEHQEKUgCIIgaIhSEARBEDREKQiCIAgaohQEQRAEDVEKgiAIgkYmqqQKI5BIhLG/pQuN7T6UeoZ/VdVsk1cQzEKUgpB24j2d4y084/0XFlWWDcsbbbbJKwhmIuYjIe1kW0/nbJNXEMxElIKQdrKtp3O2ySsIZiJKQUg72dbTOdvkFQQzEaUgpJ1s6+mcbfIKgplQtOFZ9lJTU8Pbtm3LtBjCCYhH82RLT+dsk1cQjEJE25m5pu+4RB8JQ0K29XTONnkFwSzEfCQIgiBoiFIQBEEQNEQpCIIgCBqiFARBEAQNUQqCIAiChigFQRAEQUOUgiAIgqAhSkEQBEHQEKUgCIIgaIhSEARBEDREKQiCIAgaohQEQRAEjYwpBSL6DhHVEdFOIvoTEbmIaAoRbSGiPUS0kYgcmZJPyDyRCGNfcyde33sE+5o7EYlkd0VfQcgGMqIUiGg8gNUAapi5CoAVwFcB3A7gV8w8DUArgJWZkE/IPPE+yRfd/RquuG8LLrr7NWyuaxDFIAhpJpPmIxuAPCKyAXADqAewEMATsdcfBrA0M6IJmUb6JAtCZsiIUmDmwwB+AeAgosqgDcB2AMeYORQ77BCA8ar3E9EqItpGRNuam5uHQmRhiJE+yYKQGTJlPioCsATAFADjAOQDWNTf9zPzOmauYeaakpKSNEkpZBLpkywImSFT5qPzAXzMzM3MHATwFIDPAhgVMycBwAQAhzMkn5BhpE+yIGSGTLXjPAhgPhG5AfQAOA/ANgAvAVgGYAOAKwE8kyH5hAxjsRAWVZZh1uoF0idZEIaQjCgFZt5CRE8AeAtACMDbANYB+CuADUS0Njb2QCbkE4YH0idZEIaeTO0UwMxrAKzpM7wPQG0GxBEEQRAgGc2CIAhCLzK2UxCEExGJMPa3dKGx3YdSj/gUBGEoEKUgDEviGc3xBLZ49NGiyjJRDIKQRsR8JAxLJKNZEDKDKAVhWCIZzYKQGUQpCMMSyWgWhMwgSkEYlkhGsyBkBnE0ZzG5HJ0jGc2CkBlEKWQpIyE6RzKaBWHoEfNRliLROYIgpANRClmKROcIgpAOxHyUpcSjc3orhnRE5+j5LYbCn5HLPhNBGK6IUshS4tE5fX0KZkbn6PktLji5FH/f1ZhWf8ZI8JkIwnCEmLO7EXpNTQ1v27Yt02JkhPhKOl3ROfuaO3HR3a8l7UY2rpqPy9e9kTT+3OoFpjmF9eY2cw5BGMkQ0XZmruk7Lj6FLCYenTN/6hhMLSkwfQWt57eob0u/P0N8JoKQGUQpCLroZRWXe/PSnm0sGc2CkBlEKQi66GUVV5Z70p5tLBnNgpAZxKcgpETPb5Fuf0aquQVBGDx6PgWJPhJSopdVPBTZxpLRLAhDjygFIeOEQhHU1behvs2Hcm8eKss9sNkskqcgCBlAlIKQUUKhCJ5+9zB+/PROLR9h7dIqLK4eh3982CR5CoIwxPTb0UxE1ekURBiZ1NW3aQoBiIad/vjpnXjv0zap7SQIGcBI9NFviWgrEf07EXnTJpEwotDLeWiQPAVByAj9VgrMvADA1wBMBLCdiB4jos+nTTJhRKCX81AmeQqCkBEM5Skw824APwbwQwCfA3A3EX1ARF9Kh3BC7lNZ7sHapVUJ+Qhrl1bhlHFeyVMQhAzQ7zwFIjoFwNUALgbwPIAHmPktIhoH4HVmnpw+MfWRPIXsJx591NDmQ5nXhcpyb0L0keQpCIL5mJGncA+A+wH8iJl74oPM/CkR/dgEGYURis1mwZyJRZgzMXFc8hQEYejpl1IgIiuAw8z8iOp1vXFBEAQhu+iXT4GZwwAmEpEjzfIIgiAIGcSI+ehjAP9LRJsAaMHizHyn6VIJgiAIGcGIUtgb+2cBUDjYiYloFKI+iioADGAFgA8BbARQAWA/gMuYuXWwcwknxqySEj5fCDvq29DQ7keZx4nqci9crtQ/M7259cpfZBIpvSHkOv1WCsx8i8lz3wVgMzMvi5ml3AB+BOAFZv4ZEd0I4EZEw1+FNGJW60ufL4RNO+px06bjJStuXVyFxdXluopBb+7zZ47Fph2fJpW/WDpnfMYUg7QIFUYCRspclBDRHUT0HBG9GP83kEljGdFnA3gAAJg5wMzHACwB8HDssIcBLB3I+QVj7G/pMqWkxI76Nk0hxM9z06ad2FHfZnju9z5Vl7+oS3GudGPW5yQIwxkjS64/AvgAwBQAtyBq3nlzgPNOAdAM4CEiepuI7ieifAClzFwfO6YBQKnqzUS0ioi2EdG25ubmAYogxDGr9WVDu195nsZ2v+G59cpcNLRlrsyFtAgVRgJGlEIxMz8AIMjMrzDzCgALBzivDcBpAH7HzKci6ri+sfcBHM2qU2bWMfM6Zq5h5pqSkpIBiiDEMav1ZZnHqTxPqcdpeO5ynfEyb+bKXEiLUGEkYEQpBGP/1xPRxUR0KoDRA5z3EIBDzLwl9vwJRJVEIxGVA0Ds/6YBnl8wgFmtL6vLvbh1cWLJilsXV6G6XL9+ot7c1eO8yvIXlSnOlW6kRagwEjBS5uISAK8hWhDvHgAeALcw86YBTUz0GoBrmPlDIroZQPwvq6WXo3k0M9+Q6jxS5sIczCopEY8+amz3o9Rg9FHfuQOBMN77tA0N7T6Ue1yoHueFw2Ed6CWagpTeEHIFvTIXGevRTERzEQ1JdQDYh2hdJQuAxwFMAnAA0ZDUo6nOI0ohN5FIH0FILwOufURE90DHtg8AzLx6IAIx8zsAkgQCcN5AzifkFnqRPrNWL5BaSIKQRvqTpyDLcGHISRXpI0pBENLHCZUCMz98omMEwWzikT69FcOJIn0k21gQBk+/M5qJqATR7OLZALS/TGYeaFiqIOgSj/Tp61PQi/QRH4QgmIOR2kd/RLQu0cUAvgXgSkQT0IQRiN6q3KzVusVCWFRZhlmrF/Qr0kd8EIJgDkaUQjEzP0BE1zPzKwBeIaKBZjQLWYzeqvyCk0vx912Npq3WjTTZER+EIJhDppLXhCxGb1VeV9+WsdpAkm0sCOZgRCmsjRWy+x6A7yOaY/CdtEglDGv0VuX1bZmrDSTZxoJgDkZKZz8be9gG4Nz0iCNkA3qRQeXePMMRQ2Zh1AchCIIaI6Wzf05EHiKyE9ELRNRMRF9Pp3DC8ERvVV5Z7jF1tR6JMPY1d+L1vUewr7kTkUjq7Pu4D2L+1DGYWlIgCkEQBoCR2kfvMPNcIvoigEsAfBfAq8w8J50Cnggpc5EZ9GoAmVUbSEJMBSG9DLjMheLYiwH8mZnbiOSPc6SiFxlkJGIoFRJiKgiZwYhSeJaIPgDQA+DfYsls0l0kRzCaX2A0TyHV+VWvNbb7UOR24EunTUB87fHk9kNo6vChojg/rTkSgjCSMeJovpGIfg6gjZnDRNSNaPtMAAARfZ6Zn0+HkEJ6MWqqMZqnkCp/AYDyXDNLC7D8jMm464Xd2vj1501HSYFzSHIkBGGkYlrpbCJ6i5lPM+VkBhCfwuDZ19yJi+5+LSlq6DkdU43e8RtXzcfl697o9/hzqxcAgPJcj66ch68/sKXf46nmEHOTICSj51MwkqdwwjlMPJcwhBjtPWw0TyFV/oJZ58pkjoQg5BJmKoXMdOsRBo3RbGDdvsqxPIX+jo8tdOmeS6/fs954uVcymgXBDMxUCkKWYjQb2GieQqr8Bd0ezeVe3LqkT7/nJdF+z+o51OOS0SwIxjDTp/AUM3/JlJMZQHwK5mA0v8BonkKq85+oR3M8muiUWI/mdOdICMJIYMA9moko5Y2emZ8apGyDQpSCIAiCcQaTvHZpitcYQEaVgpC7SN6BIAw9/WnHefVQCCIIvZEyF4KQGYxkNIOILgZQicR2nLeaLZSQvZi1upcyF4KQGYz0aL4XgBvRstn3A1gGYGua5BKyEDNX99JJTRAyg5GQ1DOZeTmAVma+BcAZAGakRywhG9Fb3Q+k85p0UhOEzGBEKfTE/u8monGItucsN18kIVsxmhmdiklFbqxdmpinsHZpFSYVuU2RVRAENUarpI4CcAeAtxCNPLo/HUIJ2YleR7aBrO4Ptnbjnhd3Y+VZU0EEMAP3vLgbp00qEvORIKQRI1VSfxJ7+CQRPQvAxcxt6RFLyEbi2cl9fQoDySpubPfhQEsPfvPSnoRx8SkIQnox4mherhgDM683VyQhWzGzT7KZuw5BEPqPEfPRZ3o9dgE4D1EzkigFQcOszmtm7joEQeg/RsxH3+79POZf2GC2QIIAmLvrEASh/xhKXutDF4ApZgkiCH0xa9chCEL/MeJT+AuO90ywAJgN4PHBTE5EVgDbABxm5kuIaAqiu49iANsBfIOZA4OZQziO0T7JvSuVNrT7UO5xoTpWqdTnC2FHfRsa2v0o8zhRXe6Fy2VDd08AOxs60NjuR6nHiaqyQrjzHLrjqeYOhSKoq29DfZsP5d48VJZ7YLNZDB+fK0gtKGEoMLJT+EWvxyEAB5j50CDnvx7ALgCe2PPbAfyKmTfEMqhXAvjdIOcQkDrbGFD3SV44vQSbdtbjpmd2auO3LqnCxZVj8dedTbhpU6/xxVVYVFWCzTubk8YvqCrB3xXjl1SVwuW0K+c+f+ZYbNrxKX789PH3rF1ahcXV4/CPD5v6ffzSOeNzQjFILShhqDCtn4LhiYkmAHgYwE8BfBfRaqzNAMqYOUREZwC4mZm/kOo8Ujq7f6Tqwwyo+yQ/sqIW33hwa9L4+hW1WG7S+JgCpyk9mlP1bp4zsWjAn9twwWgfbUE4EYPu0UxEXyKi3UTURkTtRNRBRO2DkOnXAG4AEP+VFwM4xsyh2PNDAMbryLKKiLYR0bbm5uZBiDBySJVtrPdaQ7tfOd5o4rj+3CaNt+VGj2Yzs8UFIRVG9tU/B7CYmb3M7GHmQmb2nPBdCojoEgBNzLx9IO9n5nXMXMPMNSUlJQM5xYgjVS0ho32SS00c1+33rCuTweO9qfMaQqEI3v2kFZt31uPdT44hFIqkPD5TSC0oYagwohQamXmXSfN+FsBiItqPqGN5IYC7AIwiorifYwKAwybNN+JJ1YfZaJ/k2WX5uHVxn/HFVZipMz5DZ7yqrFB/7nFeZe2jU8apezHrHV9Z7tX9TEKhCJ5+9zAuX/cGvvXoW7h83et4+t3Dw1IxGO2jLQgDpd8+BSK6C0AZgKcB+OPjg23HSUTnAPh+LProzwCe7OVofo+Zf5vq/eJT6D9m9Uk+1NaDG554B8vPnIqeQAh5DhvW/2sffr5sLsoKHAOOPuo7dzyaqKHNhzKvC5Xl3oToo/4er8e7n7Ti8nVvZI0fQnpQC2YymHaccTwAugFc0GvM7HacPwSwgYjWAngbwAMmnnvEkyruX+81h8OKmorRCWON7T5sO9CGbQfeThiP1yWqnVKcdH53nkM5nmpum82COROLMGfi4I7Xo75N3w/R33MMJZK3IQwFRjKa09KWk5lfBvBy7PE+ALXpmEdIjZEY+IHUJRpIjkS64/LLvXnK6ziRH0IQchkjyWslAL4JoKL3+5h5hfliCUOJ0Rj4eK+DvjkBer0OBpIjccHJpfj7rsa0xuVXlnuU15HKDyEIuY4Rn8K/ALyGaKZxOD7OzE+mR7T+IT6FwWM0Bn5fcyd+9rc6fG3+FLR2BTE6345H3/gYN15YiYri/KTV/f6WLlz9h6245JTxoNj9/C/vHsZDV0U3haq5N66ar7T3P7d6gXKOge4sjPohch3Jmh45mOFTcDPzD02USRgmGO2H3NYTwLkzy/H/PLJdW2GvuaQSXf6gctVf7nXi8ppJuPvF3dr46oXTcbTLj2CYlXPr2fsb2334oKHDtJ2FUT9ELiNZ0wJgLCT1WSK6KG2SCBnDaAx8OALc8mxdQi/m+HNVj+ZwBJpCiI/f/eJu2K0W/TwFr3rc7bAq56irbzOtP/RIxcwe20L2YkQpXI+oYugxKaNZGCYYjYE/0qmTodyhHtc7vjsQ1p27slydjxAIRwztLIZzxm8kwtjX3InX9x7BvuZORCKZKTkTR7KmBcBY9FEhEY0GMB3RJjtCjmC0d8H4UeqonXKdqCS9KJ9Sjyvl3Krx/S1dhuYYrhm/w9FUI93uBMBY7aNrALwCYDOAm2P/35QesYShJh4DP3/qGEwtKUh5YyIA1583PWEVf/150+F2WHRW/Z6UOxG9uVXj+juL1HMMN4ajqUaypgXAWPTRDkRbcr7BzHOJaBaA25j5S+kU8ERI9NHQs3lnPX770h5cc/ZJ6AmE4HbYcN+re3HtudPw+dllyqzbgWTjnih/QS+jOdpPYXhHEr2+9wiuuG9L0viGVfMwf+qYDEgURbKmRw5mRB/5mNlHRCAiJzN/QEQzTZRRyBImFOXhwupy3PDEu5rp4/rzpmN8UZ5u1q3RbNwTmVf6nisS4bTnNZjJcDXVSNa0YGQZdSjWl/lpAM8T0TMADqRDKGF447JZcdcLidFEd72wGy6bVfc9Rp2qRs0rw9Eckwox1QjDFSOO5i/GHt5MRC8B8CLqVxBGGM060URHuvyYhsKk4wfiVDWaO2H0+Exj1LkvCEOFEfORBjO/YrYgQmbR62+sqm6ayvTR2ePD+w1d2vGzy/LR1BnC028fxO+/cXpCBvSsskJMLSlQzl3qcaFmsjdaidUfgttpw8P/2qdrXjF6fByjGbxmZvyKqUYYjgxIKQi5RbyvQN8aQItml+A5nZ7Lty6uShofXWBTHl89vgDnnZyYAX3L4kp0+IK6c188uwyXfWZygt/i1iVVmODNU17DBG+eoeMB4zuY4RhGKghmk7EezWYh0UeDR6+vQKreyj944l2tlhEz8Ox7h3HHsjmGezQ7bRbl3Hr9ofV6HQykN8JAaj5Jn2QhVzAj+kjIUfTrDOn3Vj7Q0oPfvLQn4bWB9Gi2W8lQf2i9XgcD6Y2Q634LQRgII1IpSCXIRPQzjp2mjU8uzkuqklrqccJps6p7Guicq8zrUn5/A+mNYDQsdLiGkQqCmQzPzJ40ErcLX3T3a7jivi246O7XsLmuIeN1ZzJJvK9A3/7Ger2Y9XouW4ix5pLKhPE1l1TCbSdce850PPDPffj/XtyD+1/bh2vPmY7KsgLduavL1T2XTy71KL+/k0sLDfdoNhoWKmGkwkhgxPkUcskubOaOR6+vgF5v5bYeHz7sFWU0sywfLZ0hZe/m/7qkUrc3Qu/oo75zx/tDN7T7UO5xoTrWH1rv+5tU5DbcG0Ev6upEn7mEkQrZjvgUYuSKXdjsSBi9vgKq3sqhUATPv9+cFDG0uHocVpx1UpJMXf6wbm+EqSUFyrkjEcY/PmxKOleR257y+zPSG2EgWdASRirkOiNOKeSKXVgvg3dWis5kZlFX34Z7XtyNlWdN1XwE97y4G9PHFigTsnYcblN+5m6Hfga03vVtXHWGad9fqs9QbvrCSGXE+RRyxS6st+NpbPel3WfS0uXH5TWTEnwEl9dMwtEuv7KyaSAcxuqFiVVVVy+cjkA4ojuH3vUFw2HTvj/pHyAIyYy4nUKulBfQ2/G4HVbc+NR7Cav42zfv0rKHjaKyuXtcDmzcdjBhjo3bDuKXX5mLnp4gdjS0a76G6jIPHFYr3vmkJSmj+bPTomYplW+k1OPCBbPHJPWBHp3vxNwJRdi4an5CNdSBfH+5smuUaDrBTEacUgBywy4c3/H0tYdHmJX9kFu6/IavVy/beEpxnnKOUCSMv+xsSMponje1EOf3yWi+dXElXHbS9Y18btponH/yuD7vqUJpgd20aqiTitxYu7Qq6fomFbkNnSeTSJa1YDYjLvool1BFwrx1sBVff2BL0ur30ZXzcNqkIt0VpWq1uePwsUFlOgPRfIRUmc5jCpzHI5Z61Sy6YdHJhrKjnxuAL2Vfcyeu/sPWpMzsh66qzZoFw3COppMdzPBGoo9yENWO52iXOhO4tSugu6IEoHzNZoHa5q6TbXy0K6DcQbR2B1NmNH9tXgX2NHUgwoCVgK/Nq0Bjux9Fbge+dNoETcE8uf1QiuxoHz5o6DBciVWVmZ1NkWjDNZpOdjDZy4hzNOc6pZ48zQkbx2W3oCjfodtvQC8KpzjfqTzXWI96fHS+Q1MI8fPc/eJuFLntyuNLPU4QCA3tPqx7Neq0/v2r+9DQ7sP4US4sP2NygjN7+RmTMX6US3kut8NquJ9C3KeQdH1Z5FMYrteQbf0thOOIUsgx9DKEGay7otRbbUbAynOFwiFlNFG7T70jONYdwE+WJJ7nJ0uqUOC0osMfwoY3o07r6xZOwzULpmLDmwcBQNnIx0JQRh8FwhHDkUS5EIk2XK9BIruyFzEf5Rg2mwVL54zH9LEFCZm9B1u7U0baqF4rznfi1AlFSed651BrQvQRczT66I5lc5TnKXI70NoVwKqzpyLCgIWASCQCt8OG5g4/Vpw5BS3dAc18tOLMKWjs0DGDdQeV0WP7W7oMRxLlQiTacL2GXInsGomIUshBVBnCetFK8RWl3msWCyWdqzjfgW99bhpu+UuddvyaSysRCEXzEfr6FHzBMNb85f2kG8Rfv70Ao/Ic6A52YN2r+7T3XH/edJQUOJQ3ldFuh9KXMtBIolyIRBuO13Ci35swfBGlMEI40YrSyGpzf0s3/rTlAH6+bI5W4+j+V/fixotmKXcQp0+eo1z1N3f6EAqz0kz00FU1SgUTjKgT3g62didkWTNHs6xPm1Q0rG6WI4XhuoMRTkxGlAIRTQSwHkApAAawjpnvIqLRADYCqACwH8BlzNyaCRlzkVQrSiOrzXynDR81dWL1n97Wxlx2C1xWK3646OSk1eHYQnUZ7JICF/Yd6VQqjA5fWKlgFlWVKWXKhUiiXGM47mCEE5OpnUIIwPeY+S0iKgSwnYieB3AVgBeY+WdEdCOAGwH8MEMyZi0+Xwg76tvQ0O5HmceJ6nIvXC4bunr8qGvo1LKNK8sKkJ/nBKBfLbS9x4cPelVDnVWWjwhHsPn6M9DUHkZjRzQGfWyhFY3tQSyYVoT1K2q1408uy0dLVwi3XDobh9t8mt9gnNcFqwUoLVTbnss8TtzwhVn43p+Pt9f85VfmoKI4X7en8zWfnYjzK8ejObYyfb7uMMYWqvsvpMrPkNWsMJLJiFJg5noA9bHHHUS0C8B4AEsAnBM77GEAL0OUgiF8vhA27ahPyir+QlUJ/kfRP/niqrFw2u3KzOXzZ5fg74r3nD29EK/ubsdNm+p6jVfi7Oke/E1x/OmTC2GxWBL8Bj9ZUoVOXxBWKyn7PXvyCOEWTnBOhyOMbr8fmxUVWhfNLsGMsiJc9dDWpAxoo/kZEksvjGQyntFMRBUAXgVQBeAgM4+KjROA1vhzPUZyRrOKNz9uUfY2Hkif5FTvUY5fXYvlDxmbmwB836R+z0YzoAEYzgaWnYWQKwzLjGYiKgDwJID/YOZ2ouN/XMzMRKTUWES0CsAqAJg0adJQiJo16PU2HkifZMM9l3XCSFOdhwim9Xs2Mt7U4QOzTsa2jh9CsnSFkUDGlAIR2RFVCH9k5qdiw41EVM7M9URUDqBJ9V5mXgdgHRDdKQyJwFlCWYp+yCrbfao+yUPRo9lKZPg9ZsiUKj9DL5Ze+i8II4GMZDTHTEMPANjFzHf2emkTgCtjj68E8MxQy5btVJYVKvsh6/VVriwrwIwx6tfKR9li1Ux7j1diUpFNmaE8sciqPL4434rrzk2U6bpzozJVj/Pi1j7nunVJFWaW5SuvY1ZZvroXc1mB8hqqygp1M36NZgNLlq4wEsiIT4GIzgLwGoAdAOJ/ZT8CsAXA4wAmATiAaEjq0VTnEp9CIu9+0qr0D2xcNR/TxriV0Ufb9h/F9/78TpJd/9eXzUVPMAibxRaNMip0IRQJIc9ux+6mDhw65tOcwBNGuTBtbCHufWU3vjZ/Co51BTEq344/vvExvrlgmtLP8eiKWtRMKdZ6Mcft9KeM82JXY7v6Or45H8d8AXT0hNEVCCHfYUNhnhULpo2Fzx9U9pNO1Vc5MZIpdV/n4VyRVBCMMqx8Csz8TwB6RtjzhlKWXKO+Tb2abWjzYc7EItROcSa9p0Enxv/QMR9ue26XVqn0g4ZOPPXWIdz+5WrctCk5Q/nBq2owd2JxQg+EaC+HgFKmI10BAIDDYUVNxeiE1/VW5fXtPvzHxneS5t58/QK8X69fJVUVL2+0R7Nk6QojAclozjHKvXlKW3yZV7/mTLlOnZpSjxPV4wtQO6VI6362u/EYeoJhdQG9CJRVUh+6+jPK8xc4oz8/Vd5BvEKrqszFjLEFuObsk7T+C/e9uheN7X48+M+90Szr2PiD/9ybsuOcUR+BZOkKIwFRCjnGjDFRW3zfuP8ZY/TNG3G7/k3P9HrPkioUua3KjmljdeoSdQVCSmXREwjh+vOma+Us4vWNrBb97m6V5QW4ZXEl1vTKhbhlcSWcduCKeZNxwxPHk9rWXFoJm4Xx5dMmJY5fUom2noDudQ+kF4Fk6Qq5jiiFHKOusUNTCED0JnfTpp2YWpKfZKKJ43BYcWFlCSqKEzORdzV04Tcv70noxfybl/fgF8vm4AdfmIk7/udD7Qb8gy/MRHG+WlkUOu2oqfBg/dW1WgY0URhWsqKuvg0bth5IWOHHO6/9edtBrb6S23F8/N5XEmW695U9uGPZHNzybF3Cdd/ybB0eWVGr+1mVelzKXVUuVfI0K69C8jNGDqIUcowGndVvQ7t+hEx3T0CZiVzqdSg7qR3zBTHa7UjINh7tdiCCCNZcUqndnOOrdU+eFe9+0p3cu/kkBw62+JUr/HZfEAtnlSWMx3s2qHtQq/0WLV36O4VJRW58e+H0rO7RnAqz8iokP2NkIU12cozSQnVXtNJCJyIRxr7mTry+9wj2NXciEolGnu1sUO8u8uw2pY/A67LjbzsP4/TJRZhWUoCayUX4287DAFtw76t7tIY5K8+aintf3YO2nrDy/I1tYThtNuUK3+OyK+fWG4/vUvpe9+h8h+5ndeBot6YQ4uf68dM7ceBot+5npTc+HDGr+5l0URtZyE4hx3A7o7kCfesSFeZZ8eKHjXjvUJtWlK56ghcLZ5amyPpVjx/rDuC8Pr6GWxZHV/dGs5OtOn2gm3Xm1htv7Q7iO+fPwK/+8ZEm03fOn4G2nqCu6ePA0S7luQ4e7cKHjcmRTBecXGooWinTmNW/ebj2gRbSgyiFHKMnEEnwAzBH/QC/vmwudjd2JjWzmVZSkCLrVz0+yu3A6g2JK8c1m+qwfkWt4Wzj+OO+r5UUOlEz2YvlZ05N8DWU6JbhdqCxrSfBpJVnt2BMvkPX9JHvsCnP5bBZ8G9/fCtpZfz4qvnKFfPMby/ASWMzd3PUU3pmdT8brl3UxM+RHkQpZDGqP4qjXX4EQsdNGkRAIMTo8IeUzWxOm1QEu5WVlUp9oZCy0U1rdxBFboeWvwAAT24/hCOdfvxkSRX+q1cU00+WVCHPDuXuxW5ltPvCSj9EoYtwWU1ilNGti6tQ5rHi5ksrcXOvrm83X1oJqwW47W8fJN24Hrtmnm7Yab7Tih9dOAtHuo63Ai3OdyDPZlVen56/5kBL14CUghk3tVT2frPyKoZjfob4OdKHKIUsRe+P4qSSPCw/Y3JS+GcgpG5s3+kPotTjQonHjnXfOB1HY/kIYY7A47IrG938+rK5yjnGeVx473Bbwmq9JxBCJGKBN8+Kh676DI50+jGmwIn2Hj8sZIXbTrj31bqEOe59NRrhpPJDPLKiFkVuO36xbI6W0Wy3kq6J6kin2gHd1OGD02aBw2ZN2D3dfGklGBHl9aXaWZj1/Rm9qZ0o18KMvIrhmJ8hdajShyiFLEC1otT7o3hkRa1yR6Bn2il02REIRfBBfWfSTfDMk4rx1c9MShqPMJRz1KyoVa7WH766Frf97UMt9PPDxk785d3D+MWyOWjt1vFD6PgOmjr8WuOd3nPc940a5fXl2dXF/sYWutDhC2o7jvj5b/5Lne5n+MjKWt18CyPfncVCpt3UTmTvNyuvYrjlZ4ifI32IUhjm6K0oi9x2tfM2hSNWzxSUZ7cob4Kzx3mw/vUDCav49a8fwIQit3IOvbLdbT3qMNKjsblVN229nIeifIdyDruVlDdsu410TR9bPm4x5GDv8AWR77Am7ITyHVbYdLRCqt2AWTe14WrvTzcj9bqHAlEKgyTdzi69FeXGVWfoOodVCVlFbjte/KAhKUmspuJktHarTSz+YASt3YGEVfyJHMcq5/Aod6IZCoiaoe5YNgd2C+GOZdXw5jk001VbTwBuhxW3f7kae5u7NHv/1JJ85DusyrabeXYrCl22hBt2ocsGl82K82cW4dGV89DQ7kNZrOBe3BGrLs+tHve6HOhwh7SaTQDgddvhslkN7eZmrV5g2k1tONr7h4KRet1DgSiFQTAUzi69FWUwHFb+UbjtwLXnTEty6pYUWJWO2xll+djfrO5pUOZ1KB3Q5aPUYa/lReo5gLBypxDhCIoL7djdxPjBE9sT3lNaSPiokZNaeJYWkrrtpodw4KgdTR1+7Ro8LjvGeAibdnyalKC2dM543eS12aUFylIhbocFfYsKMwMM9e+gpFC9q2ls92HelGLTbmoOGyUoQ4ct9x2tw9HPkStkvB3nYMlk6eyhKKWcao74arT3H8W2A0cNt6u0WQjbDrTizuePx/h/9/MzUDO5CBu27sfS0yahpdOP4gInnn7rIJbVTMYPTGqhCcC0tps/37wraZdyw6KTlcdvXDUfhS678rN9dOU8fP2BLco5el83AM03oiwNrnOejavmY87EopQlvfuLlPMWBsqwKp2dKwyFsyvVNlnl/NOzh+smqLX7YbdZ4LRaElabTqsFzZ0BVI4rwhv7WqImnKZOVI4rQmO731CSWpPOeHOHHwwowz+Nttds7Q4qy2X06BTpa2jzoTugrvba2KH+Xlu71b6RYz1B5fFHuwJKP04wHD3WDOetOFwFsxGlMAgGYhc26oOwWAgLp5fgkRW1aGj3o8zjRHV51CauKjmtl3Cmm6AWSyC7fsPbSa89sqIWjW09CSacH104y3DLz7E64yWFTtgshP+8cAZGuZ2aT2HGWDdKPU5cMHsMvjZ/ila2+9E3Ptado8htT7gGXzBaLmP91bVKP0e0lLjabFZSoD+HqsTGIzqRXWUeJ9a9uifJj7Ooqkz3+zbKSCjqJwwtohQGgVFn10B8EIFAGJt21ieVtb60sgx/fb8hyR5e5nEoyz0wwkrHrc0SwbFu9Yq5rSeYEGLqC0Zw298+wKbrzsB1505PSlJzWlmZiGaliDKpbXS+FV3+CHqCjBv6lMwoKVCX7S4psCr9HG09eo18/Eo/x8ySQrz3aZtyJR8Kqz+rLr96R9DhCyojnzwuG75amzi32QX3JnjzlP6PCd480+YQRhaiFAaBUWfXQGLT3/u0TVMI8ffc9MxOVBS7cc+LuxMieu55cTd+ddlcFLlt0eQufwj5Lhu6/UG4bHY0tnckrPq/+/kZmFjkRoFLHRaa77SpM5c7gtoNPi7Tfz0TTSyLF8TrnYh2x7I58IfCCeYpfyiMtu4IQhHWeibEz7VmU3SFf1Of8Zs21eHhq2tR39qBh6+uRVOsRejrexoxtaRUd9X/3ce39jnPTswsKwARlMl5tRVzcKQzkPBZ/eALMzG5yK1clec77crQ3VMmeJXf0WmTilKadozsJt9vaFcm+c0oLcDcSUW6cwiCHqIUBokRu/BA7L96pRVau4P4P7WTk3YEYY6gpSuotcuMr1p9wYjmSI6f487nP8L9y2vQHQzjPxfNQkv38XIPo90OhCNhZWZvh19tpz/SGVD6Go51B7D2r7uSbtj3L6/RbczT1Kn2HXT6gxjrLcCVvaKP1lxSCSIoV+sRZqViq2/zwWohrPzsFG03FDePBSMRrVdEfN47/udDrF9Ri299bhpu6VViY02sxIYqdNcfUkddHe3y637fRneTh4/1KD+nw8d6RCkIA0KUwhCSygehtzos97iUNvHRbjtu23ogYRX62NYDqBxXjbcPtuD33zg9wRZfUZyvvHl0B8IoKXBg/5GupF2E227HhjcT8ws2vHkQa5dW62RHq8tA5DlsOGPKaFx11nH/wEP//BjdgTBGu9VJanr5FoVOO659Ntl3cN/yGvxtR31CU577Xt2LaSUF+LfPTU2ob/Rvn5uKcq8LLjshEon0KZkBXQf00a6AphC0uf9Sh/uX1yhrKHldDqUPYuM35+v+RozuJsfo+JDGFER9RZksGicF67ITUQpDiJ4PYlKRW7es9ezSQiw/Ywr2NHVory0/Ywr8oZByp+CyQ1nWutSrkyHstqMnGFbvIq6swYozpyTsIFacOQV5sVV431V5oU7Z7lKPHYuqFTIVOtDuD2LNpZVJq2+iiNJv0a5j1w+Gw7iwujzBfn/9edNRUujAnubk6rDMQJc/Gs3U92Y+e5xH+VkVOG26itXttGFdrx3H2qVV6NCRtbXbvBahLpv6u3DZLRktGicF67IXUQpDiJ4P4uDRLt2y1p3+EDr9oYTXbrpkNsaNytMUAhC9cfzqHx9h/YpapY3+kRW16p7HNotu17Iefxj+cCRpB2G3WjDabe/Tec2OMAMfNRzDH66uTcg2rhiTr/YbrKiFw2pNaK/JHG2v+ctlc3T9Fuobtl1ZquP0yeo6RpXjPLBZCV2BcNLnHgHjl1+Zi+/9+fgN7ZdfmatbR8mbZ8fqPpFPP356p269KbdD/88uVTSRauXd5Q8r/RlzJ47KaNG4/S1duH3zroRd5u2bd2FWWaGEyg5zRCkMMSofRGO7X7estc0CPPXWJ0lhjavOPkl5I08V3+8P9nH2BsPoCYZ1w1iL3HZ8e0PyDmL9is9gVL4D+U77cbOLjeCwATPKRvXJNq7EsW71irmpww+n1aL0QxzRUVTHuoPK3Yhe9JFejoQvGEEeWZWf+/3La+C0J2YJO+2ENl9AGdnVHVBfX7tPXW8qEA4rfhlR9LKsJ3jzlLvJKcUFOqVIXBnNYWjp8hv2pwjDA1EKQ0wgEMZ7n7ahod2Hco8L1eO8us7W7kAIDhuUSVkFLpturoAqvn9soRN3/P0D7fgIAw/878dYu7QaDhuUJhwQMGNsAa45+yRNId336l50B8I41NqjOWPj0TnFbocyYkhvdV9a6IQ/FNGNfFJdX57DirXPJTcRumPZHPUKWyevYbTbjlYdZdUTCOPbiryNR1fW4rGtiavyx7YewC+WzVH6fYryHMroprOmnar7+zjY2q2MWJpVVqjcTU4fW5AyLNpoHo1ZOKwWtT9llb4/Reg/6fTXiFIYQgKBMJ5+79OknIO5E7zKP96JRW60dAVwy7OJXcBuebYOG1fNx7+fMy3JHOR1q+P7812kXLmFIxF0+FhpwvnZl6pxxbzEOPs1l1ai0GnD+tf3J9y41r++Hz++6GTd1b3K7s1g3eqmDhspI32cdlLuLHqCIeXnwYgom/L4QiF48tSOcU+e2nfQ0qWO+OoOhHDZZ/rkQiypgsthUR4fCCeeuzd6K+zOFE2S9MKiM1k0Ts9Z3x3Q3yUJ/SPd/hrj3UGEAaOXc9DlD+HGRbPgske/DpfdghsXzYLNSjjS6UeR24Frz52G6xZG/xW5HejwhZR2+rbusHK13uVn5cot32GDLxhWd2vzhZXRNu3+IFacOUXrIxB3QHtjkUS9cdktKMyzaXbv6xZOw8qzpmL96wdwrDsEu9WilaO+buE0rDp7KvIdVtgtFk1RXbdwGq5ZMBX3vrIHYFLOkWezKT8PCyz4Xa/zrDxrKn73yh44bDYEI2GsubQy4XNfc2klQuGIco4it13bKcTP9djWA8hz2JTfa08gguffj0ZE3f6latyxbA6ef78eo/OPRwbta+7E63uPYF9zJyIR1l1hh8Ksu5uMmyTnTx2j9VAAjvuwnlu9ABtWzcNzqxcMmaM3HmnXm7hZSxgcer6i/S1dppxfdgpDSFOHT2mOafcF4c1LdNx68+w41h3A+FEuZa6A3kpMr/ZRU7tfGa/f0hXA2EKHco5Cl7otZYHTjjyHHzMKChMS5CwEpQ29OxDC+FFOzCwr1K57/CgnvHk2NHcGQJR4kyIitPn06wyp5jjSpb7uli517kRzpx/F+Q71DunLp2Dt0qoku74nz6qsQHtEJ6fiaFdAN6NZb7U3Kk/dJyOoY2abNDr1qj9TzXGktHX6SLevSJTCEDKxKE9pjsl32nHTproEe/+vX/gIv/zKXFgIylyB2798im58v2p8nI5yGZ1vRySinmP+lLm4+rMVSdVTHRYCg/D9Xtdxy+JKAOoM4f/7xWpc3se88pMlVch32OCwWtDY1pPwOTEzvC51naH1K2qVc9yxbI5uGKlqvKzQhaM6CqOzJ5Rg12eO2vXv/uppCdFVJYUu/KPuMC6oGq+cY5RbHZV0WiypzEifjImj3cqb7JQxw/MmK6Wt00e6GwyJUhhCugMRpTnmt187VbkqbvcFwAxdX4DKOdwTDOlEvESUNuk/rpyHNl9AOYcvrM5feEQn7HX91bXK9p0uu1UZXrr+6lpYLaRs4Xn/8hrlaqitO6C004fCYaVvwptnU44TAUX5dt2buUpZMEcwa1xiL4e1S6tgtUSU4b4dPr2oKx+YoXytwxdQymuzZt9Ndri18MwVJhW5lTtZs2pqiVIYQpp1zAx5dptyVfzoynkIhiPKrmU1k+coTR8/XVqtXElPGaPOaG7pCmCUTvXPuy6fq3zPkU51+OfR7gCK8+0JGcLdgaCuSetodwBBHVt5hz+kzglwO/DY1l1JEUC3fbEa40a5Ekxw40a58GmbXxnHP3G0G/kOizLEtNMfVM5NRNofYlzOHz8dzZ347cuJ38VvX96DX+jsXuIrOtVrdqtVKe+pk0ahYkyB3GSFpAi1+E72RDW1+osohSFkTIE6q7jTpw5JDYSimbrKXYRf3fDeZbcoV+t6c3vz7Lo28TyHOlmrQKecRb7ThgMt3Ql1l75z/gyML3LrHu+wqYvxjXY7lDuenkBIeX02C+HO5z/STHDhCHDn8x/hjmVzlHH8ZR4XAuGwMsT0ti9WK001en4cPb8FAynt6qrXSj1OpbxSCluI09juU/7ectanQESLANwFwArgfmb+WYZFMo1RbpvS5KNnxsh3WhEIkXIV//DV6tj/UISVzeX1nMDtviDG6PQPKHCqTS8FDqty3GkjZZb1n66Zp3uePKdFaXpxOyzKHc8vls1RXp/TasHyMyqScie8eVbcuqQqKQz4lHFefNTcoVQwXpcdZ0wdk2Sq2d/Spfyc9D4/p82S0uSjeg1IrUgEId0+hWHVjpOIrAA+AvB5AIcAvAngCmZ+X+89mWzHaZRQKIIXP2pCIBjRzCsOuwWj8uzYtr816YZdO6UIx7qDuGb99qRz3bf8dOxp6ky6oU0bW4CbnqnTIoaYgafeOoRbF1di7XPvJ7XQ/GXM1FP3aUfS/HMnenC0O4SPGju0G/CM0kJMHePGWwePoanDr42PLXRiZlkBvnLvliRZH7rqdHT4w9jT1KkdP21sAaaPdaPASdjT7IOVLGjtDqLIbUeYI3BYCe9+kizTnIkeHO0KYnevc00fW4CpJfn4qKETH7d0aeNTivMxZYwbs8u8eO/TNi3R55RxXjgcVry5vwXvH27XajtZCCh2O1A53oOaiuKk69CLGCr3OvH63qPK7091nhNhRptOIXcxK08hW9px1gLYw8z7AICINgBYAkBXKWQTNpsFC2eMRV19GxrafCjzulBZ7sXB1m7lqnhRVRkCYXUoYp5dbXu+66tqc8nYQgeuPXd60orZ67aCiNR9BabMAUd6cNZJY9AY613Q2N6JY90BePPsaOrwa3N48+zaXMk7Djv2NrYlnOe9T45gTL4DLlseRudZ4Q9ZYKEgXDYrnDYCWezYuK0uSaYvVNZiXoUbk0Ynf4Z/3LI/mlUcCCHPYcP6f+3Dz5fNhcNhRU3F6KTvozjfiQf/9XGCyenBf32Mh66qVX5/ehE1+1u6dL+/gSAOWiEV6Y7sGm47hWUAFjHzNbHn3wAwj5mv63PcKgCrAGDSpEmnHzhwYMhlNZNUmv9ASxc21zUk7QhOrxiF5Q+8mXQDfuyaedh3pCspMqGyvBBtPX5YyKbdmCMcQr7TAbuNsPNwR9J7ZpUWYtnvX0+a40/fnIef/vX9pBvw2qWVqKvvwv/73zu08/z0i9WYXV6A13a3JIW2njdzLJo6/fju4+8m7WzuuWIumjoC/V4NDWT1ZNaKSyqCCtmI3k4hK5VCb7LJfJQKPZNBJMIJhdAsBJwywYuTxhQolcWFlaXY3dwFfy8TldNuwWenjMaLu48kmYIumFWKcDiCNw62JJlwphbn45WPjiQ1oampKMKepi7c8OR72vjPv3wKqsd7UN/WAyILjnT6MabACeYIaiYV45U9zdhx+Pg1VI/34rxZpdjf0oWL7n4tSfE8t3qBtgrv72poIGYXs0w1YvIRso1sUQpnALiZmb8Qe/6fAMDM/1fvPbmiFFKhuuEAUCqLhTNLEQpFlDb0eDG+vuMA0NMTxI6GdjS2+1HqcaK6zAOn04YXPmhMupmfO2MsXvqoSXmTj0Q4yTxms1lSKj1ZZQvC0JMtSsGGqKP5PACHEXU0/x9mrtN7z0hQCnoMxeo01c3crLlllS0IQ09WKAUAIKKLAPwa0ZDUB5n5p6mOH8lKQRAEYaBkS/QRmPk5AM9lWg5BEISRiJTOFgRBEDREKQiCIAgaohQEQRAEDVEKgiAIgsawiz4yChE1AzhRSvMYAEeGQJzhhlz3yEKue+QxmGufzMwlfQezXin0ByLapgq9ynXkukcWct0jj3Rcu5iPBEEQBA1RCoIgCILGSFEK6zItQIaQ6x5ZyHWPPEy/9hHhUxAEQRD6x0jZKQiCIAj9QJSCIAiCoJHTSoGIFhHRh0S0h4huzLQ86YSIHiSiJiLa2WtsNBE9T0S7Y/8XZVLGdEBEE4noJSJ6n4jqiOj62HhOXzsRuYhoKxG9G7vuW2LjU4hoS+w3v5GIHJmWNR0QkZWI3iaiZ2PPc/66iWg/Ee0goneIaFtszPTfec4qBSKyAvgNgAsBzAZwBRHNzqxUaeUPABb1GbsRwAvMPB3AC7HnuUYIwPeYeTaA+QCujX3PuX7tfgALmXkOgLkAFhHRfAC3A/gVM08D0ApgZeZETCvXA9jV6/lIue5zmXlur9wE03/nOasUANQC2MPM+5g5AGADgCUZliltMPOrAI72GV4C4OHY44cBLB1KmYYCZq5n5rdijzsQvVGMR45fO0fpjD21x/4xgIUAnoiN59x1AwARTQBwMYD7Y88JI+C6dTD9d57LSmE8gE96PT8UGxtJlDJzfexxA4DSTAqTboioAsCpALZgBFx7zITyDoAmAM8D2AvgGDOHYofk6m/+1wBuABBv7F2MkXHdDODvRLSdiFbFxkz/nQ+7JjtCemBmJqKcjT8mogIATwL4D2Zujy4eo+TqtTNzGMBcIhoF4L8BzMqsROmHiC4B0MTM24nonAyLM9ScxcyHiWgsgOeJ6IPeL5r1O8/lncJhABN7PZ8QGxtJNBJROQDE/m/KsDxpgYjsiCqEPzLzU7HhEXHtAMDMxwC8BOAMAKNivc6B3PzNfxbAYiLaj6hJeCGAu5D71w1mPhz7vwnRRUAt0vA7z2Wl8CaA6bGoBAeArwLYlGGZhppNAK6MPb4SwDMZlCUtxOzJDwDYxcx39nopp6+diEpiOwQQUR6AzyPqT3kJwLLYYTl33cz8n8w8gZkrEP2bfpGZv4Ycv24iyieiwvhjABcA2Ik0/M5zOqOZiC5C1P5oBfAgM/80sxKlDyL6E4BzEC2l2whgDYCnATwOYBKi5cUvY+a+zuishojOAvAagB04bmP+EaJ+hZy9diI6BVHHohXRxd3jzHwrEU1FdAU9GsDbAL7OzP7MSZo+Yuaj7zPzJbl+3bHr++/YUxuAx5j5p0RUDJN/5zmtFARBEARj5LL5SBAEQTCIKAVBEARBQ5SCIAiCoCFKQRAEQdAQpSAIgiBoiFIQBEEQNEQpCIIgCBqiFAShHxDR07FCZHW9ipGpjvsdEW3r3eMgNn4REX0QO8fdvfoA5Md6YWyN9QfI2Uq+QnYgyWuC0A+IaDQzH42VlHgTwOeYuSXFcVZE69uvBvARgN0Azmbmj2PZ54WxTNzbALzPzI/GylZsBXAqM3cN1bUJQm9kpyAI/WM1Eb0L4A1ECy1O1znuMiJ6C9FSC5WINniaBWAfM38cO+ZPvY6/AMCNsRLYLwNwIVqyQBAygpTOFoQTEKuxcz6AM5i5m4heRvTm3fe4KQC+D+AzzNxKRH9QHdf3bQC+zMwfmimzIAwU2SkIwonxAmiNKYRZiLb9VOEB0AWgjYhKEW0FCwAfApgaawIEAJf3es//APh2rNoriOhUs4UXBCPITkEQTsxmAN8iol2I3uDfUB3EzO8S0dsAPkC069//xsZ7iOjfAWwmoi5EfRJxfoJoJd/3iMgC4GMAl6TrQgThRIijWRCGACIqYObO2I7gNwB2M/OvMi2XIPRFzEeCMDR8M+ZMrkPUHPX7zIojCGpkpyAIA4CItgBw9hn+BjPvyIQ8gmAWohQEQRAEDTEfCYIgCBqiFARBEAQNUQqCIAiChigFQRAEQeP/B7JZy3hEII5bAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.scatterplot(x=df_contracts.a_age,y=df_contracts.annual_salary)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other ways to measure correlation exist. For example, if you are interested into how one variable will increase (or decrease) as another variable increases (or decreases), the *Spearman’s or Kendall’s rank correlation coefficients* might work well."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions\n",
    "\n",
    "* Try to explore the correlation of other variables in the dataset.\n",
    "* Can you think of a possible motivation for the trend we see: older apprentices with a shorter contract getting on average a higher annual salary?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sampling and uncertainty\n",
    "\n",
    "Often, we work with samples and we want the sample to be representative of the population it is taken from, in order to draw conclusions that generalise from the sample to the full population.\n",
    "\n",
    "Sampling is *tricky*. Samples have *variance* (variation between samples from the same population) and *bias* (systematic variation from the population)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further reading\n",
    "\n",
    "* For a complementary introduction to statistics and data analysis, see https://www.humanitiesdataanalysis.org/statistics-essentials/notebook.html.\n",
    "* Related to statistics and data analysis is the realm of probability theory, which allows us to formally model and calculate the likelihood of events. For an introduction, see https://www.humanitiesdataanalysis.org/intro-probability/notebook.html."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2: working with texts\n",
    "\n",
    "Let's get some basics (or a refresher) of working with texts in Python. Texts are sequences of discrete symbols (words or, more generically, tokens).\n",
    "\n",
    "Key challenge: representing text for further processing. Two mainstream approaches:\n",
    "* *Bag of words*: a text is a collection of tokens occurring with a certain frequence and assumed independently from each other within the text. The mapping from texts to features is determinsitic and straighforward, each text is represented as a vector of the size of the vocabulary.\n",
    "* *Embeddings*: a method is used (typically, neural networks), to learn a mapping from each token to a (usually small) vector representing it. A text can be represented in turn as an aggregation of these embeddings."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import the dataset\n",
    "Let us import the Elon Musk's tweets dataset in memory.\n",
    "\n",
    "<img src=\"figures/elon_loop.jpeg\" width=\"400px\" heigth=\"400px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "root_folder = \"../data/musk_tweets\"\n",
    "df_elon = pd.read_csv(codecs.open(os.path.join(root_folder,\"elonmusk_tweets.csv\"), encoding=\"utf8\"), sep=\",\")\n",
    "df_elon['text'] = df_elon['text'].str[1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>849636868052275200</td>\n",
       "      <td>2017-04-05 14:56:29</td>\n",
       "      <td>'And so the robots spared humanity ... https:/...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>848988730585096192</td>\n",
       "      <td>2017-04-03 20:01:01</td>\n",
       "      <td>\"@ForIn2020 @waltmossberg @mims @defcon_5 Exac...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>848943072423497728</td>\n",
       "      <td>2017-04-03 16:59:35</td>\n",
       "      <td>'@waltmossberg @mims @defcon_5 Et tu, Walt?'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>848935705057280001</td>\n",
       "      <td>2017-04-03 16:30:19</td>\n",
       "      <td>'Stormy weather in Shortville ...'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>848416049573658624</td>\n",
       "      <td>2017-04-02 06:05:23</td>\n",
       "      <td>\"@DaveLeeBBC @verge Coal is dying due to nat g...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   id           created_at  \\\n",
       "0  849636868052275200  2017-04-05 14:56:29   \n",
       "1  848988730585096192  2017-04-03 20:01:01   \n",
       "2  848943072423497728  2017-04-03 16:59:35   \n",
       "3  848935705057280001  2017-04-03 16:30:19   \n",
       "4  848416049573658624  2017-04-02 06:05:23   \n",
       "\n",
       "                                                text  \n",
       "0  'And so the robots spared humanity ... https:/...  \n",
       "1  \"@ForIn2020 @waltmossberg @mims @defcon_5 Exac...  \n",
       "2       '@waltmossberg @mims @defcon_5 Et tu, Walt?'  \n",
       "3                 'Stormy weather in Shortville ...'  \n",
       "4  \"@DaveLeeBBC @verge Coal is dying due to nat g...  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_elon.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2819, 3)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_elon.shape"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Natural Language Processing in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import some of the most popular libraries for NLP in Python\n",
    "import spacy\n",
    "import nltk\n",
    "import string\n",
    "import sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /Users/matteo/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nltk.download('punkt')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A typical NLP pipeline might look like the following:\n",
    "    \n",
    "<img src=\"figures/spacy_pipeline.png\" width=\"600px\" heigth=\"600px\">\n",
    "\n",
    "### Tokenization: splitting a text into constituent tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import TweetTokenizer, word_tokenize\n",
    "tknzr = TweetTokenizer(preserve_case=True, reduce_len=False, strip_handles=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"@ForIn2020 @waltmossberg @mims @defcon_5 Exactly. Tesla is absurdly overvalued if based on the past, but that's irr\\xe2\\x80\\xa6 https://t.co/qQcTqkzgMl\"\n"
     ]
    }
   ],
   "source": [
    "example_tweet = df_elon.text[1]\n",
    "print(example_tweet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\"', '@ForIn2020', '@waltmossberg', '@mims', '@defcon_5', 'Exactly', '.', 'Tesla', 'is', 'absurdly', 'overvalued', 'if', 'based', 'on', 'the', 'past', ',', 'but', \"that's\", 'irr', '\\\\', 'xe2', '\\\\', 'x80', '\\\\', 'xa6', 'https://t.co/qQcTqkzgMl', '\"']\n",
      "['``', '@', 'ForIn2020', '@', 'waltmossberg', '@', 'mims', '@', 'defcon_5', 'Exactly', '.', 'Tesla', 'is', 'absurdly', 'overvalued', 'if', 'based', 'on', 'the', 'past', ',', 'but', 'that', \"'s\", 'irr\\\\xe2\\\\x80\\\\xa6', 'https', ':', '//t.co/qQcTqkzgMl', \"''\"]\n"
     ]
    }
   ],
   "source": [
    "tkz1 = tknzr.tokenize(example_tweet)\n",
    "print(tkz1)\n",
    "tkz2 = word_tokenize(example_tweet)\n",
    "print(tkz2)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Question: can you spot what the Twitter tokenizer is doing instead of a standard one?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'!\"#$%&\\'()*+,-./:;<=>?@[\\\\]^_`{|}~'"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string.punctuation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# some more pre-processing\n",
    "\n",
    "def filter(tweet):\n",
    "    \n",
    "    # remove punctuation and short words and urls\n",
    "    tweet = [t for t in tweet if t not in string.punctuation and len(t) > 3 and not t.startswith(\"http\")]\n",
    "    return tweet\n",
    "\n",
    "def tokenize_and_string(tweet):\n",
    "    \n",
    "    tkz = tknzr.tokenize(tweet)\n",
    "    \n",
    "    tkz = filter(tkz)\n",
    "    \n",
    "    return \" \".join(tkz)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\"', '@ForIn2020', '@waltmossberg', '@mims', '@defcon_5', 'Exactly', '.', 'Tesla', 'is', 'absurdly', 'overvalued', 'if', 'based', 'on', 'the', 'past', ',', 'but', \"that's\", 'irr', '\\\\', 'xe2', '\\\\', 'x80', '\\\\', 'xa6', 'https://t.co/qQcTqkzgMl', '\"']\n",
      "['@ForIn2020', '@waltmossberg', '@mims', '@defcon_5', 'Exactly', 'Tesla', 'absurdly', 'overvalued', 'based', 'past', \"that's\"]\n"
     ]
    }
   ],
   "source": [
    "print(tkz1)\n",
    "print(filter(tkz1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_elon[\"clean_text\"] = df_elon[\"text\"].apply(tokenize_and_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>text</th>\n",
       "      <th>clean_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>849636868052275200</td>\n",
       "      <td>2017-04-05 14:56:29</td>\n",
       "      <td>'And so the robots spared humanity ... https:/...</td>\n",
       "      <td>robots spared humanity</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>848988730585096192</td>\n",
       "      <td>2017-04-03 20:01:01</td>\n",
       "      <td>\"@ForIn2020 @waltmossberg @mims @defcon_5 Exac...</td>\n",
       "      <td>@ForIn2020 @waltmossberg @mims @defcon_5 Exact...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>848943072423497728</td>\n",
       "      <td>2017-04-03 16:59:35</td>\n",
       "      <td>'@waltmossberg @mims @defcon_5 Et tu, Walt?'</td>\n",
       "      <td>@waltmossberg @mims @defcon_5 Walt</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>848935705057280001</td>\n",
       "      <td>2017-04-03 16:30:19</td>\n",
       "      <td>'Stormy weather in Shortville ...'</td>\n",
       "      <td>Stormy weather Shortville</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>848416049573658624</td>\n",
       "      <td>2017-04-02 06:05:23</td>\n",
       "      <td>\"@DaveLeeBBC @verge Coal is dying due to nat g...</td>\n",
       "      <td>@DaveLeeBBC @verge Coal dying fracking It's ba...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   id           created_at  \\\n",
       "0  849636868052275200  2017-04-05 14:56:29   \n",
       "1  848988730585096192  2017-04-03 20:01:01   \n",
       "2  848943072423497728  2017-04-03 16:59:35   \n",
       "3  848935705057280001  2017-04-03 16:30:19   \n",
       "4  848416049573658624  2017-04-02 06:05:23   \n",
       "\n",
       "                                                text  \\\n",
       "0  'And so the robots spared humanity ... https:/...   \n",
       "1  \"@ForIn2020 @waltmossberg @mims @defcon_5 Exac...   \n",
       "2       '@waltmossberg @mims @defcon_5 Et tu, Walt?'   \n",
       "3                 'Stormy weather in Shortville ...'   \n",
       "4  \"@DaveLeeBBC @verge Coal is dying due to nat g...   \n",
       "\n",
       "                                          clean_text  \n",
       "0                             robots spared humanity  \n",
       "1  @ForIn2020 @waltmossberg @mims @defcon_5 Exact...  \n",
       "2                 @waltmossberg @mims @defcon_5 Walt  \n",
       "3                          Stormy weather Shortville  \n",
       "4  @DaveLeeBBC @verge Coal dying fracking It's ba...  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_elon.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save cleaned up version\n",
    "\n",
    "df_elon.to_csv(os.path.join(root_folder,\"df_elon.csv\"), index=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building a dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2819, 7864)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "count_vect = CountVectorizer(lowercase=False, tokenizer=tknzr.tokenize)\n",
    "X_count = count_vect.fit_transform(df_elon.clean_text)\n",
    "X_count.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6617"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_list = count_vect.get_feature_names_out()  \n",
    "count_list = X_count.toarray().sum(axis=0)\n",
    "dictionary = dict(zip(word_list,count_list))\n",
    "count_vect.vocabulary_.get(\"robots\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_count[:,count_vect.vocabulary_.get(\"robots\")].toarray().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dictionary[\"robots\"]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions\n",
    "\n",
    "* Find the tokens most used by Elon.\n",
    "* Find the twitter users most referred to by Elon (hint: use the @ handler to spot them)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Tesla', 322),\n",
       " ('Model', 236),\n",
       " ('that', 223),\n",
       " ('will', 218),\n",
       " ('with', 177),\n",
       " ('@SpaceX', 169),\n",
       " ('from', 163),\n",
       " ('this', 159),\n",
       " ('@TeslaMotors', 149),\n",
       " ('launch', 124)]"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dictionary_list = sorted(dictionary.items(), key=lambda x:x[1], reverse=True)\n",
    "[d for d in dictionary_list][:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('@SpaceX', 169),\n",
       " ('@TeslaMotors', 149),\n",
       " ('@elonmusk', 85),\n",
       " ('@NASA', 48),\n",
       " ('@Space_Station', 19),\n",
       " ('@FredericLambert', 17),\n",
       " ('@ID_AA_Carmack', 15),\n",
       " ('@WIRED', 14),\n",
       " ('@vicentes', 14),\n",
       " ('@BadAstronomer', 11)]"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[d for d in dictionary_list if d[0].startswith('@')][:10]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Representing tweets as vectors\n",
    "\n",
    "Texts are of variable length and need to be represented numerically in some way. Most typically, we represent them as *equally-sized vectors*.\n",
    "\n",
    "Actually, this is what we have already done! Let's take a closer look at `X_count` above.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                                           849636868052275200\n",
       "created_at                                  2017-04-05 14:56:29\n",
       "text          'And so the robots spared humanity ... https:/...\n",
       "clean_text                               robots spared humanity\n",
       "Name: 0, dtype: object"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This is the first Tweet of the data frame\n",
    "\n",
    "df_elon.loc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's get the vector representation for this Tweet\n",
    "\n",
    "vector_representation = X_count[0,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# there are 3 positions not to zero, as we would expect: the vector contains 1 in the columns related to the 3 words that make up the Tweet. \n",
    "# It would contain a number higher than 1 if a given word were occurring multiple times.\n",
    "\n",
    "np.sum(vector_representation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "1\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "# Let's check that indeed the vector contains 1s for the right words\n",
    "# Remember, the vector has shape (1 x size of the vocabulary)\n",
    "\n",
    "print(vector_representation[0,count_vect.vocabulary_.get(\"robots\")])\n",
    "print(vector_representation[0,count_vect.vocabulary_.get(\"spared\")])\n",
    "print(vector_representation[0,count_vect.vocabulary_.get(\"humanity\")])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Term Frequency - Inverse Document Frequency\n",
    "We can use boolean counts (1/0) and raw counts (as we did before) to represent a Tweet over the space of the vocabulary, but there exist improvements on this basic idea. For example, the TF-IDF weighting scheme:\n",
    "\n",
    "$tfidf(t, d, D) = tf(t, d) \\cdot idf(t, D)$\n",
    "\n",
    "$tf(t, d) = f_{t,d}$\n",
    "\n",
    "$idf(t, D) = log \\Big( \\frac{|D|}{|{d \\in D: t \\in d}|} \\Big)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2819, 7864)"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "count_vect = TfidfVectorizer(lowercase=False, tokenizer=tknzr.tokenize)\n",
    "X_count_tfidf = count_vect.fit_transform(df_elon.clean_text)\n",
    "X_count_tfidf.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.7226760995112569"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_count_tfidf[0,:].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_count[0,:].sum()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sparse vectors (mention)\n",
    "How is Python representing these vectors in memory? Most of their cells are set to zero. \n",
    "\n",
    "We call any vector or matrix whose cells are mostly to zero *sparse*.\n",
    "There are efficient ways to store them in memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1x7864 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 3 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_count_tfidf[0,:]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Spacy pipelines\n",
    "\n",
    "Useful to construct sequences of pre-processing steps: https://spacy.io/usage/processing-pipelines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "ename": "OSError",
     "evalue": "[E050] Can't find model 'en_core_web_sm'. It doesn't seem to be a Python package or a valid path to a data directory.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mOSError\u001b[0m                                   Traceback (most recent call last)",
      "\u001b[1;32m/Users/matteo/Dropbox/My Mac (MacBook-Pro-9.local)/Documents/ADA-DHOxSS/notebooks/3.2 Exploratory data analysis and working with texts.ipynb Cell 121\u001b[0m in \u001b[0;36m<cell line: 4>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/matteo/Dropbox/My%20Mac%20%28MacBook-Pro-9.local%29/Documents/ADA-DHOxSS/notebooks/3.2%20Exploratory%20data%20analysis%20and%20working%20with%20texts.ipynb#Y230sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m \u001b[39m# Load a pre-trained pipeline (Web Small): https://spacy.io/usage/models\u001b[39;00m\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/matteo/Dropbox/My%20Mac%20%28MacBook-Pro-9.local%29/Documents/ADA-DHOxSS/notebooks/3.2%20Exploratory%20data%20analysis%20and%20working%20with%20texts.ipynb#Y230sZmlsZQ%3D%3D?line=1'>2</a>\u001b[0m \n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/matteo/Dropbox/My%20Mac%20%28MacBook-Pro-9.local%29/Documents/ADA-DHOxSS/notebooks/3.2%20Exploratory%20data%20analysis%20and%20working%20with%20texts.ipynb#Y230sZmlsZQ%3D%3D?line=2'>3</a>\u001b[0m \u001b[39m#!python -m spacy download en_core_web_sm\u001b[39;00m\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/matteo/Dropbox/My%20Mac%20%28MacBook-Pro-9.local%29/Documents/ADA-DHOxSS/notebooks/3.2%20Exploratory%20data%20analysis%20and%20working%20with%20texts.ipynb#Y230sZmlsZQ%3D%3D?line=3'>4</a>\u001b[0m nlp \u001b[39m=\u001b[39m spacy\u001b[39m.\u001b[39;49mload(\u001b[39m'\u001b[39;49m\u001b[39men_core_web_sm\u001b[39;49m\u001b[39m'\u001b[39;49m)\n",
      "File \u001b[0;32m~/.pyenv/versions/3.10.0/envs/ada-dhoxss-2022/lib/python3.10/site-packages/spacy/__init__.py:51\u001b[0m, in \u001b[0;36mload\u001b[0;34m(name, vocab, disable, exclude, config)\u001b[0m\n\u001b[1;32m     30\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mload\u001b[39m(\n\u001b[1;32m     31\u001b[0m     name: Union[\u001b[39mstr\u001b[39m, Path],\n\u001b[1;32m     32\u001b[0m     \u001b[39m*\u001b[39m,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     36\u001b[0m     config: Union[Dict[\u001b[39mstr\u001b[39m, Any], Config] \u001b[39m=\u001b[39m util\u001b[39m.\u001b[39mSimpleFrozenDict(),\n\u001b[1;32m     37\u001b[0m ) \u001b[39m-\u001b[39m\u001b[39m>\u001b[39m Language:\n\u001b[1;32m     38\u001b[0m     \u001b[39m\"\"\"Load a spaCy model from an installed package or a local path.\u001b[39;00m\n\u001b[1;32m     39\u001b[0m \n\u001b[1;32m     40\u001b[0m \u001b[39m    name (str): Package name or model path.\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     49\u001b[0m \u001b[39m    RETURNS (Language): The loaded nlp object.\u001b[39;00m\n\u001b[1;32m     50\u001b[0m \u001b[39m    \"\"\"\u001b[39;00m\n\u001b[0;32m---> 51\u001b[0m     \u001b[39mreturn\u001b[39;00m util\u001b[39m.\u001b[39;49mload_model(\n\u001b[1;32m     52\u001b[0m         name, vocab\u001b[39m=\u001b[39;49mvocab, disable\u001b[39m=\u001b[39;49mdisable, exclude\u001b[39m=\u001b[39;49mexclude, config\u001b[39m=\u001b[39;49mconfig\n\u001b[1;32m     53\u001b[0m     )\n",
      "File \u001b[0;32m~/.pyenv/versions/3.10.0/envs/ada-dhoxss-2022/lib/python3.10/site-packages/spacy/util.py:427\u001b[0m, in \u001b[0;36mload_model\u001b[0;34m(name, vocab, disable, exclude, config)\u001b[0m\n\u001b[1;32m    425\u001b[0m \u001b[39mif\u001b[39;00m name \u001b[39min\u001b[39;00m OLD_MODEL_SHORTCUTS:\n\u001b[1;32m    426\u001b[0m     \u001b[39mraise\u001b[39;00m \u001b[39mIOError\u001b[39;00m(Errors\u001b[39m.\u001b[39mE941\u001b[39m.\u001b[39mformat(name\u001b[39m=\u001b[39mname, full\u001b[39m=\u001b[39mOLD_MODEL_SHORTCUTS[name]))  \u001b[39m# type: ignore[index]\u001b[39;00m\n\u001b[0;32m--> 427\u001b[0m \u001b[39mraise\u001b[39;00m \u001b[39mIOError\u001b[39;00m(Errors\u001b[39m.\u001b[39mE050\u001b[39m.\u001b[39mformat(name\u001b[39m=\u001b[39mname))\n",
      "\u001b[0;31mOSError\u001b[0m: [E050] Can't find model 'en_core_web_sm'. It doesn't seem to be a Python package or a valid path to a data directory."
     ]
    }
   ],
   "source": [
    "# Load a pre-trained pipeline (Web Small): https://spacy.io/usage/models\n",
    "\n",
    "#!python -m spacy download en_core_web_sm\n",
    "nlp = spacy.load('en_core_web_sm')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*.. the model’s meta.json tells spaCy to use the language \"en\" and the pipeline [\"tagger\", \"parser\", \"ner\"]. spaCy will then initialize spacy.lang.en.English, and create each pipeline component and add it to the processing pipeline. It’ll then load in the model’s data from its data directory and return the modified Language class for you to use as the nlp object.*\n",
    "\n",
    "Let's create a simple pipeline that does **lemmatization**, **part of speech tagging** and **named entity recognition** using spaCy models.\n",
    "\n",
    "*If you don't know what these NLP tasks are, please ask!*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweet_pos = list()\n",
    "tweet_ner = list()\n",
    "tweet_lemmas = list()\n",
    "\n",
    "for tweet in df_elon.text.values:\n",
    "    spacy_tweet = nlp(tweet)\n",
    "    \n",
    "    local_tweet_pos = list()\n",
    "    local_tweet_ner = list()\n",
    "    local_tweet_lemmas = list()\n",
    "    \n",
    "    for sentence in list(spacy_tweet.sents):\n",
    "        # --- lemmatization, remove punctuation and stop wors\n",
    "        local_tweet_lemmas.extend([token.lemma_ for token in sentence if not token.is_punct | token.is_stop])\n",
    "        local_tweet_pos.extend([token.pos_ for token in sentence if not token.is_punct | token.is_stop])\n",
    "    for ent in spacy_tweet.ents:\n",
    "        local_tweet_ner.append(ent)\n",
    "\n",
    "    tweet_pos.append(local_tweet_pos)\n",
    "    tweet_ner.append(local_tweet_ner)\n",
    "    tweet_lemmas.append(local_tweet_lemmas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['robot', 'spare', 'humanity', 'https://t.co/v7JUJQWfCv']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweet_lemmas[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['NOUN', 'VERB', 'NOUN', 'NOUN']"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweet_pos[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[https://t.co/v7JUJQWfCv]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweet_ner[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Shortville]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# but it actually works!\n",
    "\n",
    "tweet_ner[3]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: we are really just scratching the surface of spaCy, but it is worth knowing it's there.*"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Searching tweets\n",
    "\n",
    "Once we have represented Tweets as vectors, we can easily find similar ones using basic operations such as filtering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "robots spared humanity\n"
     ]
    }
   ],
   "source": [
    "target = 0\n",
    "print(df_elon.clean_text[target])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "condition = X_count_tfidf[target,:] > 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 5198)\tTrue\n",
      "  (0, 6617)\tTrue\n",
      "  (0, 6949)\tTrue\n"
     ]
    }
   ],
   "source": [
    "print(condition)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_filtered = X_count_tfidf[:,np.ravel(condition.toarray())]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<2819x3 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 16 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_filtered"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 0)\t0.495283407359234\n",
      "  (0, 2)\t0.6406029997190412\n",
      "  (0, 1)\t0.5867896924329815\n",
      "  (217, 0)\t0.2972381925908634\n",
      "  (271, 0)\t0.3284547085372313\n",
      "  (464, 0)\t0.2273880239746895\n",
      "  (473, 0)\t0.5667220639589731\n",
      "  (734, 1)\t0.3846355279044392\n",
      "  (940, 0)\t0.27312597149485407\n",
      "  (1004, 0)\t0.28161575586607157\n",
      "  (1550, 1)\t0.33303254164524276\n",
      "  (1862, 0)\t0.3196675199194523\n",
      "  (2493, 0)\t0.2685018991334563\n",
      "  (2559, 0)\t0.31145247014227906\n",
      "  (2565, 0)\t0.2645117238497897\n",
      "  (2661, 0)\t0.2729016388865858\n"
     ]
    }
   ],
   "source": [
    "print(X_filtered)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([   0,  217,  271,  464,  473,  940, 1004, 1862, 2493, 2559, 2565,\n",
       "        2661,    0,  734, 1550,    0], dtype=int32),\n",
       " array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 2], dtype=int32),\n",
       " array([0.49528341, 0.29723819, 0.32845471, 0.22738802, 0.56672206,\n",
       "        0.27312597, 0.28161576, 0.31966752, 0.2685019 , 0.31145247,\n",
       "        0.26451172, 0.27290164, 0.58678969, 0.38463553, 0.33303254,\n",
       "        0.640603  ]))"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy import sparse\n",
    "\n",
    "sparse.find(X_filtered)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweet_indices = list(sparse.find(X_filtered)[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TARGET: robots spared humanity\n",
      "1)@JustBe74 important make humanity proud this case particular duty owed American taxpayer\n",
      "2)@pud Faith restored humanity French toast money\n",
      "3)humanity have exciting inspiring future cannot confined Earth forever @love_to_dream #APSpaceChat\n",
      "4)@ShireeshAgrawal like humanity\n",
      "5)Creating neural lace thing that really matters humanity achieve symbiosis with machines\n",
      "6)@tzepr Certainly agree that first foremost triumph humanity cheering good spirit\n",
      "7)@ReesAndersen @FLIxrisk believe that critical ensure good future humanity\n",
      "8)@NASA #Mars hard x99s worth risks extend humanity x99s frontier beyond Earth Learn about neighbor planet\n",
      "9)Astronomer Royal Martin Rees soon will robots take over world @Telegraph\n",
      "10)@thelogicbox @IanrossWins Mars critical long-term survival humanity life Earth know\n",
      "11)humanity wishes become multi-planet species then must figure move millions people Mars\n",
      "12)Sure feels weird find myself defending robots\n",
      "13)Neil Armstrong hero humanity spirit will carry stars\n"
     ]
    }
   ],
   "source": [
    "print(\"TARGET: \" + df_elon.clean_text[target])\n",
    "\n",
    "for n, tweet_index in enumerate(list(set(tweet_indices))):\n",
    "    if tweet_index != target:\n",
    "        print(str(n) +\")\"+ df_elon.clean_text[tweet_index])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Questions\n",
    "\n",
    "* Can you rank the matched tweets using their tf-idf weights, so to put higher weighted tweets first?\n",
    "* Which limitations do you think a bag of words representation has?\n",
    "* Can you spot any limitations of this approach based on similarity measures over bag of words representations?\n",
    "\n",
    "#### Exercises\n",
    "\n",
    "* Find the highest IDF tokens in the corpus and discuss your results.\n",
    "* Find the most frequent named entities and discuss your results.\n",
    "* Are there trends in the topics of tweets during the day? And over time?\n",
    "* Perform a **sentiment analysis** of the corpus, see here https://spacy.io/universe/project/spacy-textblob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install spacytextblob\n",
    "#!python -m spacy download en_core_web_sm\n",
    "#!python -m textblob.download_corpora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.125"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import spacy\n",
    "from spacytextblob.spacytextblob import SpacyTextBlob\n",
    "\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "nlp.add_pipe('spacytextblob')\n",
    "text = 'I had a really horrible day. It was the worst day ever! But every now and then I have a really good day that makes me happy.'\n",
    "doc = nlp(text)\n",
    "doc._.blob.polarity "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}