AB_Testing_Project_Part1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "E-Learning Company Webpage A/B Testing - Part 1\n",
    "-----------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Key Project Steps\n",
    "1. Problem statement\n",
    "2. Dataset description\n",
    "3. Data cleaning and wrangling\n",
    "4. AB testing\n",
    "5. Regression approach\n",
    "6. Influence of other factors\n",
    "7. Recommendation and conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Problem Statement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**An e-learning company has developed a new web page to try and increase the number of users who enroll in their free data science program (in this context, the enrollment will be referred to as converting). They run an A/B test to understand if they should implement this new page. Further, they also tested this web page separately in three countries - US, UK and Canada to understand if the country of residence plays a role in determining the conversion rate of users. The goal here is to analyze these results and help the company make a decision**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import relevant libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "import random\n",
    "import statsmodels.api as sm\n",
    "%matplotlib inline\n",
    "random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Dataset Description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294473</th>\n",
       "      <td>751197</td>\n",
       "      <td>2017-01-03 22:28:38.630509</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294474</th>\n",
       "      <td>945152</td>\n",
       "      <td>2017-01-12 00:51:57.078372</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294475</th>\n",
       "      <td>734608</td>\n",
       "      <td>2017-01-22 11:45:03.439544</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294476</th>\n",
       "      <td>697314</td>\n",
       "      <td>2017-01-15 01:20:28.957438</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294477</th>\n",
       "      <td>715931</td>\n",
       "      <td>2017-01-16 12:40:24.467417</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>294478 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        user_id                   timestamp      group landing_page  converted\n",
       "0        851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1        804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2        661590  2017-01-11 16:55:06.154213  treatment     new_page          0\n",
       "3        853541  2017-01-08 18:28:03.143765  treatment     new_page          0\n",
       "4        864975  2017-01-21 01:52:26.210827    control     old_page          1\n",
       "...         ...                         ...        ...          ...        ...\n",
       "294473   751197  2017-01-03 22:28:38.630509    control     old_page          0\n",
       "294474   945152  2017-01-12 00:51:57.078372    control     old_page          0\n",
       "294475   734608  2017-01-22 11:45:03.439544    control     old_page          0\n",
       "294476   697314  2017-01-15 01:20:28.957438    control     old_page          0\n",
       "294477   715931  2017-01-16 12:40:24.467417  treatment     new_page          0\n",
       "\n",
       "[294478 rows x 5 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/part1_results.csv')\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 294478 entries, 0 to 294477\n",
      "Data columns (total 5 columns):\n",
      " #   Column        Non-Null Count   Dtype \n",
      "---  ------        --------------   ----- \n",
      " 0   user_id       294478 non-null  int64 \n",
      " 1   timestamp     294478 non-null  object\n",
      " 2   group         294478 non-null  object\n",
      " 3   landing_page  294478 non-null  object\n",
      " 4   converted     294478 non-null  int64 \n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 11.2+ MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(df.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1 Preliminary Data Exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No of entries: 294478\n",
      "No of unique users: 290584\n",
      "No of times new_page and treatment don't line up: 3893\n",
      "Rows with missing values:\n",
      "user_id         0\n",
      "timestamp       0\n",
      "group           0\n",
      "landing_page    0\n",
      "converted       0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "no_of_rows = len(df)\n",
    "print(f\"No of entries: {no_of_rows}\")\n",
    "\n",
    "no_of_unique_users = df.user_id.nunique()\n",
    "print(f\"No of unique users: {no_of_unique_users}\")\n",
    "\n",
    "no_of_mismatch = len(df.query('group == \"treatment\" and landing_page != \"new_page\"')) + len(df.query('group != \"treatment\" and landing_page == \"new_page\"'))\n",
    "print(f\"No of times new_page and treatment don't line up: {no_of_mismatch}\")\n",
    "\n",
    "rows_with_na = df.isnull().sum()\n",
    "print(f\"Rows with missing values:\\n{rows_with_na}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Data Cleaning and Wrangling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No of times new_page and treatment don't line up in the new dataframe: 0\n",
      "No of entries in the new dataframe: 290585\n",
      "No of unique users in the new dataframe: 290584\n",
      "No of duplicate users in the new dataframe: 1\n",
      "Duplicate user id: 2893    773192\n",
      "Name: user_id, dtype: int64\n",
      "No of entries in the new dataframe after dropping duplicate users: 290584\n"
     ]
    }
   ],
   "source": [
    "# create a new dataframe, ignoring entries from the original dataset\n",
    "# where treatment and new page do not align\n",
    "df2 = df[((df.group == 'treatment') & (df.landing_page == 'new_page')) | ((df.group == 'control') & (df.landing_page == 'old_page'))]\n",
    "\n",
    "# double check that there are no misaligned entries\n",
    "no_of_mismatch_df2 = len(df2.query('group == \"treatment\" and landing_page != \"new_page\"')) + len(df2.query('group != \"treatment\" and landing_page == \"new_page\"'))\n",
    "print(f\"No of times new_page and treatment don't line up in the new dataframe: {no_of_mismatch_df2}\")\n",
    "\n",
    "# no of entries in the new dataframe\n",
    "no_of_rows_df2 = len(df2)\n",
    "print(f\"No of entries in the new dataframe: {no_of_rows_df2}\")\n",
    "\n",
    "# check for duplicate users\n",
    "no_of_unique_users_df2 = df2.user_id.nunique()\n",
    "print(f\"No of unique users in the new dataframe: {no_of_unique_users_df2}\")\n",
    "print(f\"No of duplicate users in the new dataframe: {no_of_rows_df2 - no_of_unique_users_df2}\")\n",
    "\n",
    "# find the duplicate user and drop him\n",
    "dup_user = df2[df2['user_id'].duplicated()].user_id\n",
    "print(f\"Duplicate user id: {dup_user}\")\n",
    "df2.drop_duplicates(subset=['user_id'], inplace=True, keep = 'first')\n",
    "\n",
    "# no of entries in the new dataframe after dropping duplicate users\n",
    "no_of_rows_df2_after_drop = len(df2)\n",
    "print(f\"No of entries in the new dataframe after dropping duplicate users: {no_of_rows_df2_after_drop}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. AB Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.1 Probabillities and Conversion Rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overall conversion rate irrespective of page: 0.11959708724499628\n",
      "Control group conversion rate: 0.1203863045004612\n",
      "Treatment group conversion rate: 0.11880806551510564\n",
      "Probability that the individual received the new page: 0.5000619442226688\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "145274"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# overall conversion rate irrespective of page\n",
    "convert_prob_overall = df2['converted'].mean()\n",
    "print(f\"Overall conversion rate irrespective of page: {convert_prob_overall}\")\n",
    "\n",
    "# control group conversion rate\n",
    "convert_prob_control = df2['user_id'][(df2['converted']==1) & (df2['group']=='control')].count() / df2['user_id'][(df2['group']=='control')].count()\n",
    "print(f\"Control group conversion rate: {convert_prob_control}\")\n",
    "\n",
    "# treatment group conversion rate\n",
    "convert_prob_treatment = df2['user_id'][(df2['converted']==1) & (df2['group']=='treatment')].count() / df2['user_id'][(df2['group']=='treatment')].count()\n",
    "print(f\"Treatment group conversion rate: {convert_prob_treatment}\")\n",
    "\n",
    "# probability that the individual received the new page\n",
    "prob_new_page = df2['user_id'][(df2['group']=='treatment')].count() / df2['user_id'].count()\n",
    "print(f\"Probability that the individual received the new page: {prob_new_page}\")\n",
    "\n",
    "df2['user_id'][(df2['group']=='control')].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.2 Hypothesis Formulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Under the null, p_new and p_old are equal to the overall conversion rate (irrespective of the page). Therefore the null and alternate hypothesis can formulated as follows:\n",
    "\n",
    "* **Null Hypothesis: The conversion rate of the old and new page is the same and is equal to the overall conversion rate**\n",
    "\n",
    "* **Alternate Hypothesis: There is a difference in conversion rates between the old and new pages**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Conversion rates of the new page under the null: 0.11959708724499628\n",
      "Conversion rates of the old page under the null: 0.11959708724499628\n",
      "Number of individuals in the treatment group: 145310\n",
      "Number of individuals in the treatment group: 145274\n"
     ]
    }
   ],
   "source": [
    "# conversion rates p_new under the null\n",
    "p_new = convert_prob_overall\n",
    "print(f\"Conversion rates of the new page under the null: {p_new}\")\n",
    "\n",
    "# conversion rates p_old under the null\n",
    "p_old = convert_prob_overall\n",
    "print(f\"Conversion rates of the old page under the null: {p_old}\")\n",
    "\n",
    "# number of individuals in the treatment group\n",
    "n_new = len(df2[df2.group=='treatment'])\n",
    "print(f\"Number of individuals in the treatment group: {n_new}\")\n",
    "\n",
    "# number of individuals in the control group\n",
    "n_old = len(df2[df2.group=='control'])\n",
    "print(f\"Number of individuals in the treatment group: {n_old}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Simulated conversion rate for the new page: 0.12008808753698988\n",
      "Simulated conversion rate for the old page: 0.11853279196201225\n",
      "Difference in simulated conversion rates: 0.0015552955749776332\n"
     ]
    }
   ],
   "source": [
    "# simulate n_new transactions with a conversion rate of p_new under the null\n",
    "sim_new_page_converted = np.random.choice(2, size=n_new, p=[1-p_new, p_new]).mean()\n",
    "print(f\"Simulated conversion rate for the new page: {sim_new_page_converted}\")\n",
    "\n",
    "# simulate n_old transactions with a conversion rate of p_old under the null\n",
    "sim_old_page_converted = np.random.choice(2, size=n_new, p=[1-p_old, p_old]).mean()\n",
    "print(f\"Simulated conversion rate for the old page: {sim_old_page_converted}\")\n",
    "\n",
    "# difference in simulated conversion rates\n",
    "diff_sim_page_converted = sim_new_page_converted - sim_old_page_converted\n",
    "print(f\"Difference in simulated conversion rates: {diff_sim_page_converted}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.3 Sampling Distribution\n",
    "\n",
    "* A sampling distribution of the difference in conversion rates between the two pages over 10,000 iterations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_diffs_sim = []\n",
    "for i in range(10000):\n",
    "    p_diffs_sim.append(np.mean(np.random.choice(2, size=n_new, p=[1-p_new, p_new])) - np.mean(np.random.choice(2, size=n_old, p=[1-p_old, p_old])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observed difference in conversion rates: -0.0015782389853555567\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuwAAAIUCAYAAABB4XmIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdebwkVXnw8d8DBhBQUQENozBIxIXEhUVRUAY0bghGQUWD74uKKBEUXBJXGEMUNyIBccElIyIuKKK4Iq8OoqIIhIjIIsKACjqAgGzD+rx/nOq55zbd93bP3Lldfef3/Xzq09WnTp06Xd3V/dTpU6ciM5EkSZLUTmuMugKSJEmS+jNglyRJklrMgF2SJElqMQN2SZIkqcUM2CVJkqQWM2CXJEmSWsyAXdJYi4glEZHNtKBKX1yl7zO6Gs6cfq91BPVYVNVjYZW+sEpfNML6LajqsWRU9ZgJEbFhRHyiee/vmguvSdLwDNilOS4iNoiI90XE+RFxS0TcHhF/iojzIuLzEfGKUddxddJ1IpFNEHZzRFwRET+MiHdExENW4fb3aQLrhRHxhFW1nVWpCcg7r+GfRl2fVeyzwGuBzYA1B10pIg6KiBMj4vKuz9s+U6yzYUQcERG/jYhlEfGXiPhBRDx/2EpH8X8j4icRcWNE3BoRv46IQyJivT7rbBYRxzbHwu0RsTQivhERO8zUNqRxFd44SZq7IuKBwFnA302R7aeZueMsVWnGNa2NmzVPd87MxU36PwAPaNIvycyls1+7e4uIxcBO02S7BXhtZn6ha91tgXWap+dn5o0ruf1XZuaiFSjjkUDnpOLKzLyySV8IHNqkfy4z9xm27AG3P+V2IuIBwD80T5dl5tmroh6rWkSsBdzKRKB+IHAeA7ymiLiBic9/red7HhGbAT8GNu1T5CGZediAVSciPgf8nz6L/4dyrN5Y5d8aOA14YI/89wCvyszPrcw2pHF2n1FXQNIq9UYmgvUrgcOAy4ANm/TnUX4M55zMPH/UdRjAd4H3AesDTwReRwmY1gM+HxF3Z+aXOplHHXhGxPqZeXNm/hb47SjrMpUmSPvJqOsxAx7K5Fb1Y3LwVrbzgUuAs4GFwMbT5P8ME8H6L4D3A48G3kv5N/49EfHDzPzpdBuOiL2ZCKRvA94CXAP8B7Al5bP+Qco/B0TEfYATmAjWvwN8gnJi+eZm+5+IiDMy87IV2YY09jLTyclpjk6UH75spjf0yXO/rufbAsdTfvCvAe4EbqK07L0HWL8r/8JqG4soJwHnUH5Efwcc0OR7JPBN4K/ADcCXgI26ylpclbUPsC/wK2AZ8AfgA8A6XessqdZZ0K+sKn1Rlb4Q2B34eVPfa4BPAuv12E+vBS4EbgcuBd4K7FKVtWTA96Su16KuZQ+gBFid5Uvr/T3Fa3008AXg98AdlBb6JcC3Ou97sz9zimlRn/fz2cDPmjLP67UPp/gsPBn4f8DNwPXNe/7wqd7zrmV1/eY301SvYUmz3oKp3pfmPf9Os3/vBK4FTgX27JG33ufPoASGlzSfg8uBNw15TD4eOA64oinjr5R/wd4CrN1nv/R8r4bYZv0a9umx/O+r5fcAD6uWfapaduKA2zunWuedVfoOVfoy4EFN+m5V+o3Afat1flAt+9CKbsPJadwnW9ilua3+O/hfIuLPwOLM/HMnMTNv6lpnW+Cfu9LWpwQajwd2jYjtM/OuHtvbEXgFE9fHPAI4OiIeBrwGeFCV96XABsBz+tT9rcBjq+fzgH8FHhcRz8vMmejP93ImulZA6W6yH+XH/nWdxIg4hHKy0rEFpfXu3Bmow3KZeWNEHASc0SRtRDkB+kq/dSLiwcBPmbxv/4bSTWgzyj8pR61glZ7G5PdzGNsBewFrV2kvBXaIiG1yRF2UIuIoSteS2oOBfwT+MSKOzcx+rbIfp5x4dswHjoiIq7L6J2SKbe9FCdb/pkpei7KvtgP2iohdMvOvA72YmfOMav6KzPxD9fynlBNngJ2nK6jpjvTEKqn+p+MXwF2Uf/fXpgTXp1BOfDvOzczburb/zGZ+l5XYhjTWvOhUmtu+Xc0/itLC+aeI+ENEfDEido+I6FrnV5S/oV9ICWJ2Bl4M/LJZvk2zrJctgBOBXYGvVen/RmmlfymTg6VnR8Sj+pT1GEpQ/DzgP6v051AC7ZnwSOCLwPMpwVjHqyNifYCI2Bw4pFp2SpP/3ygtkzPtZ5R+yx3bTpN/ZyaC9R9R6vZs4FXAp4E/Nsu+QwnAz6vWfV+T9jRK14dujwAuogTtzwaOHvRFUE62vt/U50BKKzvAw/psaxBXN3X97yrtu0y8hj2nWjkidmfy5+8jlM/XBygnaQD7RcSL+xTxCMqJ2/OB06v0N05X8Yh4KKXbSSdY/y6lZflfmDix3gY4vJk/kHLc1aZ6r1bGI6r5P3Utq58/OCI2GKCs+jtl+frNSf511bIthtx+nX/YbUhjzRZ2aQ7LzOMjYntKUFD/wM2jtH7uBXwjIl5YtVifBWxN+Yv+sZRuGt0n99tTAvNuVwF7Z+ZdEXENsEe1bP/M/C5ARLwO2KpJ3xK4uEdZX83Mf2vmvxsRW1ICJYCXULqArKwLgH/OzIyI7wL/F1iX8t24OaVb0IuY6Ee8FHhxZt4OfDsiNqLspxmTmfdExPVNPaD8CzGV+l+Uqyn78vLMvJsqsG1atJdGRJ3/t5k5VV/vW4Fd6n9khnAVZV/dARARawMfbpbtGRGvzcyhrp9o9vtPIuKZVfLSaV5DbZ9q/pTMfFMz/92I2IKJgH8fen++P5GZCwEi4lpKVyoon+HpvISJ9/Qa4EWZuawpaw3go82yvSPiDZl5fkRM+vdriNc5rHpElTu6lnU/X5/SpW2QsqYrb/0ht98r/6DrSGPNFnZpjsvMAyit1YdQ+ul2j5rwAkrLd8dnKS2pO1AuAuv1PdFrJAeAs6quMtd1LTuzmr+2mn8QvXUHJ/XzRzIzftg5UWmCx+t71Kve1jlN0NivjistItZk8j6ZKjiC0n3mgmb+5ZSLQTvD230sIh7bf9Vp/XQFg3WAX3SC9Ua9rzagXPg82x5dzU/1+Xo0vf2wmq8/3/0+w/22fXYnWO+x7fsDmwxQ3ky6pZpfu2tZ9/ObmdotXc+nKq9T1qDb75V/0HWksWbALq0GMvPizDwsM59N6a/7HCYHp08GiIh5lO4PHUcCz6L8DX9cld7vu6M+GZjUepqZ/QLP7i45s+kvXc/rfvmdetV95Wei3/x0dgTuWz2fcmSYJvDbgdLS/23Khb5rUv7B2B/4WUT0G6pvOlev4HrDqPfp8n99m38vZtrKftbqz0uvazhW5bZXpcuq+Yd2Lfvbav66KY7juqz6PV1eXkT8DeX7p+N3Q26/zj/sNqSxZsAuzWERsXMzFvtymXl3Zn6fcnFWR+e74OFV2nWZeXBm/qD5K37eKq5ut+6bpdTPL53FetTDF27dBAQdT5vJDTX9g4+skq6l9D2fap3IzBsz84jMfH5m/h2llbZzDcEDKP20O+oTqel+A1bmBOVJXfuqfv9uZOJflvrE8WHV/G5TlD3Ma6hd2Kc+3c8vGqLMFdn2NhGxTvW83vZfmZ0TpVr9z8GmXSd4T6/mfzRdQVmG1PyfKqk+RnZgonvZHZQLSru3v3VErFs9r7f/w5XYhjTW7MMuzW2vBl4YEd+m/Nj+jhKE7cjEyAsw0V2lbul6cES8k9LCuyeTR5KYDXtGxOGUi/t2YaL/OkwxasoqcBLl4tc1KS15X4qIz1BasA9YybI3jogdKX1yt6a0iHdOmhI4MDOn+0v/SRHxKeDrlP7rf6J00diqylMHh3VXjhc3N566A7g4M69Z0RfSwzzgKxHxacpoKvVoPF+t+q9fUqUf3PTbvj9TXxtQv4anRcSulJOAP2XmVCdzi5i4YHq3iPgw5WY9T2fy9RaLpihjRX2FckHpupQx0b8aEZ/g3hfhHt9nBKahRMSzmOgzXwfAWzc3VQL4SWZe2/SX/xHlAuYATmyOvccyMdZ5Av9VlT+fMqxlWZhZ/4PwX0DnJkdvj4jrKP32D6/yHJeZnX8svks5MX4kcD/Kvvk45bjfqclzO2Vs9hXdhjTeRj2upJOT06qbKOOpTzVudVIC4jWrdb7YI89dlLsg3mscaLrG3q7S59dldNVrcbVsnz7p5/ap76nAGtU6S6plCwbYxqIqfWFXvfqV9e4+dfmfan7JgO/J4j5l1dPNlIt3u9e9V/0oFwBPVdZfgc2qMvbrk2/vqd7Prnr03Idd615KGeO8ezt/AB5SrbM5JRjrznd+1/P51TqPAe7usc6nm+UL+r0vlOszptpfxw74mZhfrzfge78X5eSo37bPBu6/MtvoU+9+U/16NqeM498vb/exMmXdmPq75zxgg67821Ku1+iVv3On05XahpPTOE92iZHmtoWUIedOpvwl/xdKoHMDpVX9zcCzsowo0rEvpVvGHyg3E/oFpUtF/bf1bDiKMmrLrygB3VWUUUZekEOOLrKystySfX9KC/YdlJbFd1DuqtjRfSHcoO5p1r2Sso/fAWyRmccPuP6llBba0yn76HZKoHwlJaB5cmZeUeX/DKUV8g+s2rvc/oQyLOgZlNFmbqS0Mu+Qk+8DcDnwT5QA6w7KPwQfZYruRpl5IaXl9wLKax1YZh5IaWX/HqVbzl2U4+I04CWZud8w5Q257S8BT6K8L7+n1P1myk2A/hXYMWd/DPZO3S6nDCt5JOWfuDso3xM/pBxzC7tWqe/Aejv39n8oQ4ueSXmNy4DfUIbF3DG7+sJnuYvvEymfzz9Q9s11lGFUd8rMz67sNqRxFpk56jpIEgARsZiJv8BfmZmLRlebCU0/8Xt9WUbEEUBnaMCTM7Pf+PTSnBIRL6Xc1wHgXZk502PDS6rYh12SpvfKiHgy8FVKX9v1KBdF1n3Yj+u1ojRHda6B+V/KjackrUIG7JI0vbUofb/7dZc4JjO/Pov1kUbtGZTuda/OGbhIVtLUDNglaXpnUe58uS3wEMp35zVN+qeyuYOrtLrIzEeMug7S6sQ+7JIkSVKLOUqMJEmS1GIG7JIkSVKLGbBLKykiFkZENtOiUdenIyKWVPVaMOr6DCIiFlR1XjIL2+u7jyJifkScEBF/iIi7mzyLq+WPi4hTIuJPbXz/55rm/ejsZ/tytthsH8fS6sCLTqUuEbEB5SYmuwGPoBwn11Nu6HI+cGpmfn50NZxdEfEEyo1toNw1ctEIq3MvXWO3Qxm5Yhnlpiu/o9wQ5zP1zXoGKHMN4OvAE/osX59yO/VNVqzWaruImAe8jnLzpy2B9SkXGl9BuZnP8Zn5+9HVUMPq8V0B5eZhNwGXUG4wd2Rm3rqS29mHcidYKPdnOG9lypPAgF2aJCIeSBn54++6Fj2kmR5PuYV3HbB/lhIUAgwcFI6RJwCHNvOnU25L32ZrUsZJXw/YFNgZeEdEvDYzv9CVd09gnWb+/Cp9cyaC9bsod1T8PeVunQBPZiJY/wuwD+Wkbi6+/21xNVPc/XQmRcSrKHdbvW/Xok2a6SmU74ODZqM+Y+h/mHivlo2yIgNYA3gAsF0zPTsiFvS6UdoQ9mHixGAJ5S6+0koxYJcmeyMTwfqVwGHAZcCGTfrz6Lqde2Ze2eTV6H0XeB+lNfSJlBbSTSnB++cj4u7m9vDA8tuh9zKvmr8qM784xfJfZ+YpK13zPiJi/cy8eVWVPy4y83bgJ6t6OxHxz8BnqqTfAkcBF1JO7ramnMCtNob9DGbmjczCe7WSOt8V96d8T+zWpD+dckL2sxHVS+otM52cnJoJ+A6QzfSGPnnu1/V8YbXOoip9nyp9MaVV9sfArcAfmvXWpLTUfZ7SUnsz8G1gi65tLK7K2qdrWVbT/Cp9SZW+oEp/GfAN4FLgBuBOSveR04FX0Qz32qPse01d9XhWU+6fgDso3Qe+CTytz378F+Ai4HZKUPRmYJeq/CUDvmf1vlnUtewBwNnV8qXA+lPto6607mnhFMsmvTfAk4AvUlrm76C0wJ8G7D7Na3glpeX2wma9I6t8jwE+TTmJXAb8Ffgp5bMWXWUu6qr37sDPgdua9+aTwHo96rIFcEzz3txK+UxeBBwLrN2V96XAqcC1TV2vbl7z43qU+9wm7zWUz9wNTbknAM8d4H2eT//PXv0ePBZ4L6Xryu3NNvYe8LN0v+a1dMr6Zf15qfKtCTymK+0RwMcpx9WyZr/9L/DvwAZTfWcAOwA/BG5p9suXgY2r/GdW+ffqKmtTSiNCNu/XA1bVZxDYnnKMX928h39tXu/XgH+uylpQlXWv47j5LH6Hcjze2ezzU4E9e+RdUpX1DOAtlO4rtwOXA28a5L2d6ruC0iBTf4b2qpatSTlhO4PyvX1rs+0rgC8AT+jznd9rqre5LqXr5VnNfux8D/4nsFFXvdcA3lDlvZNyHJ1DOY4fPeg+cBrfaeQVcHJq09T8uHW+XC+iBCQPmWadhX2+kOsv799Tfoy7v8A/Qeln3Z1+AbBGVVb9Q7NP1/aHDdi/NM2PypF9yr7XVOV7/xT57gZe11Xn9/TJe241v2TA96zeN4t6LN+xaxsvmWofMQMBO+Vk5O4p8r1vitdwSa/3g3IdwW1TlHk8k0+2Fk1R5vLPX1c9dqX357QzbdDkW4MSrPTLtwzYrSp3FyaCymnr0ed9nl+vM8Ux0O+1PmWAbezVtU7Pk80e6+1E6Qfd7/VdBszr851xGSUA617ne1X+fav0U7q2/bZq2XFV+ox+BoFHN+9rv/Lq+i6o0pd0beeoKcpI4JNd+ZcM8N7uNeD7VL/G+ru6O2DfsVq2zjT1vR14co/v/F7Tomp750+R7w/A5gN8Xw71+p3Gexp5BZyc2jQBe0/xBfpFSstQd0vmwj4/At1f3qcBzwc+1pV+HaVFa29K600n/dlVWfUPzT5d26/Lml+l1z90C6r0lwOvpfwFvIASTL2K0mKTlD7bD23y7khpreyU8z9N2o6dHzVKy2ln+a3AW4FnAm9i4gf+TmDLJv/mzTY663ydEii+hckBwZIB37N63yzqsXwNJgehH5xqH1HuZnpglX519Zo3nWafbAxsxUSgdHeT9x+B/Sj/onTW26XPa0hKK+Y/AS9o9s1GTA4IPw48m/KZqV/Dq6oyF3WVeUJTVv35u5OmBbnZxo3Vst81dX4WpcvAz5kI2Pev8l1DCQ6fSelC1gnMbwAe2OT/aJX/GEpL6W7A64GT6vdkivd5fv16pjgGbgEOphyrdVD0xQG28aGuctYYYJ11gD9W6/0CeCHwCsr3Rif9232+M5LSurwb9w7MHtXkvx+lxT4pLd4bVmX9qsq/U5O2Kj6Db6mWfYXy+XtuU+bxlItwO2UtqPIuqdJ379rGfzZlvJ/JJ3Qv7nOM3tXsu1276nvmCnxXfIdyzD6XcrFpJ/2XTD7xvQ/lX5KXA8+hnJw9t6n7pPeWcvzvSPlO6Cx7LxPfD49s8n25Wv4/lBPF5wBfrdJ/XNXh10wcrwdQrsvZk3KythjYY1S/mU6zN428Ak5ObZsowcVUrYEnd32hL6yWLarS96nSbwMe1KRv1FXe/tU636rSD6zS6x+afbrqW5c1v0qvf+gWVOkPpvxA/ooSBPR6rbv1eR2Le+yv+kfmuOrHaUdK957OssOb/G+u0v4ErFWV9YFq2ZIB36963yzqk6cOnI4dYB8tmKoeU+0T4MPVsh907Y/PVMu+2Oc1nN1jewdUy8/vKvM/qmVnVussqtJ/zcSdrbtPYP6hSX99lXYTVYtwj/rU3Yw+2FWf+l+S1zb56xOcNwF/uwLH5fyqjJziGHhLlf7SKv2cAbbxqSr/HwasVx2E3l6/Nkpg2Vl2D003FyZ/Z1wD3Lda58JqWX0cfrZK379J+/sq7bfVe7wqPoOvqZZ/mHLyGn32yYIq75Iq/aQq/Ztd65xYLatPbpZU6R+t0p9cpV+3At8V3dPdlBPhB/ZY76lN/a5s3uPuda+bYjvd39cbMLnB4mXVe7OAckLWWdY5Yftp8/xWygn0AwZ5vU5za/KiU6lLZh4QEUcDL6F8iT6Z0he64wWUQOBLPVbv56LM/Eszf13XsjOr+Wur+QcNUf5AIuK+lC//R02T9YFDFPvYav4VzdTLVs1jPQLPOZl5R/X8p0NsdyARsSaT9+UNM72NLvX+eGYz9bJVn/STpinz7yn9aYcp84eZ5Zc/M++JiOspfWhhYt/U2/hFZv6xT1nded/aTFPV5/OUC7rXA44AjoiImyhdv74PHFUdHyvrh9V8fawNcjzVn40NImKNzLynb+7i0dX87zLz6up5feFlUI67pV3rn5mZt1XP+9X5s5R/4gD+mRJc/nO9vPMes2o+g9+gtDQ/lHLS/Wbgtoi4iLLP/yunH+ay3lfdF6X+hNJq3J2vtjLv7XTWoATm61L6+gMQEf9IuUB1zSnWHeb7csuusk6YIu9WwMWUrpNPpYxa9P2mXkspo898jfLe3zVEHTSGvHGS1ENmXpyZh2Xmsykt0s+h+hKnBPHD6AwHSI8AoF8AGXWVqvnlJ9oRsdGQ9XghE8H6LZQLmXamDMFWD2u4Kr4b7tc8xhR5plq2onZk8vB8/UaGmW3365N+dZ/0lSmzOxiuf9yj6xEmf95Wxv0AMvMiyjCZh1Mubr66WbY9ZcjQU5sTq5lQv9Zer3Mq51Tz61FGC5nOVOUOsh8HeW/IzJ9QgjeAp0bE5pSuFFBahxcNsK1uA38GM3MpZYScQyit9ldSugM9kRK8nxER959meyt7fPd7b1fE5yiB87ZMjPL1OO7dEPNWJgLssygnFU+jtIx3rIrvLZg4fj5P6YrzyaYON1C63zyrSfvQKtq+WsSAXapExM7NWOzLZebdmfl9St/Ujtk+duqThYdV87t1Z5zGptX89zLz6MxcTOke87Deq0waxrLX676wmj88M6N7Av6GMiQmlFElOraOiL+png8SIA2suQnWkVXStZS+q6tSvT++2Gd/rEFpKe+lV5BXl/mzXmU25U4XME3lgmp++4iY6qZQdX1e26cu96X0byYiIjMvzcx3ZOaCzNwE+FtKdweAbYBHrkTdZ8q3mfwv139GxHrdmSJizYh4TPO03hdbRMRDq+c7VvPJRMC9oj7bqQKl69785vl3ulr2Z/wz2LyHVzcNGc/KzM0oLdudfwg3o7QCT6Wu1w5dy+rnF01TzozIzHsy8xzKRb0dO0bEi6rn9XfmYZn5tebkaaoeClN9Z15COcHqeFS/YzkzPwfL9/2PM/N1mfnkzHwgkxuN6pMHzVF2iZEmezXwwoj4NvAjJkZw2ZHJfyuf2WPdVemSav7gpjvB/SkXgg3jsmr+GRHxCkrr/1vo/7du/dfz45ofs6XADZn5a0qf2M4P3FubltLTKT9am1JasF4IvJjSt/MkSl/1NSh/r38pIj5D+Rv8jUO+nm4bR8SOlNbRrSkXRz68WZaU6wJW9ZjmiyhD4q0BvKx5r75FuaD24ZSbb70IeDeDt4p+mYnx5Z8aEV+l/JV+I2VM+Mc2ZZ5A6R+9Ir5CaQG/X7Od0yPig5Sgej6lO8bzMvMGynu+dbPeEc0/Pb8E1mpe4w6Uvt2Pa9Z/c0Q8lxIQX0FpKX0k5XqOjnUYscy8KSIOolxECWVYxHOaLnKdcdifQBmH/XuU9/lU4CrKDZXWAr4eER+g7MPDq+K/27RSr4zPUa4HuA8TJ8Awedx4WDWfwRdHxJsoXWMuo3wHbEK5iLxjuvdwEeW7AGC3iPgw5WL8pwN7dOWbNZn5g4j4CRMnWIdExNebLkaXMfGv5MERcSdl6NP/mKLI+jvzxRGxhNI3/eLMvCYiTqJ8HwJ8JyI+RGnI2IByrD2bsl87J7EnRsRdlO/PP1L+HX1WtY2RHzuaBaPuRO/k1KaJ8kPd76KkznQ6sGa1zsJq2aIqfZ8qfXHXdury5lfpi6r0hVX65vS+2Kl7aLC6rCVV+oImbV3KD0N3OVcz+WK3fapyHkjvof5Oq/J8oMfy7mlBlf8/+uS5qJpfMuB7tniAbd9Mj7G4e+2jJn3BVPWY6r1tlr+eqYfU697Hi3uld5X5QqYe1rH7M9PzszTN696dySMVdU/1sI4nDLDf5zf53zZNvnOZZkQWBr/otD4Gpnwfp9jWq6bZD8nk4U93YvphHR823XfGIJ8FJo9okpQLt++zqj+D3HvIy+7p9zT3qJhqvwNHT1POsV35l1TL6s9q38/DgN8V3fv92V31eEGT/pw+9fxRv+1T/lnqtc7ezfKNmHpYx0n7jXJyOFXe/xr0s+00vpNdYqTJFlJaeU+mBLB/ofzo3UBpVX8z8KzMvLtfAatCZl5OGWLtPEpLzZ8of4kPdav2zLyVMqTe1ymv7UbKzY12BP7cZ53rKa1xZ1NOGnrl+TdKi8/XmbipyvXAbygjx+xJGRawk/9dlKETL2lezxWUfX/gMK+nj3soJxhXUi5SewflRlTHT7nWDMrMYyj9s7/Q1OMOyg1PLqaMNvF/6H1h31Rlfp3SX/hYJm7Oc0sz/y3K0IsfW8l6f5PSgvwJyqgjyyiB6yWUGzbd1uS7JzNfTrkw+3uUkU7uonQn+RWlX+3zKEEcTZ6PUgLzpU3e2yifjw8Bz8jpL+6cNZn5WUrr5nuZ6DN8J+WzfSbwTsrFs538p1P22ycpwfkdlNd3PuXkdOvM/MMMVa+7Nf1z2eOCw1XwGfwFZXSYM5m4OdrtlM/fJyjj3N80XSGZeSDl5PN7lM/LXZTvotMo90jYb4g6zZgs3R5/WSW9u0n/HqX1/zzKe3ol5eLbqer5Gcq/K3+g687YTZnXUP69eQvle/FGyufrqub5e5n8j8PHKRduX0T5Xr27WefnlN+rg4d5rRpPnSGgJEmSJLWQLeySJElSixmwS5IkSS1mwC5JkiS1mAG7JEmS1GKOwz5DNtxww5w/f/6oqyGt3q66amJ+k6nu+yOthjw+pNY755xzrs3Me93F3IB9hsyfP5+zz27LHc+l1VRUdwivgxNJHh/SGIiIK3qlG7BLmjs8aZb68/iQxpYBu6S5Y5ttRl0Dqb08PqSx5UWnkiRJUosZsEuSJEktZsAuSZIktZh92CXNHT0fWgEAACAASURBVPUoGJmjq4fURh4f0tiyhV2SJElqMQN2SZIkqcUM2CVJkqQWM2CXJEmSWsyAXZIkSWoxA3ZJkiSpxQzYJUmSpBYzYJckSZJazIBdkiRJajEDdkmSJKnF7jPqCkjSjPnjH0ddA6m9PD6ksWXALmnu2GSTUddAai+PD2ls2SVGkiRJajFb2CVpRPbef3+uWLp01NWYNZttvDHHf/zjo66GJI0dA3ZJc8dVV03Mj8Hf/1csXcrDDzts1NWYNVe8+92jrsLqbcyOD0kTDNglzR3z5k3MZ46uHlIbeXxIY8s+7JIkSVKLGbBLkiRJLWbALkmSJLWYAbskSZLUYgbskiRJUosZsEuSJEktZsAuSZIktZgBuyRJktRiBuySJElSi3mnU0nSrLj4oot42h57jLoas2azjTfm+I9/fNTVkDQHGLBLmju83Xqr3Z7Jww87bNTVmDVXvPvdo67CZB4f0tiyS4wkSZLUYgbskiRJUosZsEuSJEktZh92SXPHOedMzG+zzejqIbWRx4c0tgzYJc0d2247Me8FdtJkHh/S2LJLjCRJktRiBuySJElSixmwS5IkSS1mwC5JkiS1mAG7JEmS1GIG7JIkSVKLGbBLkiRJLWbALkmSJLWYAbskSZLUYt7pVNLc8bd/O+oaSO3l8SGNLQN2SXPHVVeNugZSe3l8SGPLLjGSJElSixmwS5IkSS1mwC5JkiS1mH3YJc0dp5wyMb/bbqOrh9RGHh/S2DJglzR37L77xHzm6OohtZHHhzS27BIjSZIktZgBuyRJktRiBuySJElSixmwS5IkSS1mwC5JkiS1mAG7JEmS1GIG7JIkSVKLGbBLkiRJLWbALkmSJLXYSAP2iNgyIv49In4eEddExE0RcV5EvDMi1uvKuzAiss/0lh5lrxERB0fERRGxLCJ+HxFHdJdb5X9URJwcEddHxC0RcUZE7LKqXrukVWDrrScmSZN5fEhj6z4j3v6rgNcD3wS+ANwJ7Az8B/CSiNg+M2/rWudg4NqutHN6lP0R4A3A14EjgMc0z58YEc/MzHs6GSNiC+BnwF3AB4EbgdcA34+I52bmaSv1KiXNjnN6fRVIAjw+pDE26oD9q8DhmXljlfaJiPgt8E7g1cBHu9Y5OTOXTFVoRGwFHAiclJl7VOmXA0cBewEnVKscDmwAbJOZ5zV5jwMuAI6JiEdnZq7A65MkSZJWyki7xGTm2V3BeseXm8e/77VeRNw/IqY62XgZEMCRXemfAm4F9q7KWg/YHVjcCdabut0MfBrYEthumpciSZIkrRJtvej0Yc3jn3ss+xWly8qyiPhZRDy3R57tgHuAs+rEzFwGnMfkAPxxwNrAmT3K+XlVniRJkjTrRt0l5l4iYk3gEEp/8rrbyg3AsZS+5tcDjwIOAr4dEa/KzEVV3k2AazPz9h6b+CPw1IhYKzPvaPJ20nvlBZjXp677AfsBbLrpptO/OEmr1rHHTszvt9/o6iG1kceHNLZaF7BTurFsD7wjMy/uJGZmd/cWIuKzwK+Bj0TEV5tuLADrAr2CdYBlVZ47mkf65K/z3ktmHks5iWDbbbe1j7s0aq997cS8AYk0mceHNLZa1SUmIg4DDgCOzczDp8ufmdcBn6BcMPrUatGtlG4uvaxT5akfe+XvzitJkiTNqtYE7BGxEHgX8N/A64ZYdUnzuGGVdhWwYUT0CsLnUbrL3FHl7aT3ygu9u8tIkiRJq1wrAvaIOBQ4FDgO2HfIIRQf2TzWF6j+kvLantS1nXWAJwBnV8nnU7rDPKVH2ds3j2f3WCZJkiStciMP2CPiEGAh8HnglfUNjao894mIB/RIfziwP3Ad5WLUji8DSbkotfYaSn/0L3QSmn7vpwALIuLxVdnrA/sCv6VrtBlJkiRptoz0otOIeD3wHuBK4DTg5RFRZ/lzZv4AWB+4PCJOBi5kYpSYfZtlL6vviJqZ50fEMcABEXES8B0m7nR6OpNHnwF4O/AM4NSI+AjwV0pwPw/Y1ZsmSZIkaVRGPUpMZ3zzTYHP9Vh+OvAD4Dbga8CTgX+iBOnXUoL8D2Zmrxbwgyj92/cDdm3yHw0c0t2Kn5mXRsQOwPuBtwFrAecCz8nM01bi9UmSJEkrZaQBe2buA+wzQL7bKa3pw5R9N3BEMw2S/0LgBcNsQ5IkSVrVRt6HXZIkSVJ/BuySJElSi426D7skzZznP3/UNZDay+NDGlsG7JLmjlNOGXUNpPby+JDGll1iJEmSpBYzYJckSZJazIBdkiRJajH7sEuaOxYu7D0vyeNDGmMG7JLmjve8Z2LegESazONDGlt2iZEkSZJazIBdkiRJajEDdkmSJKnFDNglSZKkFjNglyRJklrMgF2SJElqMQN2SZIkqcUM2CVJkqQWM2CXJEmSWsw7nUqaO17zmlHXQGovjw9pbBmwS5o7jj121DWQ2svjQxpbdomRJEmSWsyAXZIkSWoxA3ZJkiSpxezDLmnu2G+/iXn760qTeXxIY8uAXdLc8alPTcwbkEiTeXxIY8suMZIkSVKLGbBLkiRJLWbALkmSJLWYAbskSZLUYgbskiRJUosZsEuSJEktZsAuSZIktZgBuyRJktRiBuySJElSi3mnU0lzx6GHjroGUnt5fEhjy4Bd0tyxcOGoayC1l8eHNLbsEiNJkiS1mAG7JEmS1GIG7JIkSVKL2Ydd0tyx224T86ecMrp6SG3k8SGNLQN2SXPHt7416hpI7eXxIY0tu8RIkiRJLWbALkmSJLWYAbskSZLUYgbskiRJUosZsEuSJEktZsAuSZIktZgBuyRJktRiBuySJElSixmwS5IkSS3mnU4lzR2f/OSoayC1l8eHNLYM2CXNHfvtN+oaSO3l8SGNLbvESJIkSS1mwC5JkiS1mAG7JEmS1GL2YZc0d2yzzcT8OeeMrh5SG3l8SGPLgF3S3HHuuaOugdReHh/S2LJLjCRJktRiIw3YI2LLiPj3iPh5RFwTETdFxHkR8c6IWK9H/kdFxMkRcX1E3BIRZ0TELn3KXiMiDo6IiyJiWUT8PiKO6FXusGVLkiRJs2XULeyvAg4Gfgf8O/BW4GLgP4CfRcR9OxkjYgvgZ8BTgA82edcHvh8Rz+xR9keA/wR+AxwInAi8ATglIia97hUoW5IkSZoVo+7D/lXg8My8sUr7RET8Fngn8Grgo0364cAGwDaZeR5ARBwHXAAcExGPzsxs0reiBOknZeYenYIj4nLgKGAv4IRqmwOXLUmSJM2mkbawZ+bZXcF6x5ebx78HaLqx7A4s7gTUzfo3A58GtgS2q9Z/GRDAkV3lfgq4Fdi7k7ACZUuSJEmzZtRdYvp5WPP45+bxccDawJk98v68eayD6u2Ae4Cz6oyZuQw4ryvvsGVLkiRJs6Z1AXtErAkcAtzFRLeVTZrHP/ZYpZM2r0rbBLg2M2/vk3/DiFhrBcuWJEmSZk3rAnZKN5btgUMy8+Imbd3msVcAvqwrT2e+V95e+Ycte7mI2C8izo6Is6+55po+m5MkSZJWXKsC9og4DDgAODYzD68W3do8rt1jtXW68nTme+XtlX/YspfLzGMzc9vM3HajjTbqszlJkiRpxY16lJjlImIh8C7gv4HXdS2+qnns1TWlk1Z3abkKeGxErN2jW8w8SneZO1awbElt9c1vjroGUnt5fEhjqxUBe0QcChwKHAfs22MIxfMpXVae0mP17ZvHs6u0XwLPAp4EnFFtZx3gCcCPV6JsSW21226jroHUXh4f0tgaeZeYiDgEWAh8HnhlZt7TnacZYvEUYEFEPL5ad31gX+C3TB4R5stAAgd1FfUaSn/0L6xE2ZIkSdKsGWkLe0S8HngPcCVwGvDyiKiz/Dkzf9DMvx14BnBqRHwE+CslAJ8H7Fq3ymfm+RFxDHBARJwEfAd4DOVOp6cz+aZJQ5UtSZIkzaZRd4npjG++KfC5HstPB34AkJmXRsQOwPuBtwFrAecCz8nM03qsexCwBNgP2BW4FjiaMvrMpFb8FShbkiRJmhUjDdgzcx9gnyHyXwi8YMC8dwNHNNOMli2ppTbZZGL+qqv655NWRx4f0tgadQu7JM2cq68edQ2k9vL4kMbWyC86lSRJktSfAbskSZLUYgbskiRJUosZsEuSJEktZsAuSZIktZgBuyRJktRiBuySJElSixmwS5IkSS1mwC5JkiS1mHc6lTR3nH32qGsgtZfHhzS2DNglzR3bbDPqGkjt5fEhjS27xEiSJEktZsAuSZIktZgBuyRJktRi9mGXNHdETMxnjq4eUht5fEhjyxZ2SZIkqcUM2CVJkqQWM2CXJEmSWsyAXZIkSWoxA3ZJkiSpxQzYJUmSpBYzYJckSZJazIBdkiRJajEDdkmSJKnFDNglSZKkFrvPqCsgSTPmj38cdQ2k9vL4kMaWAbukuWOTTUZdA6m9PD6ksWWXGEmSJKnFDNglSZKkFrNLjKS546qrJub9+1+azONDGlsG7JLmjnnzJuYzR1cPqY08PqSxZZcYSZIkqcUM2CVJkqQWM2CXJEmSWsyAXZIkSWoxA3ZJkiSpxYYK2CPiwRHxmK60zSPi6Ij4QkQ8e2arJ0mSJK3ehh3W8b+ALYEnAUTE+sAZQGdA15dGxC6Z+eOZq6IkSZK0+hq2S8xTgO9Wz19KCdaf1zxeCPzrzFRNkiRJ0rAB+0OAK6vnzwXOzszvZeafgEXAE2eobpIkSdJqb9iA/U7gvtXznYDTq+c3AA9e2UpJkiRJKoYN2C8B9ohid+BBwP+rlj8c+MtMVU6ShpI5MUmazONDGlvDXnR6DKXby/XAusBlTA7Ynw6cPyM1kyRJkjRcwJ6Zx0XEPcALgRuB92XmnVCGfAQeAHxsxmspSZIkraaGbWEnM48Hju+Rfh2wzUxUSpIkSVIxVMAeEZcBB2XmN/ssfz5wVGY+YiYqJ2n1svf++3PF0qUrvP6W11+/fP6SBz5wJqq0Sl186aU8fNSV0OrjnHMm5rexfU0aJ8O2sM8H1p9i+XrAZitcG0mrtSuWLuXhhx22wut/Zqutls+//IILZqJKq9Sv9txz1FXQ6mTbbSfmvfBUGivDjhIznYcAt85wmZIkSdJqa9oW9oh4OrCgSnpRRPxdj6wPAvYCzpuZqkmSJEkapEvMzsChzXwCL2qmXi4FDp6BekmSJElisID9SMrY60EZd/0g4BtdeRK4OTO9aZIkSZI0g6YN2DPzRsqY60TEzsCFmbniwzhIkiRJGtiwN046fVVVRJKkueTiiy7iaXvsMepqLHdGNb8q6rXZxhtz/Mc/PuPlSlqBGydFxKbAa4FHAg+mdJWpZWY+YwbqJknS2Lo9c6WGKZ1xJ520fHZV1OuKd797xsuUVAx746TnAl8H1gJuAuyzLkmSJK1Cw7awHw5cC/xTZp69CuojSZIkqTJswP5o4F0G65La6PqNNhp1FaTW8viQxtewAfs1wB0ztfGIeDuwNbANsDlwRWbO75N3IRPjwXd7a2Z+uCv/GsAbKf3t51Pq/hXgkMy8pUf5jwI+AOxE6fJzLnBoZv5w2NclaTRev3jxqKsgtZbHhzS+hg3YPw/sARw1Q9t/H6Uf/LnABgOuczClW07tnB75PgK8gdLn/gjgMc3zJ0bEMzPznk7GiNgC+BlwF/BByjCWrwG+HxHPzczTBn5FkiRJ0gwaNmBfBOwcEd8A/gu4HLi7O1NmXjlgeVtk5mUAEfFrYP0B1jk5M5dMlSEitgIOBE7KzD2q9MspJxt7ASdUqxxOOWHYJjPPa/IeB1wAHBMRj87MHPA1SZIkSTNmjSHzXwQ8CdgN+AFwKSVo754G0gnWhxUR94+IqU42XkYZbvLIrvRPAbcCe1dlrQfsDizuBOtN3W4GPg1sCWy3IvWUJEmSVtawLez/Doy6pflXwP2AuyPiLOCwzPxuV57tgHuAs+rEzFwWEecxOQB/HLA2cGaPbf28Ku+sHssltcjWP/rR8vlzd955hDWR2sfjQxpfw97pdOEqqscgbgCOpfQ1vx54FHAQ8O2IeFVmLqrybgJcm5m39yjnj8BTI2KtzLyjydtJ75UXYN4M1F/SKvaWAw5YPv/yCy4YYU2k9vH4kMbX0Hc6HZXM7O7eQkR8Fvg18JGI+GrTjQVgXaBXsA6wrMpzR/NIn/x13nuJiP2A/QA23XTT6V6CJEmSNLRh73T69EHyZeaPV6w6w8nM6yLiE8BC4KnAqc2iW4GN+6y2TpWnflx7gLzd2z+W0urPtttuO+quQpIkSZqDhm1hX8xgfdjXHL4qK2xJ87hhlXYV8NiIWLtHt5h5lO4yd1R5O+ndOmm9ustIkiRJq9ywAfsr+5SxBbAPJXj+5MpVaWiPbB7/XKX9EngWZUSbMzqJEbEO8ASg/gfgfEp3mKf0KHv75tE7u0qSJGkkhr3o9HP9lkXEhyg3QJpxzRCO62XmjV3pDwf2B66jXIza8WXgHZSLUs+o0l9D6Y/+hU5CZt4cEacAL4qIx2fm/zZlrw/sC/wWR4iRJEnSiMzYRaeZeX1EfBr4V6BvYF+LiFcAmzVPNwLWioh3Nc+vyMzPN/PrA5dHxMnAhUyMErNvs+xlmXlbVZfzI+IY4ICIOAn4DhN3Oj2dyTdNAng78Azg1Ij4CPBXSnA/D9jVmyZJkiRpVGZ6lJjrgUcMkf/VwE5daYc1j6cDnYD9NuBrwJOBf6IE6dcCpwEfzMxeLeAHUbro7Afs2uQ/GjgkM++pM2bmpRGxA/B+4G3AWpR/C56TmacN8XokSZKkGTVjAXvTP/wVwJ8GXSczFwyY73ZKa/rAMvNu4IhmGiT/hcALhtmGJEmStKoNO6zjZ/ssehDlos2NgLeubKUkSZIkFcO2sO/TJ/0vwCXAwZnZ3T9ckmbF5Y997KirILWWx4c0voYdJWaNVVURSVpZ7zzxxFFXQWotjw9pfBmAS5IkSS22QhedRsT9gWcyMSLMZcAPMvOmmaqYJEmSpBUI2CNiX8rIK+sD0SQncHNEvCkzPzOD9ZMkSZJWa8OOErM7cCylRf0Q4NfNoq2AA4FjI2JpZp4yo7WUpAHs8pWvLJ//4UteMsKaSO3j8SGNr2Fb2P+VcqfRJ2fmzVX6/4uI/wZ+DvwbYMAuadbt+573LJ83IJEm8/iQxtewF50+HljUFawD0PRf/1yTR5IkSdIMWJFRYmKKZbmiFZEkSZJ0b8MG7P8L/N+IWK97QUSsT7mx0v/OQL0kSZIkMXwf9g8DJwHnRsRRwG+a9M5Fp38HvGjmqidJkiSt3oa90+nJEXEA8AHgaCa6wARwC3BAZn5jZqsoSZIkrb6GHoc9Mz8WEScA/whsTgnWf0e5cdKNM1w/SZIkabW2Qnc6zcwbgBNnuC6SJEmSukx70WlErBkR74+I102Tb/+IeF9ETDWKjCRJkqQhDDJKzN7AW4FfTpPvLMpNk162spWSJEmSVAzSJeYlwGmZec5UmTLznIj4PiVgP2EmKidJwzh3p51GXQWptTw+pPE1SMC+DXDEgOX9CHjTildHklbchz/2sVFXQWotjw9pfA3SJeZBwNIBy7umyS9JkiRpBgwSsN8EbDhgeQ8Gbl7x6kiSJEmqDRKwXwA8a8Dy/rHJL0mSJGkGDNKH/STgiIh4wVR3MY2I3SkBu33YJY3EHsccs3z+a69//QhrIrWPx4c0vgYJ2D8J7A98JSI+DHwqM5d0FkbEfGBf4C3AJU1+SZp1e1QX1RmQSJN5fEjja9qAPTNvi4hdgW8BbwfeFhE3AX8F7gfcHwjgYuD5mblsFdZXkiRJWq0M0oedzLwUeALwRuAnwF3AQ4G7gTOa9K0z83erqJ6SJEnSammQLjEANC3nRzeTJEmSpFkwUAu7JEmSpNEwYJckSZJazIBdkiRJajEDdkmSJKnFDNglSZKkFjNglyRJklps4GEdJantfrjnnqOugtRaHh/S+DJglzRnfPo97xl1FaTW8viQxpddYiRJkqQWM2CXJEmSWsyAXZIkSWox+7BLmjP2PfTQ5fP215Um8/iQxpcBu6Q5Y5evfnX5vAGJNJnHhzS+7BIjSZIktZgBuyRJktRiBuySJElSixmwS5IkSS1mwC5JkiS1mAG7JEmS1GIG7JIkSVKLGbBLkiRJLWbALkmSJLWYdzqVNGd87V/+ZdRVkFrL40MaXwbskuaMr73+9aOugtRaHh/S+LJLjCRJktRiBuySJElSixmwS5IkSS1mH3ZJc8ZbqovqPvyxj42wJlL7eHxI48uAXdKcsfXpp4+6ClJreXxI48suMZIkSVKLGbBLkiRJLTbSgD0i3h4RJ0bEZRGREbFkmvyPioiTI+L6iLglIs6IiF365F0jIg6OiIsiYllE/D4ijoiI9Va2bEmSJGm2jLqF/X3ALsDvgOunyhgRWwA/A54CfBB4K7A+8P2IeGaPVT4C/CfwG+BA4ETgDcApETHpda9A2ZIkSdKsGPVFp1tk5mUAEfFrSpDcz+HABsA2mXles85xwAXAMRHx6MzMJn0rSpB+Umbu0SkgIi4HjgL2Ak5YkbIlSZKk2TTSFvZOsD6dphvL7sDiTkDdrH8z8GlgS2C7apWXAQEc2VXUp4Bbgb1XomxJkiRp1oy6S8ygHgesDZzZY9nPm8c6qN4OuAc4q86YmcuA87ryDlu2JEmSNGvGJWDfpHn8Y49lnbR5Xfmvzczb++TfMCLWWsGyJUmSpFkzLgH7us1jrwB8WVeeznyvvL3yD1v2chGxX0ScHRFnX3PNNX02J0mSJK24UV90Oqhbm8e1eyxbpytPZ37jPmV15x+27OUy81jgWIBtt93Wi1KlEfv0oYeOugpSa3l8SONrXAL2q5rHXl1TOml1l5argMdGxNo9usXMo3SXuWMFy5bUUj98yUtGXQWptTw+pPE1Ll1izqd0WXlKj2XbN49nV2m/pLy2J9UZI2Id4AldeYctW5IkSZo1YxGwN0MsngIsiIjHd9IjYn1gX+C3TB4R5stAAgd1FfUaSn/0L6xE2ZIkSdKsGWmXmIh4BbBZ83QjYK2IeFfz/IrM/HyV/e3AM4BTI+IjwF8pAfg8YNf6xkaZeX5EHAMcEBEnAd8BHkO50+npTL5p0lBlS5IkSbNp1H3YXw3s1JV2WPN4OrA8YM/MSyNiB+D9wNuAtYBzgedk5mk9yj4IWALsB+wKXAscDRySmffUGVegbEkt9N4Xv3j5/DtPPHGENZHax+NDGl8jDdgzc8GQ+S8EXjBg3ruBI5ppRsuW1E6b/+Y3o66C1FoeH9L4Gos+7JIkSdLqyoBdkiRJajEDdkmSJKnFDNglSZKkFjNglyRJklrMgF2SJElqMQN2SZIkqcUM2CVJkqQWM2CXJEmSWmykdzqVpJn04Y9+dNRVkFrL40MaXwbskuaMc3feedRVkFrL40MaX3aJkSRJklrMgF2SJElqMQN2SZIkqcXswy5pzjhmwYLl869fvHhk9ZDayONDGl8G7JLmjAdec82oqyC1lseHNL7sEiNJkiS1mAG7JEmS1GIG7JIkSVKLGbBLkiRJLWbALkmSJLWYAbskSZLUYgbskiRJUosZsEuSJEktZsAuSZIktZh3OpU0Z7zzK18ZdRWk1vL4kMaXAbukOePyrbYadRWk1vL4kMaXXWIkSZKkFjNglyRJklrMgF2SJElqMfuwS5ozTqj66L78ggtGWBOpfTw+pPFlC7skSZLUYgbskiRJUosZsEuSJEktZsAuSZIktZgBuyRJktRiBuySJElSixmwS5IkSS1mwC5JkiS1mAG7JEmS1GIG7JIkSVKL3WfUFZCkmfIvP/rRqKsgtZbHhzS+DNglzRk3bLzxqKsgtZbHhzS+7BIjSZIktZgBuyRJktRidomRNGdssHTp8nn//pcm8/iQxpcBu6Q542M777x8/uUXXDDCmkjt4/EhjS+7xEiSJEktZsAuSZIktZgBuyRJktRiBuySJElSixmwS5IkSS1mwC5JkiS1mAG7JEmS1GIG7JIkSVKLeeMkqcX23n9/rqjuTjjXXXzppTx81JWQJKllDNilFrti6VIefthho67GrPnVnnuOugqSJLWOAbukOcPbrUv9eXxI48s+7JIkSVKLjVXAHhHZZ7q5R95HRcTJEXF9RNwSEWdExC59yl0jIg6OiIsiYllE/D4ijoiI9Vb9q5IkSZL6G8cuMWcAx3al3Vk/iYgtgJ8BdwEfBG4EXgN8PyKem5mnda3/EeANwNeBI4DHNM+fGBHPzMx7ZvxVSJIkSQMYx4D9ssw8fpo8hwMbANtk5nkAEXEccAFwTEQ8OjOzSd8KOBA4KTP36BQQEZcDRwF7ASfM/MuQNNM2r/roXr7VViOsidQ+Hh/S+BrHgJ2IWAtYKzN7dYVZD9gdWNwJ1gEy8+aI+DTw78B2wFnNopcBARzZVdSngPcDe2PALo2F977kJcvnvcBOmszjQxpfY9WHvbEncCtwU0QsjYijI+IB1fLHAWsDZ/ZY9+fN43ZV2nbAPUwE8ABk5jLgvK68kiRJ0qwatxb2s4ATgUuB+wPPAw4AdoqIpzYt7ps0ef/YY/1O2rwqbRPg2sy8vU/+p0bEWpl5R/fCiNgP2A9g0003XYGXI0mSJE1trAL2zHxyV9JxEfEr4L3AG5vHdZtlvQLwZc3julXaun3ydue/V8CemcfSXAC77bbb5nT1lyRJkoY1jl1iun2IEkzv2jy/tXlcu0fedbrydOZ75e2XX5IkSZo1Yx+wZ+adwFXAhk3SVc3jvB7ZO2l1d5mrgA0jolfQPo/SXeZereuSJEnSbBj7gD0i1gEeBvy5STqf0sXlKT2yb988nl2l/ZKyH57Uo9wndOWVJEmSZtXYBOwR8eA+iw6j9MU/Bcrwjc38goh4fLX++sC+wG+ZPCLMl4EEDuoq9zWUvutfmIn6S5IkSStinC46fVdEbA/8CLgSWJ8ySszOwC+Ao6u8bweeAZwaLcVQ0wAAFqBJREFU8f/bu/douar6gOPfH0KIgIISgpJi8AEKFgVRfIEICrjQlmqiVYsVK6hRsKC29YW8tPXRiFUkLaIioF0CYqmrtFjEKILy8AUGE3klaqKmCBggBDDZ/WPvSc49OffeuclN5szM97PWXjOzz5599uw9597fnNmzT5wBrCAH4DOAl3cumgSQUropIj4LHBcRlwCXse5Kp9/BNdglSZLUQ/0UsM8H9gLeCOwIrCafLf8A8MmybjoAKaVbI+KF5AsfvReYAvwIeFlK6YqGuk8AFpOXaHw5cCf5A8CHUkprNtHrkSRpYCxauJADZ80av+CAmDl9OhfMm9frZmhI9E3AnlK6FLh0AuV/DhzZZdnVwNySJPWpu3faqddNkFprUx8fD6bErqefvkn30SZLTjqp103QEOmbgF2SxvOO+fN73QSptTw+pP7VNz86lSRJkoaRAbskSZLUYgbskiRJUos5h13SwHjWt7+99v6PDj64hy2R2sfjQ+pfBuySBsZ7jjtu7f3XL1jQw5ZI7ePxIfUvp8RIkiRJLWbALkmSJLWYAbskSZLUYgbskiRJUosZsEuSJEktZsAuSZIktZgBuyRJktRiBuySJElSixmwS5IkSS3mlU4lDYw79tqr102QWsvjQ+pfBuySBsYHLrqo102QWsvjQ+pfTomRJEmSWsyAXZIkSWoxA3ZJkiSpxZzDLmlgHHLhhWvvX/ma1/SwJVL7eHxI/cuAXdLAOObUU9feNyCRRvL4kPqXU2IkSZKkFjNglyRJklrMgF2SJElqMQN2SZIkqcUM2CVJkqQWM2CXJEmSWsyAXZIkSWoxA3ZJkiSpxQzYJUmSpBbzSqeSBsaPDjqo102QWsvjQ+pfBuySBsY/n3VWr5sgtZbHh9S/nBIjSZIktZgBuyRJktRiBuySJElSizmHXdLAmPXZz669/7V3vKOHLZHax+ND6l8G7JIGxqzKj+oMSKSRPD6k/uWUGEmSJKnFDNglSZKkFjNglyRJklrMgF2SJElqMQN2SZIkqcUM2CVJkqQWM2CXJEmSWsyAXZIkSWoxA3ZJkiSpxbzSqaSBceXs2b1ugtRaHh9S/zJglzQwzjn11F43QWotjw+pfxmwq68cNWcOS5Yv73UzNptFt97Krr1uhCRJ6ikDdvWVJcuXs+vpp/e6GZvNjX6FLUnS0DNglyRJmqBFCxdy4KxZvW7GZjNz+nQumDev180YWgbskgbGMSefvPa+83WlkTw+JteDKQ3VN75LTjqp100YagbskgbGIRdfvPa+AYk0kseH1L9ch12SJElqMQN2SZIkqcUM2CVJkqQWM2CXJEmSWsyAvYiILSLixIhYGBGrIuJXETE3IrbtddskSZI0vAzY1zkD+CRwM3A8cBHwTuAbEWE/SZIkqSdc1hGIiKeTg/RLUkqzKvl3AJ8GXgt8pUfNkyRJ0hDzzHH2OiCAT9XyPwesBI7a7C2SJEmS8Ax7x3OANcB11cyU0qqI+EnZ3kpHzZnDkuXLe92MzWbRrbeya68bIUnSkFm0cCEHzpo1fsEBMXP6dC6YN6/XzVgrUkq9bkPPRcRNwPSU0s4N2y4EXg1snVJ6qLbtLcBbysOnAos2dVv73DTgzl43Qj3h2A8vx354OfbDy7HfcDNTSjvVMz3Dnm0DPDjKtlWVMiMC9pTS2cDZm7BdAyUibkgpPbvX7dDm59gPL8d+eDn2w8uxn3zOYc9WAluPsm1qpYwkSZK0WRmwZ8uAaRHRFLTPAO6sT4eRJEmSNgcD9ux6cl/sX82MiKnAPsANvWjUAHL60PBy7IeXYz+8HPvh5dhPMn90CkTE3sBPga/X1mE/nrwO+xtSShf0qn2SJEkaXgbsRUR8BjgO+DpwGbAn+UqnVwOHpJTW9LB5kiRJGlIG7EVEPAI4gbxM427k5Yi+CnwopXRfD5smSZKkIWbALkmSJLWYPzrVhETEX0fEjyPigYj4XUScExHrLfA/Th3PjYgrIuLeiFgREf8TEfuMUnbriDgtIu6IiAcj4raI+GBEbNXFfj4eESki/IZkErR17CNiakQcGxGXRsTi0r7bI+LfI2LPjXnNwyAitoiIEyNiYUSsiohfRcTciNh2AnUcERHXRMT9EXFXRFwUEU8cpez2EfGZiFha9rcgIuZERGxs3ZqYto59RDwmIv42Ir5Z2vRARCyKiLMjwotdT4K2jv0oz72w/C//2URe48BJKZlMXSXgRCAB88lTh04D7gMWANt2WcfzyBejuq3Ud2K5fy+wd0P5/yj7/DxwTLlNwLnj7Gcf4OFS73297rt+T20ee+BpJf8q4CTgzcBHgLvIF0Q7uNf91+YE/Evpv0uAY4FPlmPnSmCLLp7/KmAN8GPg7cD7gN+Rl8vdpVZ2CnBdqf+TZX+XlP2fsjF1mwZn7IGXAX8ELgf+oRzTZ5Cvh3IPsFev+67fU1vHvmE/rwBWl7H/Wa/7radj1usGmPojkS8zfH856B5Ryf+zctC9v8t6rgNWADMqeTNK3jdrZY8odc+t5c8t+S8YZR+PIC/V+Z/kANOAfYDHHtgR2Kdhf3uRA/Ybet2HbU3A08s/3a/V8o8v/fz6cZ6/FbAUWAJsV8nfp/yTPbtW/u2l3uNr+V8jX0l65obWbRqosd8NeHLDPl9a6ri41/3Xz6nNY1/bvh3wS/JqfYsxYO99I0ztT+QznIm8xGV9223AzV3U8ZRSx+cbtn2+/AF5XCXvglJ+11rZXUv+WaPs50RygDkTA/ahGvuGun8IrOp1H7Y1AR8u/XlgLX9qOYYuG+f5nQDqpIZt3wL+AGxVyfteqXdqreyBpZ6/39C6TYMz9uPs9/fAwl73Xz+nfhl78rcAy4BHY8DuHHZ17Tnl9vsN234APC0ittvIOgLYr1Z+aUrpV9WC5fGySn1rRcRM4HTg1JTSknHao+70xdjXRcQWwOPJX9Oq2XPIH5auq2amlFYBP2H8fh5vXB8N7AFrx+NZwI9L/VXXlXZU99d13dogbR77RhGxPfAoPKY3VuvHPiL2Jy+1fUJKacU47RkKBuzq1i7ldmnDtqXkgGuXhm0TqQPyFIlq+aaynfIzGvLnAXeQ58lpcvTL2NfNIQfsX+qi7LDaBbgzpfRgw7alwLSImDLO8ztlm54P68bqMcAjm8qW/f+e9d8D3datiWvz2I/mg+TpGB7TG6fVYx8RWwKfI0+VvHCMdgyVLXvdAG1eEbEDeb35bn06pXQXsE153HSAdz41b9OwrWqidWwzStlO+RH7i4jXkX+sdEBK6Y/jtGXoDPLY10XEC8jz3W8E/nGctg2z8fq5U+ahMZ7PKHXUx3Wssp3y9fdAt3Vr4to89uuJiNnAu8k/RP3iWGU1rraP/d8BuwOvHOU5Q8mAffjsAJw8gfIXkFfbWFkebw08UCsztdyuZGzVOuqa6lg5StlO+bVlI+KxwKfIc6SvGacdw2ogx74uIvYD/os8deaIhq9htc5KYPoo27oZ24mM61hlO+Xr74Fu69bEtXnsR4iII4Avk3+T8ppUJjhrg7V27CPiKcCHgA+nlG4fow1DxykxQyaltDilFBNIt5anLiu3TV9bziD/cGRZw7aq8eqAkV+bLRulbKd8tezJwLbA5yLiKZ1E/iouyuOhXr93gMd+rYh4FvC/5B89HZxSGm1ajbJl5K+/m/6ZziB/bT7aWbbO8ztlm54P68bqbvIHvvXKlv3vyPrvgW7r1sS1eeyr219GXgJwAXCY85knRZvHfi75RNHXa//LtwSmlMePH6NtA8uAXd26vtw+v2Hbc4FFKaXxLlA0Vh3PIwd+P6yVn1EPtMvjXYAbKtkzyQH7tcAtlbQ/+eu2W4D/Hqd9atb2se9s25ccrN9LDtb90fH4rif/H9i/mhkRU8lLtK3Xzw3Ph9HHdQXwC4CU0hrgR8C+DYHC/qUd1f11Xbc2SJvHvtOWw4GvAwuBl6aU7h6nTepOm8d+Jvlv/AJG/i+fQZ4mcwt5fvvw6fUyNab+SMBO5K+trqV5Le4P1spPI1/QZvta/vXkg3mXSt4uJe+KWtmXM/Za3AdU8p4PzG5IC8if7mcDh/a6H/sxtX3sS/6+5B8v/RJ4Uq/7rF8SsDdjr8d8VCXv8WVct6nkbUU+21Zfj/mZ5PWYz6nV+w5GX4/5YeCJG1q3aXDGvuQfVv52/xTYsdf9NUipzWNPXjKy6X/58vL3fTbwwl73YU/GrdcNMPVPIv/gJwHfJl/t8lTy1S5/Xj1oS9lTStmja/kvIP/45DbyDyBPKPfvA57ZsM9vlHrOIV/t7pzy+Pwu2zwf12Ef6LEnn5G5s/wDOhk4qiF1dTXWYUzAZ1h3xcNjyB+KHi7HzhaVcueWci+uPf/VjLzi4XvJy+79lspFskrZKeSzaQ+X/RzDuisent7Qtq7rNg3O2APPJgfrq8rfifWO6V73Xb+nto79GO1dzJCvw97zBpj6KwFHk894rCJ/4v0CML2h3Ck0BG1l2/PJF1e4jzx94XLgWaPsbyr5Ig+LycHe7eTLz3d1wRQM2Ad+7IEXl/2NlXbrdf+1NZGvDPxuYFHp56XkZVHrH8Qa/3GXba8gr7+8kjxn9WIarlRZyu4AnEk+Q/cgcDN5veUYpXzXdZsGY+zL35oxj+le912/p7aO/RjtXcyQB+xROkKSJElSC/mjU0mSJKnFDNglSZKkFjNglyRJklrMgF2SJElqMQN2SZIkqcUM2CVJkqQWM2CXJEmSWsyAXZLGERGLI2K+7RhbROwWESkiTtmE+zil7GO3Wv4+EfGtiLi72oaImBYR50XEspI/f1O1TZI2lS173QBJ6oWIeBL5ctovAp5Avvreb4DrgXNTSt/uYfM2iYg4GtghpfSplrTli5WsPwIrgCXAdcD5KaWru6xrS+BrwFbkq+HeA9xYNs8F/hL4CPlqub+bhOZL0mZlwC5p6ETEs4HvAA8D5wELgEcCewB/BtwLVAP2p5Ivz93vjgZ2A3oesFd8mvwhaQtge+BPgVcBb42IrwBvSik9VCn/YeCj5A9YHU8q6d0ppTNr9R8KXJ5SOm0TtV+SNjkDdknD6GRgG2DflNJPqhsi4jjgcdW8lFI1ONTkuiqldHE1IyJOAL4AvJ581n1OZ1tK6Y/ks/FVnfG6q6H+x42Sv1Ei4lEppXsnu15JauIcdknDaHfg9/VgHSCltCaltKya1zR3vJMXEc+MiCsi4r6IWB4R/xwRW0bE1HJ/aUSsiojvRsSetToa52OPts8mEXFYRHw1Im6PiAci4p6I+GZEHFSvDzgImFn22UkvrpTZPSLOj4jfRMRDpQ2fiIhtG/Z7QERcXfb5u4g4E9huvPZ2I6X0APBG8hSWY6v9U++z0kffKZu/WHld50ZEAgJ4YyX/6EpdLy19dU8Zoxsj4m0Nr7Uz1vtGxOUR8QfWTbnput86bYqI7SNiXnm/rCr9+NyG/UZEHBsR15b3130RcVNEnFYrt3VEvD8iFpT67omIb0TEvhPpd0nt5Rl2ScPoNuCpEfGqlNIlG1HPnwD/C3wVuBg4DHg3sBp4OnmazUeBacB7gP+IiD1TSms2pvE1RwOPJU/t+TUwAzgG+FZEHJxSuqqUOwH4p9KWEyvP/zlAROwHXEme//1vwFLgmcA7gRdGxEEppYdL2ecCV5CnDn2sPOe1pQ2TIqX0UEScT/425PDSpiYfAa4G3g+cDXRe702ljeeXvLNL/jXlNbwF+FfgB6WO+8nTZ+ZFxJNTSn9X288TyP1zEXm+/Halnq77reJy4P+A04AdgXcBl0XEbrWz9ucDfwVcW9p4D/A0YDbwobL/rYD/AV5Qyp9Jnlp0LHB1RLwopXTDKH0nqV+klEwmk2moEvB84CHyvPRfkKdfzAH2HKX8YmB+Q14CXl3L/yGwBrgUiEr+O0v5wyt5p5S83Sawz3retg3P3Rm4E7islj8fWDzKa/wpsBB4VC3/laWNR1fyrin9t0clbwr5x6IJOKWLMTi6lJ09RplXlTJzx+oz4MX1Nla2JfKPiKt5jwdWAV9pKP8v5A9cT24Y62M2st/OLXln1cq+uuS/tZL3mpJ3PrBFrfwWlfsn1t9XJf/RwC/r7xeTydSfySkxkoZOSun7wH7Al8hnI98EnAXcHBFXlRVkurE0pXRRLe975GkYn0kpVX+o2jnzu/uGt3x9KaX7O/cjYruI2JEccF4LrDfNoklE7A08A/gKsHXkpRCnRcQ08uu5n/ztARExnfyB59KU0i8q7XgIOGNyXtVaK8rtoye53tnA1sDnq6+1vN5vkKeLvqT2nLsYuarNhPqtpt5PV5bb6nvjr8rte1LtG5na46PIHxh+WNv/FPK3PwdExCObu0FSv3BKjKShlFK6iXyWl4iYSZ7ffQxwIHBpROyXRq5O0uSOhry7R9nWyd9xgxo8ioh4Mnm6xOHADrXN3a5s05lbf2pJTXYut50PMwsbytzc5f661QnUV4xZauI6r/eKMcrsXHt8W0pp9Sj1dNNvVbdXH6SUfh8RMPK9sTvwm5TSeMtQ7kmeevV/Y5SZBvxqnHoktZgBu6Shl1JaApxX5kxfBbwQ2J98lnQs9QCum21R3fUYzx/373NEbAd8F9iWvFTjTeR55WuA9wGHjFdHrU1zyfOhm9xdK9vU9mjI2xjPKLeLJrneTjv/mrz2fpPba49XjlFPN/22VkPgX6+vc7+bD1xBHvd3jVFmrGBeUh8wYJekIqWUIuJacsA+YzPssrPc4GPJ86QBiIip5HnWt47z/JcAuwB/k1KqT9f4cEP50QLAW8rt6pTSWGedIf9gF9adXa5qytsgETEFeAP5g8/lk1Vv0Xm9d3bxerupp5t+m6hFwJERsfM4Z9lvAXYCrqxPnZE0OJzDLmnoRMShka+OWc9/JOvmHE/29I4mnTngL63ln0h3f587Z2pHnNmOiMNonr9+H/CYKPMvKn4M/Ax4W9P8/cjLVD4WIKW0nLyyypERsUelzBRGrj6zwco4nEuefvNv5RuQyXQh+cJLpzbN7y7LLm7dRT1d99sG+HK5/XhEjHgv1MbvPPJa841n2COiaUqOpD7jGXZJw+gMYMeI+E/ydIKVwK7kC/XsAZxX5rhvaleQ54KfVn4segdwAPA88iov4/ke8FtgblmX/NfAPuQz0zcBe9fK/wB4BXBmRFxDDvivTCktj4g3kH/8eGNEfIF89ddtgKeQV2t5HzmIhhwczicvG/hZ1i3ruCH/Uw4s3ygEI690uhNwAXk5ykmVUvp1RMwBzgF+XqZCLSn73Bv4C2AvKt96jFJPmmC/TaSNF0XEV8nTdnYv79W7ye/Pw8n9BHlVm0OBT0TEIaUtK8jLUL6EvBrOwRPdv6R2MWCXNIzeBRxJDo5nkX+s2bkYzsfYgABrQ6SUVkfEkcCngePJSyV+k/wD2Ku7eP49EXE48PHy/C3Jy0oeAbyZ9QP2T5HPWs8G3kY+i38wsDyl9JNyoZ33AX9ett9LDlrPBb5V2e/3I+JQ8hrz7yUHiBcB88gfFCbineV2dWV/l5A/NF0zwbq6llL6YkT8grw+/lvJ74E7yVNRTiJ/EOqmnq77bQO8nvybijeT111fTf5Qt3ZlopTSwxHxcuDt5A9qnR+/LiMvs/mljdi/pJaIkauOSZIkSWoT57BLkiRJLWbALkmSJLWYAbskSZLUYgbskiRJUosZsEuSJEktZsAuSZIktZgBuyRJktRiBuySJElSixmwS5IkSS32/9gji3qwLNd7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 756x540 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# observed difference in conversion rates\n",
    "p_diffs_obs = convert_prob_treatment - convert_prob_control\n",
    "print(f\"Observed difference in conversion rates: {p_diffs_obs}\\n\\n\")\n",
    "\n",
    "# plot histogram\n",
    "plt.figure(figsize=(10.5,7.5))\n",
    "plt.hist(p_diffs_sim, color='c', edgecolor='k', alpha=0.65)\n",
    "plt.title(\"Sampling Distribution of 10,000 \\nSimulated Differences in Conversion Rates\", fontsize=18, fontweight='bold')\n",
    "plt.axvline(-abs(p_diffs_obs), color='r', linestyle='dashed', linewidth=3)\n",
    "plt.axvline(abs(p_diffs_obs), color='r', linestyle='dashed', linewidth=3)\n",
    "plt.xticks(fontsize=18)\n",
    "plt.yticks(fontsize=18)\n",
    "plt.xlabel(\"Simulated Difference\", fontsize=18)\n",
    "plt.ylabel(\"Counts\", fontsize=18)\n",
    "plt.tight_layout()\n",
    "plt.savefig('histogram.jpg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Proportion of samples where the difference is greater than the observed differece: 0.187\n"
     ]
    }
   ],
   "source": [
    "# proportion of simulated samples where the difference is greater than the observed \n",
    "# difference on both sides\n",
    "p_diffs_extreme = [abs(x) for x in p_diffs_sim ] > abs(p_diffs_obs)\n",
    "print(f\"Proportion of samples where the difference is greater than the observed differece: {np.mean(p_diffs_extreme)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Z score: 1.3109241984234394\n",
      "P value: 0.18988337448195103\n"
     ]
    }
   ],
   "source": [
    "# use Python's built in modules to obtain the statistics\n",
    "convert_old = convert_prob_control * n_old\n",
    "convert_new = convert_prob_treatment * n_new\n",
    "z_score, p_value = sm.stats.proportions_ztest([convert_old, convert_new], [n_old, n_new],alternative='two-sided')\n",
    "print(f\"Z score: {z_score}\")\n",
    "print(f\"P value: {p_value}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.4 Inference\n",
    "\n",
    "Since the p-value is greater than 0.05, we reject the null hypothesis. In other words, 18.9% of the differences are as extreme as the observed difference. Therefore, at a type 1 error rate (alpha) of 5%, we can conclude that the new page and old page are not statistically different in converting users to enroll in the free program"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Regression Approach"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Logistic regression is performed to confirm the results obtained previously"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.1 Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\knaag\\miniconda3\\envs\\ml_env\\lib\\site-packages\\pandas\\core\\frame.py:3641: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self[k1] = value[k2]\n",
      "C:\\Users\\knaag\\miniconda3\\envs\\ml_env\\lib\\site-packages\\pandas\\core\\frame.py:4913: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  errors=errors,\n",
      "C:\\Users\\knaag\\miniconda3\\envs\\ml_env\\lib\\site-packages\\ipykernel_launcher.py:9: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  if __name__ == '__main__':\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>intercept</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   ab_page  intercept  \n",
       "0        0          1  \n",
       "1        0          1  \n",
       "2        1          1  \n",
       "3        1          1  \n",
       "4        0          1  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create dummy variables from the group column\n",
    "# column ab_page should be 1 when an individual receives the treatment, 0 otherwise\n",
    "df2[['control','ab_page']] = pd.get_dummies(df2['group'])\n",
    "\n",
    "# drop the control column\n",
    "df2.drop('control', axis=1, inplace=True)\n",
    "\n",
    "# create intercept column\n",
    "df2['intercept'] = 1\n",
    "\n",
    "# check to make sure the columns are added and to compare with old df2\n",
    "df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366118\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td>Model:</td>              <td>Logit</td>      <td>Pseudo R-squared:</td>    <td>0.000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dependent Variable:</td>     <td>converted</td>          <td>AIC:</td>        <td>212780.3502</td>\n",
       "</tr>\n",
       "<tr>\n",
       "         <td>Date:</td>        <td>2022-06-01 21:35</td>       <td>BIC:</td>        <td>212801.5095</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <td>No. Observations:</td>       <td>290584</td>       <td>Log-Likelihood:</td>  <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "       <td>Df Model:</td>              <td>1</td>            <td>LL-Null:</td>      <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>Df Residuals:</td>         <td>290582</td>        <td>LLR p-value:</td>      <td>0.18988</td>  \n",
       "</tr>\n",
       "<tr>\n",
       "      <td>Converged:</td>           <td>1.0000</td>           <td>Scale:</td>         <td>1.0000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "    <td>No. Iterations:</td>        <td>6.0000</td>              <td></td>               <td></td>      \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>       <th>Coef.</th>  <th>Std.Err.</th>     <th>z</th>      <th>P>|z|</th> <th>[0.025</th>  <th>0.975]</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>-1.9888</td>  <td>0.0081</td>  <td>-246.6690</td> <td>0.0000</td> <td>-2.0046</td> <td>-1.9730</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>-0.0150</td>  <td>0.0114</td>   <td>-1.3109</td>  <td>0.1899</td> <td>-0.0374</td> <td>0.0074</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary2.Summary'>\n",
       "\"\"\"\n",
       "                          Results: Logit\n",
       "==================================================================\n",
       "Model:              Logit            Pseudo R-squared: 0.000      \n",
       "Dependent Variable: converted        AIC:              212780.3502\n",
       "Date:               2022-06-01 21:35 BIC:              212801.5095\n",
       "No. Observations:   290584           Log-Likelihood:   -1.0639e+05\n",
       "Df Model:           1                LL-Null:          -1.0639e+05\n",
       "Df Residuals:       290582           LLR p-value:      0.18988    \n",
       "Converged:          1.0000           Scale:            1.0000     \n",
       "No. Iterations:     6.0000                                        \n",
       "-------------------------------------------------------------------\n",
       "              Coef.   Std.Err.      z      P>|z|    [0.025   0.975]\n",
       "-------------------------------------------------------------------\n",
       "intercept    -1.9888    0.0081  -246.6690  0.0000  -2.0046  -1.9730\n",
       "ab_page      -0.0150    0.0114    -1.3109  0.1899  -0.0374   0.0074\n",
       "==================================================================\n",
       "\n",
       "\"\"\""
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# instantiate regression model on the two columns - converted and ab_page\n",
    "log_mod = sm.Logit(df2['converted'], df2[['intercept', 'ab_page']])\n",
    "\n",
    "# fit the model\n",
    "results = log_mod.fit()\n",
    "results.summary2()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The new page is 98.51% as likely to convert users as the old page. In other words, the new page is 1.49% less likely to convert users as the old page\n"
     ]
    }
   ],
   "source": [
    "# likelihood\n",
    "lik = np.exp(-0.015)\n",
    "\n",
    "print(f\"The new page is {round(lik*100, 2)}% as likely to convert users as the old page. In \\\n",
    "other words, the new page is {round((1-lik)*100, 2)}% less likely to convert users as the old page\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.2 Inference\n",
    "\n",
    "* All the logistic regression model is doing is check whether there is a statistically significant difference in the conversion rate between the pages. (2-tail). (It calculates the observed difference internally)\n",
    "\n",
    "* Since the p value is greater than 0.05, we fail to reject the null and can conclude that there is no statistically significant difference in the conversion rates between the two pages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Influence of Other factors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we test the effect that the country that a user lives in, has on the conversion rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.1 Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   user_id                   timestamp      group landing_page  converted  \\\n",
      "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
      "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
      "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
      "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
      "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
      "\n",
      "   ab_page  intercept  CA  UK  US  US_ab_page  UK_ab_page  CA_ab_page  \n",
      "0        0          1   0   0   1           0           0           0  \n",
      "1        0          1   0   0   1           0           0           0  \n",
      "2        1          1   0   0   1           1           0           0  \n",
      "3        1          1   0   0   1           1           0           0  \n",
      "4        0          1   0   0   1           0           0           0  \n"
     ]
    }
   ],
   "source": [
    "# read in countries dataset\n",
    "df3 = pd.read_csv('data/part1_countries.csv')\n",
    "df3.head()\n",
    "\n",
    "# create dummy columns for country\n",
    "df3[['CA','UK', 'US']] = pd.get_dummies(df3['country'])\n",
    "# drop the country column since this is not necessary\n",
    "df3.drop('country', axis=1, inplace=True)\n",
    "df3.head()\n",
    "\n",
    "# merge the countries dataframe with the current users dataframe\n",
    "new_df = df2.merge(df3, on='user_id')\n",
    "new_df.head()\n",
    "\n",
    "# combine conversion and country\n",
    "new_df['US_ab_page'] = new_df['US'] * new_df['ab_page']\n",
    "new_df['UK_ab_page'] = new_df['UK'] * new_df['ab_page']\n",
    "new_df['CA_ab_page'] = new_df['CA'] * new_df['ab_page']\n",
    "print(new_df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366109\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td>Model:</td>              <td>Logit</td>      <td>Pseudo R-squared:</td>    <td>0.000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dependent Variable:</td>     <td>converted</td>          <td>AIC:</td>        <td>212778.9383</td>\n",
       "</tr>\n",
       "<tr>\n",
       "         <td>Date:</td>        <td>2022-05-20 00:22</td>       <td>BIC:</td>        <td>212821.2568</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <td>No. Observations:</td>       <td>290584</td>       <td>Log-Likelihood:</td>  <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "       <td>Df Model:</td>              <td>3</td>            <td>LL-Null:</td>      <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>Df Residuals:</td>         <td>290580</td>        <td>LLR p-value:</td>     <td>0.067853</td>  \n",
       "</tr>\n",
       "<tr>\n",
       "      <td>Converged:</td>           <td>1.0000</td>           <td>Scale:</td>         <td>1.0000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "    <td>No. Iterations:</td>        <td>6.0000</td>              <td></td>               <td></td>      \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "       <td></td>       <th>Coef.</th>  <th>Std.Err.</th>     <th>z</th>      <th>P>|z|</th> <th>[0.025</th>  <th>0.975]</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th>  <td>-1.9888</td>  <td>0.0081</td>  <td>-246.6690</td> <td>0.0000</td> <td>-2.0046</td> <td>-1.9730</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>US_ab_page</th> <td>-0.0183</td>  <td>0.0126</td>   <td>-1.4486</td>  <td>0.1475</td> <td>-0.0430</td> <td>0.0064</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK_ab_page</th> <td>0.0074</td>   <td>0.0180</td>   <td>0.4098</td>   <td>0.6819</td> <td>-0.0279</td> <td>0.0427</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CA_ab_page</th> <td>-0.0827</td>  <td>0.0380</td>   <td>-2.1763</td>  <td>0.0295</td> <td>-0.1571</td> <td>-0.0082</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary2.Summary'>\n",
       "\"\"\"\n",
       "                          Results: Logit\n",
       "==================================================================\n",
       "Model:              Logit            Pseudo R-squared: 0.000      \n",
       "Dependent Variable: converted        AIC:              212778.9383\n",
       "Date:               2022-05-20 00:22 BIC:              212821.2568\n",
       "No. Observations:   290584           Log-Likelihood:   -1.0639e+05\n",
       "Df Model:           3                LL-Null:          -1.0639e+05\n",
       "Df Residuals:       290580           LLR p-value:      0.067853   \n",
       "Converged:          1.0000           Scale:            1.0000     \n",
       "No. Iterations:     6.0000                                        \n",
       "-------------------------------------------------------------------\n",
       "              Coef.   Std.Err.      z      P>|z|    [0.025   0.975]\n",
       "-------------------------------------------------------------------\n",
       "intercept    -1.9888    0.0081  -246.6690  0.0000  -2.0046  -1.9730\n",
       "US_ab_page   -0.0183    0.0126    -1.4486  0.1475  -0.0430   0.0064\n",
       "UK_ab_page    0.0074    0.0180     0.4098  0.6819  -0.0279   0.0427\n",
       "CA_ab_page   -0.0827    0.0380    -2.1763  0.0295  -0.1571  -0.0082\n",
       "==================================================================\n",
       "\n",
       "\"\"\""
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fit the regression model\n",
    "log_mod2 = sm.Logit(new_df['converted'], new_df[['intercept', 'US_ab_page', 'UK_ab_page', 'CA_ab_page']])\n",
    "results2 = log_mod2.fit()\n",
    "results2.summary2()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "intercept     0.136863\n",
       "US_ab_page    0.981901\n",
       "UK_ab_page    1.007417\n",
       "CA_ab_page    0.920649\n",
       "dtype: float64"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# likelihood\n",
    "np.exp(results2.params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.2 Inference\n",
    "    \n",
    "Taking into account the country a particular user lives in, the p-value for Canada was 0.0295, thereby suggesting that for residents of Canada, there is a statistically significant difference in conversion rates between the old and new pages. The new page is 7.94% less likely to convert users compared to the old page. For US and UK residents, there is no statistically signifcant difference in conversion rates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Recommendation and Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Based on the analysis of the AB test results using different techniques like z-test and regression, it can be concluded that the difference in percentage of users enrolling in the free trial of the data science program between the new and old web page is not statistically significant. \n",
    "\n",
    "* When the influence of the countries that the users resided in was taken into account, the difference was not statistically different for UK and US. For Canada however, it was observed that the new web page is about 8% less likely to convert users compared to the old page. \n",
    "\n",
    "* Based on these results, the recommendation to the e-commerce company is to not launch the new web page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}