diff --git a/math/calculus-for-ml/regression_with_perceptron.ipynb b/math/calculus-for-ml/regression_with_perceptron.ipynb new file mode 100644 index 0000000..bdb5315 --- /dev/null +++ b/math/calculus-for-ml/regression_with_perceptron.ipynb @@ -0,0 +1,1623 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "EAt-K2qgcIou" + }, + "source": [ + "# Regression with Perceptron" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "FZYK-0rin5x7" + }, + "source": [ + "In the week 2 assignment, you implemented the gradient descent method to build a linear regression model, predicting sales given a TV marketing budget. In this lab, you will construct a neural network corresponding to the same simple linear regression model. Then you will train the network, implementing the gradient descent method. After that you will increase the complexity of the neural network to build a multiple linear regression model, predicting house prices based on their size and quality.\n", + "\n", + "*Note*: The same models were discussed in Course 1 \"Linear Algebra\" week 3 assignment, but model training with backward propagation was omitted." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Table of Contents\n", + "\n", + "- [ 1 - Simple Linear Regression](#1)\n", + " - [ 1.1 - Simple Linear Regression Model](#1.1)\n", + " - [ 1.2 - Neural Network Model with a Single Perceptron and One Input Node](#1.2)\n", + " - [ 1.3 - Dataset](#1.3)\n", + "- [ 2 - Implementation of the Neural Network Model for Linear Regression](#2)\n", + " - [ 2.1 - Defining the Neural Network Structure](#2.1)\n", + " - [ 2.2 - Initialize the Model's Parameters](#2.2)\n", + " - [ 2.3 - The Loop](#2.3)\n", + " - [ 2.4 - Integrate parts 2.1, 2.2 and 2.3 in nn_model() and make predictions](#2.4)\n", + "- [ 3 - Multiple Linear Regression](#3)\n", + " - [ 3.1 - Multipe Linear Regression Model](#3.1)\n", + " - [ 3.2 - Neural Network Model with a Single Perceptron and Two Input Nodes](#3.2)\n", + " - [ 3.3 - Dataset](#3.3)\n", + " - [ 3.4 - Performance of the Neural Network Model for Multiple Linear Regression](#3.4)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "XI8PBrk_2Z4V" + }, + "source": [ + "## Packages\n", + "\n", + "Let's first import all the required packages." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "tags": [ + "graded" + ] + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "# A library for data manipulation and analysis.\n", + "import pandas as pd\n", + "\n", + "# Output of plotting commands is displayed inline within the Jupyter notebook.\n", + "%matplotlib inline \n", + "\n", + "# Set a seed so that the results are consistent.\n", + "np.random.seed(3) " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "## 1 - Simple Linear Regression" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 1.1 - Simple Linear Regression Model\n", + "\n", + "You can describe a simple linear regression model as\n", + "\n", + "$$\\hat{y} = wx + b,\\tag{1}$$\n", + "\n", + "where $\\hat{y}$ is a prediction of dependent variable $y$ based on independent variable $x$ using a line equation with the slope $w$ and intercept $b$. \n", + "\n", + "Given a set of training data points $(x_1, y_1)$, ..., $(x_m, y_m)$, you will find the \"best\" fitting line - such parameters $w$ and $b$ that the differences between original values $y_i$ and predicted values $\\hat{y}_i = wx_i + b$ are minimum." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 1.2 - Neural Network Model with a Single Perceptron and One Input Node\n", + "\n", + "The simplest neural network model that describes the above problem can be realized by using one **perceptron**. The **input** and **output** layers will have one **node** each ($x$ for input and $\\hat{y} = z$ for output):\n", + "\n", + "\n", + "\n", + "**Weight** ($w$) and **bias** ($b$) are the parameters that will get updated when you **train** the model. They are initialized to some random values or set to 0 and updated as the training progresses.\n", + "\n", + "For each training example $x^{(i)}$, the prediction $\\hat{y}^{(i)}$ can be calculated as:\n", + "\n", + "\\begin{align}\n", + "z^{(i)} &= w x^{(i)} + b,\\\\\n", + "\\hat{y}^{(i)} &= z^{(i)},\n", + "\\tag{2}\\end{align}\n", + "\n", + "where $i = 1, \\dots, m$.\n", + "\n", + "You can organise all training examples as a vector $X$ of size ($1 \\times m$) and perform scalar multiplication of $X$ ($1 \\times m$) by a scalar $w$, adding $b$, which will be broadcasted to a vector of size ($1 \\times m$):\n", + "\n", + "\\begin{align}\n", + "Z &= w X + b,\\\\\n", + "\\hat{Y} &= Z,\n", + "\\tag{3}\\end{align}\n", + "\n", + "This set of calculations is called **forward propagation**." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For each training example you can measure the difference between original values $y^{(i)}$ and predicted values $\\hat{y}^{(i)}$ with the **loss function** $L\\left(w, b\\right) = \\frac{1}{2}\\left(\\hat{y}^{(i)} - y^{(i)}\\right)^2$. Division by $2$ is taken just for scaling purposes, you will see the reason below, calculating partial derivatives. To compare the resulting vector of the predictions $\\hat{Y}$ ($1 \\times m$) with the vector $Y$ of original values $y^{(i)}$, you can take an average of the loss function values for each of the training examples:\n", + "\n", + "$$\\mathcal{L}\\left(w, b\\right) = \\frac{1}{2m}\\sum_{i=1}^{m} \\left(\\hat{y}^{(i)} - y^{(i)}\\right)^2.\\tag{4}$$\n", + "\n", + "This function is called the sum of squares **cost function**. The aim is to optimize the cost function during the training, which will minimize the differences between original values $y^{(i)}$ and predicted values $\\hat{y}^{(i)}$.\n", + "\n", + "When your weights were just initialized with some random values, and no training was done yet, you can't expect good results. You need to calculate the adjustments for the weight and bias, minimizing the cost function. This process is called **backward propagation**. \n", + "\n", + "According to the gradient descent algorithm, you can calculate partial derivatives as:\n", + "\n", + "\\begin{align}\n", + "\\frac{\\partial \\mathcal{L} }{ \\partial w } &= \n", + "\\frac{1}{m}\\sum_{i=1}^{m} \\left(\\hat{y}^{(i)} - y^{(i)}\\right)x^{(i)},\\\\\n", + "\\frac{\\partial \\mathcal{L} }{ \\partial b } &= \n", + "\\frac{1}{m}\\sum_{i=1}^{m} \\left(\\hat{y}^{(i)} - y^{(i)}\\right).\n", + "\\tag{5}\\end{align}\n", + "\n", + "You can see how the additional division by $2$ in the equation $(4)$ helped to simplify the results of the partial derivatives. Then update the parameters iteratively using the expressions\n", + "\n", + "\\begin{align}\n", + "w &= w - \\alpha \\frac{\\partial \\mathcal{L} }{ \\partial w },\\\\\n", + "b &= b - \\alpha \\frac{\\partial \\mathcal{L} }{ \\partial b },\n", + "\\tag{6}\\end{align}\n", + "\n", + "where $\\alpha$ is the learning rate. Then repeat the process until the cost function stops decreasing.\n", + "\n", + "The general **methodology** to build a neural network is to:\n", + "1. Define the neural network structure ( # of input units, # of hidden units, etc). \n", + "2. Initialize the model's parameters\n", + "3. Loop:\n", + " - Implement forward propagation (calculate the perceptron output),\n", + " - Implement backward propagation (to get the required corrections for the parameters),\n", + " - Update parameters.\n", + "4. Make predictions.\n", + "\n", + "You often build helper functions to compute steps 1-3 and then merge them into one function `nn_model()`. Once you've built `nn_model()` and learnt the right parameters, you can make predictions on new data." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 1.3 - Dataset\n", + "\n", + "Load the [Kaggle dataset](https://www.kaggle.com/code/devzohaib/simple-linear-regression/notebook), saved in a file `data/tvmarketing.csv`. It has two fields: TV marketing expenses (`TV`) and sales amount (`Sales`)." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "tags": [ + "graded" + ] + }, + "outputs": [], + "source": [ + "path = \"data/tvmarketing.csv\"\n", + "\n", + "adv = pd.read_csv(path)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Print some part of the dataset." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
TVSales
0230.122.1
144.510.4
217.29.3
3151.518.5
4180.812.9
\n", + "
" + ], + "text/plain": [ + " TV Sales\n", + "0 230.1 22.1\n", + "1 44.5 10.4\n", + "2 17.2 9.3\n", + "3 151.5 18.5\n", + "4 180.8 12.9" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "adv.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "And plot it:" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAGwCAYAAACzXI8XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAA9hAAAPYQGoP6dpAABFcUlEQVR4nO3de3SU1bn48WcySsRCQgyXQLkchCS29VKwSKMBLQzOhF5Q29M28axDrYseLfir9np0eaPtOvRylqeX02WP1RbbHkdqD5TqyqVEJBw0CIoULz2ZJMVCEWzNhASwREj27w+YaSaZyzsz72W/73w/a2UtmJm8s2fPm3mf2fvZz/YppZQAAAC4UJHTDQAAAMgVgQwAAHAtAhkAAOBaBDIAAMC1CGQAAIBrEcgAAADXIpABAACudY7TDbDa0NCQvPHGGzJ+/Hjx+XxONwcAABiglJJjx47JtGnTpKgo9biL5wOZN954Q2bMmOF0MwAAQA4OHjwo06dPT3m/5wOZ8ePHi8iZjigpKXG4NQAAwIj+/n6ZMWNG/DqeiucDmdh0UklJCYEMAAAukykthGRfAADgWgQyAADAtQhkAACAaxHIAAAA1yKQAQAArkUgAwAAXItABgAAuBaBDAAAcC0CGQAA4FoEMgAAwLU8v0UBAACFIBKJSHd3t8ydO1cqKyudbo5tGJEBAMDFotGohEIhqa6uluXLl0tVVZWEQiHp7e11umm2IJABAMDFGhoapLW1NeG21tZWqa+vd6hF9iKQAQDApSKRiLS0tMjg4GDC7YODg9LS0iKdnZ0Otcw+BDIAALhUd3d32vu7urpsaolzCGQAAHCpOXPmpL1/7ty5NrXEOQQyAAC4VFVVlQSDQfH7/Qm3+/1+CQaDBbF6iUAGAAAXC4fDEggEEm4LBAISDocdapG9qCMDAICLlZWVSXNzs3R2dkpXV1fB1ZEhkAEAwAMqKysLKoCJYWoJAAC4FiMyAADAtVscMCIDAEABc/sWBwQyAAAUMLdvcUAgAwBAgfLCFgcEMgAAFCgvbHFAIAMAQIHywhYHBDIAABQoL2xxQCADAEABc/sWB9SRAQCggLl9iwMCGQAACkiqwndu3eKAqSUAQEGKRCLS1NSkzRJjq9uTTeE73fomHQIZAEBB0a2SrV3tMVL4Tre+McKnlFJON8JK/f39UlpaKn19fVJSUuJ0cwAADguFQtLa2ppQBM7v90sgEJDm5mZPticSiUh1dXXa+ysrK7XqG6PXb0ZkAAAFQ7dKtna1x0jhO936xigCGQBAwdCtkq1d7TFS+E63vjGKQAYAUDB0q2RrV3uMFL7TrW+McjSQWbdunSxYsEDGjx8vkydPluuuu046OjoSHnPNNdeIz+dL+LnlllscajEAwM10q2RrZ3syFb7TrW+McjSQaWtrk9WrV8vOnTtly5YtcurUKbn22mvlxIkTCY9btWqVHD58OP7zne98x6EWAwDcTrdKtna1J1b4LhKJSGNjo0QiEWlubpaysjLb22ImrVYt/fWvf5XJkydLW1ubLF68WETOjMi8//3vl+9973uGjjEwMCADAwPx//f398uMGTNYtQQASKBbJVud2qNDW4yuWtIqkOnq6pLKykp5+eWX5eKLLxaRM4HMq6++KkopqaiokI9+9KNyzz33yPnnn5/0GPfff7+sXbt21O0EMgDgvFRVZYGRXBfIDA0Nycc+9jE5evSo7NixI377Qw89JLNmzZJp06bJvn375Gtf+5pcccUVsnHjxqTHYUQGAPQTjUaloaFBWlpa4rcFg0EJh8MJUxtAjOsCmVtvvVWamppkx44dMn369JSP27p1qyxdulS6uroyZliLUBAPAHSgU6E1uIOrCuKtWbNGnnrqKXnmmWfSBjEiIgsXLhQRfdezAwASubXQGtzB0UBGKSVr1qyRTZs2ydatW2X27NkZf2fv3r0iIjJ16lSLWwcAMINbC615hZs2gMzFOU4++erVq+Wxxx6TzZs3y/jx4+XIkSMiIlJaWipjx46V7u5ueeyxx2T58uVSXl4u+/btkzvuuEMWL14sl156qZNNBwAY5NZCa25XKHlJjubI+Hy+pLf/7Gc/k8985jNy8OBB+ad/+id55ZVX5MSJEzJjxgy5/vrr5e677zac70KODAA4jxwZ+7m9z12X7GsVAhkAcF5vb6/U19d7fnQgHTuXnhvd7VpnRq/fjk4tAQAKQ6yqrA6F1uzmxBSPkbwkr/Q/gQwAwDaVlZWeuYAa1dDQIK2trQm3tba2Sn19vWVTPIWUl6TF8msAALzIqaXnbt0AMhcEMgAAWMTJpedu3AAyF0wtAQCyxp5Jxjg5xWNHXpIO5wEjMgAAw6LRqIRCIamurpbly5dLVVWVhEIh6e3tdbppWtJhiqeyslLq6upMfS6dzgMCGQAoIPlWeU2XuIrkvDjFo9N5QB0ZACgAZiwB9kJtEid5Zem5XeeBqzaNBABYy4xv0OyZlB8rpnicoNt5QCADAB5n1hLgQqpNgtR0Ow8IZADA48z6Bq1D4iqcp9t5QCADAB5n5jdoLyauIns6nQck+wJAATB7J2S3Ja7qUO/Ei6w8D9j9+iwCGQAo3N2nndiwEeYgkDmLQAYA/s5tIyn5MnskCvYhkDmLQAYAChN1b9yNOjIAgIKmW70TWINABgDgSbrVO4E1CGQAAJ6kW70TWINABgDgWTrVO4E1znG6AQAAvZhRc0WXui1lZWXS3NxccKu1CgmBDABARMypuaJr3ZbKykoCGI9iagkAICLm7JBtxjGAbFBHBgA8KNupHTNqrlC3BWaijgwAFKBoNCqhUEiqq6tl+fLlUlVVJaFQSHp7e9P+nhk1V6jbAicQyACAh+Q6tWNGzZVMx5g0aVLGY+gsEolIU1OTdHZ2Ot0UDEMgAwAeEYlEpKWlJWFfIRGRwcFBaWlpSXsBNqPmSuwYyfh8Prn77rsNvAr95DrKBXsQyACAR+Q7tWNGzZVvfOMbSW9XSmUMpjJxakSEBGa9EcgAgEfkOz0Uq7kSiUSksbFRIpGINDc3Z7Vs+q233kp7fy55Mk6OiOQzygV7EMgAgEeYVZK/srJS6urqclphZMX+Rk6OiJDArD8CGQDwEKdL8pu9v5HTIyJsPKk/AhkA8BAzpofyZWYwZcWISDa5Nmw8qT+2KAAAD7K7JP/IAnxm7W9k5ohIrtsnhMNhqa+vT/g9Np7UB5V9AQA5s2NvpVAoJK2trQnTS36/XwKBgDQ3N9t2HDaetJfR6zeBDAAgZ2YFGen09vaOGhHJNlhi+wT3MXr9ZmoJAJCTWCLuSMMTcfMNDmJTVj/84Q9FRHIeETGSa0Mg404EMgCAnFgZHJg9ZcXqo/xluxGpXVi1BADIiZXBgdm1Y1h9lDvdt2ggkAEA5MSq4MCq2jFO19hxK923aCCQAQBNuHF3ZSuCA6uq6epQY8dtnC5IaAQ5MgDgMDuWMFslFhyYuTTZ6nwWu2vsuJkbkqQZkQEAh+k+dG9EPvszjUQ+iz7ckCRNIAMADnLD0L0TyGfRgxuCSgIZAHAQuysnRz6LPnQPKsmRAQAHuWHo3knksyRnZ00XK/KgzMSIDAA4KNXQfVFRkcyfP9+hVnmHG1eCpeNkTZfKykqZM2eOdHV1adWfBDIA4LBkQ/dDQ0OyZ88e7YqPuYXuRdxy5VRiuM79yaaRAKCJzs5O+fSnPy2///3vLd2EsRDYsZml3Zzc+NKJ/jR6/WZEBgA0oZSSPXv2sIIpT15dCeZUYrju/UkgAwCaYAWTObzaj04lhuvenwQyAKAJVjCZw6v96FRNF937k0AGADThhuJjbuDlfnSipovu/UkgAwAa0b34mFuY0Y9WLd3O57hOFQrU+bxk1RIAaEjX4mNmsLOYWy79aNUmnm7eHDTGzvPS6PWbQAYAYAu3XMitWmrsxSXhVnLF8ut169bJggULZPz48TJ58mS57rrrpKOjI+ExJ0+elNWrV0t5ebmMGzdOPv7xj8ubb77pUIsBALlywy7fVi011n0Js5s5Gsi0tbXJ6tWrZefOnbJlyxY5deqUXHvttXLixIn4Y+644w558skn5YknnpC2tjZ544035IYbbnCw1QCAbLnlQm7VUmPdlzC7maObRo4cSlu/fr1MnjxZXnzxRVm8eLH09fXJI488Io899pgsWbJERER+9rOfyXve8x7ZuXOnfPCDH3Si2QCALBm5kOuQC5TrUuNMeT+6L2F2M61WLfX19YmIyAUXXCAiIi+++KKcOnUqIVP6oosukpkzZ0p7e3vSYwwMDEh/f3/CDwDAWfleyO3a/DHbpcZG9yDSfQmzm2kTyAwNDcntt98uV111lVx88cUiInLkyBEZM2aMTJgwIeGxU6ZMkSNHjiQ9zrp166S0tDT+M2PGDKubDgDIINcLuRObFWaz1DibvB+dlzC7mTarlm699VZpamqSHTt2yPTp00VE5LHHHpObbrpJBgYGEh57xRVXyIc+9CH59re/Peo4AwMDCY/v7++XGTNmsGoJQN7sXDbsRb29vfKxj31MduzYEb8t06olJ1f6ZFpqnOsmjl5eWm8mo6uWHM2RiVmzZo089dRTsn379ngQIyJSUVEh77zzjhw9ejRhVObNN9+UioqKpMcqLi6W4uJiq5sMoIC4ZdmwzmJ9ODyIqa2tTduHsQThkYYnCFsZCFRWVqY9fq55P5mOi+w4OrWklJI1a9bIpk2bZOvWrTJ79uyE+y+//HI599xz5emnn47f1tHRIQcOHJCamhq7mwugQLlh2bCuYrkt11133ag+bG9vT9uHuq/0IYFXD45OLX3+85+Xxx57TDZv3pwwPFdaWipjx44VkTNTTo2NjbJ+/XopKSmR2267TUREnnvuOUPPQUE8APnIdfqg0CUbxUolVR+6oe8pcmcdVxTEe/DBB6Wvr0+uueYamTp1avxnw4YN8cf8x3/8h3zkIx+Rj3/847J48WKpqKiQjRs3OthqAIVE91EBXSUbxUolVR+6YaUPCbzO0ybZ1yqMyAD6ckPyrBtGBYbToU8z9Vmyx6dqa29vr9TX12ufn0QCr/lclewLoLC4KXk2NiqQavpAl4uWTn2aaRQrxkgfxnZ71j1QIIHXOdrUkQFQONyWPOuG6QOd+jRTEmxMNn1YWVkpdXV1BAsYhaklALZy21TNcOlGBZyc0jGzT816HamSYGtqauSuu+7SdmQF+nBFsi+AwuPm5NlkowJOVJ4dyYw+Nft1pBrF+u1vf2vLyIpZWxrYtTUC8qA8rq+vT4mI6uvrc7opAJRSHR0dSkRS/kQiEaebmJVgMKj8fn/Ca/D7/SoYDNrWBjP61KrXEYlEVGNjo23va09PjwoGgwmvIxgMqmg06shxkDuj128CGQC20+HibwadgrJ8+lSn15Evs84tr5yjbmb0+s3UEgDbuSF51gidpsny6VOdXkc+YlsaDM/LEUnc0sDO48AeLL8GYDu3LKnNRKcS9fn0qU6vIx+57n1k1XFgDwIZAI4xu/aG3SuHdKwxk0uf6vg6cmFWQOaVwK5QMLUEwPWcXDnklWkyL7wOs7Y0cMPWCPg76sgAcD0dNu5z+zRZjNHXocNWCMmYtaWBW7ZG8DKj128CGQCu5uYCe07INwDRaSuEdMwKLL0SoLoRey0BKAgkZhpjVgCSbisEu0a/jDAr/4o9lPRHjgwAVyMx0xgz9mJiWTJ0RCADwNVIzMzMrADEK/Vm4C0EMgBM5cTeNFasuPHSHjtmBSC5jn55qS+hHwIZAKZwcgl0rBhcJBKRxsZGiUQi0tzcnFPyqQ6bQJrNrOm3bEe/vNiX0JAN2yU4ir2WAHt4ZW8ar7yOkcx6XdFo1PBmil7tS9jD6PWb5dcA8uaVJdBeeR3JmF0XJdOyZC/3JezB8msAtvHKEmivvI5kzN7fKtOyZC/3JfRCIAMgb15ZAu2V15GOXXVRCqEvda1uXGhI9gWQN68sgfbK69CBl/uSJGa9EMgAMIUXNh0U8c7r0IFX+9KM4oIwD8m+AEzllb1pvPI67JZsusVLfWlFEjNTVMmR7AvAEU7sTWPFhcAte+zochFMt5eTW/rSCDOTmN2yAafumFoC4FqFnKug22svlOkWM5OYC6XPrMbUEgDbmD16EAqFpLW1NWEPIb/fL4FAQKudmK2g02svtJoxZvR9ofVZLoxevxmRAWA5K0YP3LYTs5n7Den22gttM0kzkpgLrc+sRCADwHJWDKG75UJgRRCn22vPNN0yadIkm1piDzP29iqEOjt2IZABYCmrRg/cciGwIojT6bVHo1H5f//v/6W83+fzyd13321be+xUWVkpdXV1OU0BebnOjt0IZABYyqrRAysvBGZNA1kVxOl0EUwWqA2nlNJyqk8HXq2zYzcCGQCWsnL0wOwLgdnTQFZOAelwEUwVqCWjy1SfTsyYogKrlgDP0aWuyHBWr7Axq+Ca2e20Y2WKk8XmmpqaZPny5YYeyyocZMvw9Vt5XF9fnxIR1dfX53RTAEv19PSoYDCoRCT+EwwGVTQadbppKhqNatu2mI6OjoT2jfyJRCI5HTcYDCq/359wLL/fr4LBoMmvIL2Ojg7V2NiY9HWkuy/TMdP1mVOvFd5g9PpNIAN4hC4XzHQikUhOF0w7NDY2pr0gNzY25nRcp4O4dAGuGcFvsvNO54AV7mH0+s3UEuABFNfKn9V96NQUULrpMhHJeyqtt7dX6uvrE8rs19bWym233Sbz5s3jvEPOjF6/CWQAD8iUq9DY2Ch1dXU2tsiddKqWa4ZMwVmm380mCPHSxpDQA5V9gQKiU10RN9NhJZCZMq2aSifbVUb51FQB8sHu14AHxOqKpBpN4OJiTGw5rFdGFzIFuOkQ/MItGJEBPMJrowlO8sroQqbCeboU1QPyQY4M4DFeGU2AOZIl4waDwXiAm+o+irJ5g451pYwi2fcsAhkUIjd/eMEa6QJcgl/viUaj0tDQ4OoglUDmLAIZFBIvfHgByJ8XVuCxagkoQFbstDySWRsqwp3c/P67ue3ZyGazUi/0CYEM4BFW7bQcY/aGinAXN7//bm57LoxsVuqlPiGQATzCyp2WRewZ7YG+3Pz+r1ixQrZs2ZJwm1vangsjdaXc/H6ORCADeISVRfGsHu2B3tz6/kejUVm0aJHs2LFDhoaGEu7Tve35yLTsXinlyvczFQIZwCMyfXjlsxrF6tEeGONUPoNb3/+GhgZ57rnn0j5G17bnK11dKbe+n6lQ2RfwkHA4PKouiBlF8ezcAoGl42cM74fy8nJHV6M5vQVGLudEbBQpE69WME5Xpdrp99N0Vm7BrQOj24ADXhKJRFRjY6OKRCKmHTMYDCq/369EJP7j9/tVMBg05fg9PT0qGAwmHD8YDKpoNGrK8d0iWT+Ul5db2vdGWP3+J5PPOdHY2JjweyN/ioqKbO0/3TjxfmbL6PWbQAaAIdFo1NJAww0frHZI1g/pfswMVtOx+v1PJp9zoqOjI22/1dbWFlyQPJwT72e2jF6/KYgH2MztUydWVIGNRCJSXV2d9n439lW2MvVDMo2NjVJXV2dRi0azqwqwGedEsqJwRUVFctVVV8n27dtNa6ub6VzV2daCeIODg7J3715Xrj8H7OLWug0jE0yt2FDRa8mHucrUD8kMz2ewIxnYrg01zTgnkiW8Llu2TDZv3pxX27zECxuk5hTI3H777fLII4+IyJkg5uqrr5b58+fLjBkzZNu2bWa2D/AMt9VtsDPw8lzyYY4y9cNww1ejuTVITseMcyKW8BqJRKSxsVEikYg0NzezXYfX5DJv9e53v1vt3r1bKaXUpk2b1LRp01RHR4e6++671ZVXXpnLIS1Djgx0kGm+3q48h2zYnbNCjswZyfqhqKhIlZeXp8xn8GrfefV1wRhLk32Li4vVwYMHlVJKrVq1Sn3hC19QSin1xz/+UY0fP97wcdra2tRHPvIRNXXqVCUiatOmTQn3r1y5ctQHfrYnMIEMdJBpBUVjY6PTTUzgROCVLPlw/vz58S9NhSJdEmay1WhuDJKNckNCqlt1dHSYvrLRbEav3zlNLU2ZMkVee+01GRwclObmZlm2bJmIiLz99tujinGlc+LECbnsssvkRz/6UcrHhEIhOXz4cPwn33oYgBPcNnXiRM5KbBpg165dMn/+fBER2bNnjyxYsMC10yS55Kykmw6prKyUOXPmSFdXV/yYXs4vYmrIfF6chsxpROa+++5TpaWl6qKLLlIzZ85UJ0+eVEop9cgjj6gPfvCDuRwy5YjMihUrcjpeDCMy0IWbhsmd/Jbvpn5KxYqaOKmOuWvXLs+OyLhh1EB3I/vQTX9flteReeKJJ9QDDzwQn2JSSqn169er3/zmNzkdL1UgU1paqiZNmqSqqqrULbfcot566620xzl58qTq6+uL/xw8eJBABlpw2zC5Ex94XpkmsaLv0h3TTRcnIyiOmL9kfVhbW+uqvy/bCuL97W9/y/cQSqnkgUw4HFabN29W+/btU5s2bVLvec971IIFC9Tp06dTHue+++5L+gYRyGAkp77tWVF11wpOBF5uyyVKxopgLNMxd+/e7akLv9cCMyekShp309+XpYHM6dOn1de//nU1bdo05ff7VXd3t1JKqbvvvls9/PDDuRwyaSAzUnd3txIR1dramvIxjMggk0zf9hjOTmRn4GU0CND5PbIiGNuwYYOhY7olSE7HK6NyTsrUh27pW0sDmbVr16oLL7xQ/fKXv1Rjx46NBzKPP/64qTkyyUycOFH9+Mc/NnxccmQwUqpvex/60Ic89a3WrdJ9G7d7yiGXgMmKC7HbpgTy4YVROacZ2WfKDaNdlgYyc+bMiY+KjBs3Lh7I/OEPf1ATJkzI5ZCGApmDBw8qn8+nNm/ebPi4BDIYLtNFhuFs56Wb0rJryiHfgMnMdmY6ZxctWpT1MXXGiEz+jJwzbvjCZmkgc95556nXX39dKZUYyLz66qvqXe96l+HjHDt2TL300kvqpZdeUiKiHnjgAfXSSy+pP/3pT+rYsWPqy1/+smpvb1f79+9Xra2tav78+aqysjK+SsoIAhkMl+mbCh+e+hg5TWLnBS7XQCQ2gmNmzkqmc3bDhg25vkxtkSOTv0x96IZpSEsDmfnz56tf/OIXSqnEQGbt2rWqtrbW8HGeeeaZpH+YK1euVG+//ba69tpr1aRJk9S5556rZs2apVatWqWOHDmSVVsJZDBcrnPHDGc7z64ph1wCplQjOLt37877YlGIIxRuW+GnIy/0odHr9zmSg3vvvVdWrlwphw4dkqGhIdm4caN0dHTIz3/+c3nqqacMH+eaa64RlWbz7ZaWllyaB6RUVVUlwWAw6Y64Q0NDKX9Pt4J1VtJ1d267igoaKTA3sl9S7aMlItLc3JxXe1Kds36/XwKBgFbvkVlihfB03plZdwXVh7lGStu3b1eBQEBNmjRJjR07Vl111VWqpaUl18NZhhEZjJTqm8qSJUsKejjbDbU77JhyyHYExI4REy98uwayZVsdGd0RyCCVkXPEhX6xcENegl3vUTZ9kWnK66GHHjItF8ENeQ2AWYxev31KpZnb8YD+/n4pLS2Vvr4+KSkpcbo5cAEdh2Ktnu6JRCJSXV2d9n5d+kLE+veot7dX6uvrE6a3g8GghMPhUfv8ZOq74VIdA8BoRq/fhgOZsrIy8fl8hp48Go0aa6UNCGTgZtFoVBoaGgxdUPPR1NQky5cvT3l/Y2Oj1NXVmfZ8bmE0YAqFQqNyWHw+36gcwFheS755M0AhMD2QefTRRw0/+cqVKw0/1moEMnCzZBdIKy6GbhuR0U2yEZx06E8gM9MDGbcikIFb2R1c2BU0eVlsBOfQoUOyatWqlI8r1BEuIBtGr99F+T7RyZMnpb+/P+EHQP6MLAM2UzgclkAgkHBbIBCQcDhs6vN4WWVlpdTV1cnixYvTPq6QlvMDVsupjsyJEyfka1/7mvzqV7+Snp6eUfcP/0YHOEHXWijZsKtuSkxB1Z2wWCHWfgGcktOIzFe/+lXZunWrPPjgg1JcXCwPP/ywrF27VqZNmyY///nPzW4jYFg0GpVQKCTV1dWyfPlyqaqqklAoJL29vU43LWuxi6Hf70+43e/3SzAYtOxiGBtV4GKbH0a4AHvklCMzc+ZM+fnPfy7XXHONlJSUyJ49e2Tu3Lnyi1/8QsLhsDQ2NlrR1pyQI1NYvJbnkc0yYOiJES4gN5Ym+44bN05ee+01mTlzpkyfPl02btwoV1xxhezfv18uueQSOX78eF6NNxOBTOHw8sobr18MvTAVCPMYOR84Z7zP0mTfCy+8UPbv3y8iIhdddJH86le/EhGRJ598UiZMmJDLIYG82Z0cK3Lmw7SpqUk6OztNP/ZwXp3u8dJUIPJn5HzgnMEouZQNfuCBB9T3v/99pZRSW7ZsUeedd54qLi5WRUVF6nvf+14uh7QMWxQUDjt3CXbDvkS56ujosLQM/vDju2FbBNjHyPmQzzlj9bkNc9m619Lrr7+u/ud//kf9/ve/N+NwpiKQKSx2XRi9eAG2OjhLdny7Ak/oz8gXkVy/rHj5i4eXGb1+ZzW11N7eLk899VTCbbGk31tuuUX+8z//UwYGBrI5JGAqO1aKRCIRaWlpGVVmYHBwUFpaWiyfZrJKQ0ODtLa2JtzW2toq9fX1lh0/HSumAqEvI1PDuU4fW31uw1lZBTJf//rX5dVXX43//+WXX5abb75ZAoGA3HnnnfLkk0/KunXrTG8kYFSsFkokEpHGxkaJRCLS3Nxs6gofJ3JxrGZ1cJbq+OlQNK6wGKmblEttJa9+8cDfZRXI7N27V5YuXRr//+OPPy4LFy6Un/zkJ3LHHXfID37wg3jiL+AkK5NjM32YHjp0yHUfjmYHZyOToDMdfzir6+Toxq6E8VzZ1T4jdZNyqa3kxS8eGCGb+ari4mJ14MCB+P+vuuoq9c1vfjP+//3796tx48ZlOQtmLXJkYIVkOTI+n8+1c/BmJUqnykXYtWuX4dwYN/VbPnTP23CifdFoNONzGnnMcHYuAoC5LEn2nTlzpmpra1NKKTUwMKDGjh2rWltb4/fv27dPlZWV5dBc6xDIwArJPkxH/rgt+deMBOZ0x0h3XyQS8fxqkpErZnRPGHeyfUbOh2zOGd37WmdOrvSyJJC55ZZbVE1Njdq+fbv64he/qMrLy9XAwED8/l/+8pfqAx/4QG4ttgiBDKwUiUTUQw895IlvfNl+0x0p0zff3bt3az0CYZVkIxu1tbVanzNeG8XI99y2mo7LwnUYMbQkkPnrX/+qFi1apHw+nxo/frzauHFjwv1LlixRd911V/attRCBDNIx4wOksbEx7Yd+Y2OjiS22Xq6jI0b7wY2jL/mcJ8lGA4qKirQ+Z7x2Tsfodu7pECykosMolqV1ZI4ePapOnz496vaenp6EERodEMggGTM/QLz27TVXXuyHfM+TTH2ia1958b3UkQ7BQjK6vP+W1JGJKS0tHZU1LiJywQUXyJgxY3I5JGAaI6sszKwr4dQu1brxYj/ke55kWjFTVJT4EaxLX3nhvXTDajBdl4W7bqWXLWGVgxiRKRxGvz1b8W1Dpzl4J+fbdeqHfJlxnmQ6xshcGZ36yq3vpc7TNcPpPH3nthEZAhl4htFhWis/QJycg9fpA1y3XIRcmHWeZDovde8r3ds3Uqr+rq2t1ep16BIspKLDtBeBzFkEMoUhmw8F3T9AcqXDB4+XmHWeuHVkw42M5iTp0v86/83qcN4SyJxFIFMYsv32rPMHSC7cHpzpuPxUKXPPE7eNbMSY9d7Y8R5n+hzQ7W9dh2AhEyfPWwKZswhkCkO2F3I3fIBkQ+f59nR0mg5LxmvnSTbMem/sfI+zXSWmS1Dp1iDXakav3z6llBIP6+/vl9LSUunr65OSkhKnmwMLhUIhaW1tTVgF4Pf7JRAISHNzc9Lf6ezslK6uLvH7/TI4OChz5851xYqMkSKRiFRXV6e9X8fXlct75oTYeeLW8yMXZr03dr/HyZ4vlcbGRqmrqzO9DZlEIhHp7u4uqPMpF4av37aEVQ5iRKZw5PLtWfcRgWy4bbrM7dNhXmbWe+PEe2xk+xCnzjEvfd7YwdI6MoCOysrKpLm5WSKRiDQ2NkokEpHm5mYpKytL+Ttm1pNxWjgclkAgkHBbIBCQcDicsqaGk7U2XFerooCY9d7Y+R7HzuW33nor4XNg0aJF2tTD8dLnjVZsCqwcw4iM+9iV+OnVEYHh8+2pvgF2d3c7/s3Qq/3vBW4akck0yqFLnpPVfaFrwnw+SPY9i0DGPewednVrgmw2Uk03lZeXazEN5bbpsJjhFw23XECybadZ702m4+Tbf0bb6XRCrVWfN16eriKQOYtAxj3svqh5fUTADfv86PJt2ahkFw3d257rhc6s9ybVccwYFXTT37BVbXXrlwEjCGTOIpBxTjbftJz4QHLTh2AujNbUSPbN0O5RBqe/LRuV7KKh+wUk3wudWe/NyOOk2hW8trbW8DHdNqpqdtDh9c8wApmzCGTsl8s3QCc+kNz2IZhKqqAj1xEZnff/cVI2/anLBUTXC12mdi1atMjQOafr60vF7BFIr3yGpcKqJTgml8z8OXPmpD3m3LlzTWmb0885Uj6rhqLRqIRCIamurpbly5dLVVWVhEIh6e3tFZH0OxiXl5envL29vT3hdlZVnJFpBc5wuqy4ytTmtrY2m1qSKFO7nn32WUPnnNt26c5lZWU6OnyGacGmwMoxjMjYK59vSE7M9To1v2xGgp6Rtqf6BvjHP/5x1O0jR2Kyee8KgRdHZHI57+xql9F+dFueldnIkWFqCSbLZ6jT7A8kI3keu3btUvPnz7f9QzDfD59sA8ZUeQ7Db/f6MLUZvJIj43SbjQYy2ZxzbsmzMpuXAzkCmbMIZOxlxpx1vh9IRkY7kj1m/vz5avfu3Tk9ZzbM6CMrgg4r8w3cskQ5k0xVY3W8gBitdGvne2M0Ed3t54udvBjIEcicRSBjP6eHOo08v5NtNCMIcctSTq/WuBh+0XDLBeQnP/mJNiNumc7foqIi7Ua2YD8CmbMIZOzn5FCnkQu80ysdzHp+K4Ixs987p4Na/J3T5/1I6aa8vBDsIn8EMmcRyDjHiW+qRkY7dMgFMeMCn0/QkWmqJ5f3buQxdbtwQq/AMtn5a9f0LtyBQOYsApncuTGvwcjFs7m52fELrJkjH9kEHVZM9aQ65oYNGxwPGJFIx8RQt0zNwX5Gr98+pZQSD+vv75fS0lLp6+uTkpISp5vjCtFoVFasWCE7duyI31ZbWyu33XabzJs3T7vaDCOFQiFpbW2VwcHB+G1+v18WL14sY8aMkZaWlqS/5/f7JRAISHNzs11Nlc7OTunq6pK5c+fa0q+p+iaf153qmDU1NQnn0EiRSET7c8mr7D7vgFwYvX4TyCBBNBqVqqoq6enpSfmYYDAo4XA45yJOVuvt7ZX6+vqEgCUYDMqpU6ekra0t4YI7nO6vK1+RSESqq6vT3p/tRS3TMRctWiTPPfecqYETgMJg9Pp9jo1tggusWLEibRAj8vdKr7peiGLVM4d/61RKpb3g/u53v5Nly5bZ2Mr0IpGIdHd3m/qNOVM11a6urqyfK9Mx16xZI+eff35CUBkIBCQcDmf1PACQCoEM4iKRSNqpgJjBwUFpaWmRzs5OrYelKysr4+1rampK+9jTp0/n9VwjA49cA5FoNCoNDQ2jRpPMGCmyopx5pmPOmzdvVFCp8zlTKKwIlAHHWJ+u4yySfY3LdrdktyRrdnR0qIceesiSBN9kia7l5eU5J1NavarEiuPrtBImEzcmsJvJq3V94E2sWjqLQMa4bHdL1v1ikOxD2+fzWX4RH/lj9DnsWK5sxaoVHVfCjMQF/Aw3BZ0AgcxZBDLZCQaDqqioKKsLs67fco0EGfnu5WRm4Ge0vo0Z/W3Fkledl9FyAaeuD9yHQOYsApnsJPt2PWHChKQXf52/5Wb60P7JT36S9we32VNxmdq8a9cubfs7X7HpPzPel2TH5gJuzf5cgJUIZM4ikMnNyG/Xyb5t6/wt144PbSum4tL1qc79nauenh61ZMmSUX21ZMkS0wI0LuBnENDBbQhkziKQsYbuH4p2tc/MHBmlUueb7Nq1S+v+zlUwGByVtzT8dRuVbrpN93PVTl4MhuFdBDJnEciYr6enR82fPz/txWHDhg1ON9OWD+1kgUc+q5ZiRo6AeXFUwciIVqYgw+j0JhfwM9yQmA3EuCKQaWtrUx/5yEfU1KlTlYioTZs2Jdw/NDSk7rnnHlVRUaHOO+88tXTp0qy/PRHImM/IKMSiRYucbqatH9pGpuLy4cVRBSM5RpkCNKMBChfwRDonZgMxrthrqampSZ599lm5/PLL5YYbbpBNmzbJddddF7//29/+tqxbt04effRRmT17ttxzzz3y8ssvy2uvvSbnnXeeoedgiwJzZSpJP/KxOhTbylSMzS3FwazYJ8lJRs6ldOdQLlsuUJgPcA/D129bwioDZMSIzNDQkKqoqFDf/e5347cdPXpUFRcXq3A4bPi4jMiYK5uVOrpPd+i86ioZL44q5JMj48XpNgB/Z/T6XWRi8GSq/fv3y5EjRyQQCMRvKy0tlYULF0p7e3vK3xsYGJD+/v6EH5gnU0n64XIpeW+nhoYGaW1tTbgtto+UjmJ7SEUiEWlsbJRIJCLNzc2u3uQyHA7Lhz70oVG3L1myJON+TFZsuQDAfbTda+nIkSMiIjJlypSE26dMmRK/L5l169bJ2rVrLW1bIauqqpJgMDhqimO42HSHGUP32exhlM0UUSQSSdjPKMaOfaSStTObtg/fQ8rtysrK5Omnn5bOzk5pa2sTEZGrr77a0OtLdS6aef4BcAGbRogykhFTS88++6wSEfXGG28kPO4f//Ef1Sc/+cmUxzl58qTq6+uL/xw8eJCpJZMlm+IQk6c7stnDKJcpIiemJZK1c8mSJaPqqLh9ushOXpxuA3CG0aklbUdkKioqRETkzTfflKlTp8Zvf/PNN+X9739/yt8rLi6W4uJiq5unLSPf7PNNblVJ8sNra2vltttuk3nz5pnyTTjZtE9PT0/C/4dPA6WaIkqVBOvEtESy17R169ZRj2tpaZFPfOIT8vTTT5veBq+JTbeRxAsUMHviqswkRbLvv//7v8dv6+vrI9k3BSOjEmYlt1pdkyPbirnpftItL7Wztkgur4mlsQAKmSuSfY8fPy579+6VvXv3isiZBN+9e/fKgQMHxOfzye233y7f/OY35be//a28/PLL8s///M8ybdq0hCXaOMNI4qoZya2x3JKR+THDc0vy1d3dnfcxYrq6ulLeFw6HE5LJRUSuvPLKjEmmucjlNcVyRgAAqTkayLzwwgsyb948mTdvnoiIfPGLX5R58+bJvffeKyIiX/3qV+W2226Tz33uc7JgwQI5fvy4NDc3G64h4wWRSESamprSBghGgguzApBMF+R0gYNR2ayMyuTQoUMpX1tZWZk89thjUltbG7/tf//3f6W+vl56e3tNa4OIua8JADCMTSNEjnHr1FI200BGElfNSm7VcQ8jI49N1Xd2Ti8Fg0FVVFTE1BIAGOCKLQrs4NZAxugFtqenR9XW1ma8IJoZgOi2h1Gyx44sspasfXaX/Y9Goxnfq1jblyxZYupzA4DbEMic5cZAJpsLbLrRiJEXb7MCEF33MIpEIuqhhx4y3HdOVYatra1NOzLD8mEAIJCJc2MgY/QCmyngWbRoUcIF0ewARMeN57IJTpzaiDHZ+7Bo0SK1YcMGrfoSAJzk+joyhcxojZNMibd33nlnQvl6s2tu6FhhNpv6ME5VhqX2CQCYR9u9lgpZ7ALr9/sTbvf7/RIMBuMXvVyLulVWVkpdXZ0nL55G+y4m2RLsQCBgyRLskbz8PgCAXQhkNGXkApvuol1bWytdXV2m1HVxm2yCk0wbMRpZ/g4AcI5PqST15j2kv79fSktLpa+vT0pKSpxuTlYikYhs375dRFJvpNfb2yv19fUJGyCWl5cnlPMPBoMSDoe13yU5360TRspn6iYajUpDQ0NCv2bbj2a/HgAoJIav37Zk7DjIjcm+uWwlEEu8XbRokW11UcxixtYJHR0dpiYe57PCy6ytIACgkLFq6Sw3BjK5XkSdWoWTL92Chnz70c4iewDgVa7Yawmj5bOVgB3bB5gt360TzNg/aqR8+tGOvagAAH9HIKOZXC+i0WhU/u3f/i3t76ZaxWSHVEmzOgYNua4GE3FnMAkAbkYgo5lcL6INDQ3S3t6e9L5US4/tEI1GJRQKSXV1tSxfvlyqqqokFArFN2XUMWjIdgn3cPm8HgBA9ghkNJPLRTTVyETMlVdeaUtdlGQyTf3oGjTkWl8mn9cDAMiBTTk7jnFjsm+2Wwk4tWdQJkaTZvPZOsHqxNpctmGwcy8qAPAqtihwsWxL2Os6nZFp6mfbtm3x15dryf5wODyqjo6ZlXlz2YaBLQgAwD4UxHMBI4XVQqFQyj2Dmpub7WpqgkgkItXV1YYem2/RPoIGAPAWo9dvcmQ0lilRdjgn9wxKJVW+iM/nG/XYfJdMs28RABQmRmQ0lssoi24jE8m2UEgnEolo0W4AgLOMXr8JZDSVaVrGbRf8WIB16NAhWbVqVcrHNTY2Sl1dnY0tAwDoyOj1m2RfTRmpkeKmQCaWNBuJRNI+jjorAIBskCOjKV1XIuWLOisAADMRyGgq1QW/qKhIamtrbbvgp9paIB86JiYDANyJQEZjyS74Q0NDsmPHjpSrl8ySzYqpbMXqrEQiEWlsbJRIJCLNzc05L70GABQukn1dYPHixfLss8/K0NBQ/Dara8ToWJcGAFA4WLV0ltsDGSdWL3ltxRQAwH0oiOcCRvJPzNjhOds8F6t2lQYAwGwsv7ZRbKuBiRMnyj333JNQJC5Vif5sVy8N386gvLxcGhoaDD1PPs8JAIBTmFqyQTQaHRVQ+Hw+Gd716fJPjOSrJHuO8vJy6e3tzSm3hhwZAICTmFrSSENDg7S2tibcNjJ+HBwclJaWlqTTP+FwWGpqahJuG7lcOdlz9PT0JAQxmZ5n5HOyRBoAoDumliwWiUQM7zMkMrpib2ykZceOHfHbamtrE6aHsn2OZM8zUmyJtG57NwEAMByBjMUyJc6ONDL/JNlIS3t7u9TX18eneLJ9jmTPk0psawEAAHTE1JLFMiXOxiQr0R8baRmepyIyenrI6HOkeh4AANyKQMZiqbYaGClZ/onRZdBGnyPV8wAA4FYEMjZIljgbDAZl9+7daUv0Z7MMOtlzJPPDH/6QrQAAAJ7B8msbxGq7nHPOOXL69OmsEmcnTpwoPT09o24vLy+Xt956a9TtDz/8sKxatSrl8RobG6Wurs544wEAcIDR6zfJvhZKVtslVpDOiEgkkjSIETmztPqFF16QD3zgAwm3L168OO0xKWYHAPASppYslGzFUWtrq9TX1xv6/Uw5Mv/yL/8y6rZU+TIk+QIAvIhAxiJGVxylkylHZs+ePSkL6FHMDgBQCAhkLGLGxotVVVUyf/78rI8TK2YXiUTSJhMDAOB25MhYxKyNF3/84x/LFVdckdNxKGYHAPA6RmQsYlauyoIFCyQYDEpRUeJbRc4LAAAEMpYyK1clHA7LsmXL8j4OAABeQx0ZG5i18SIbOAIACgV1ZDSST65KrJheLHghgAEA4O+YWtJUNBqVUCgk1dXVsnz5cqmqqpJQKCS9vb1ONw0AAG0QyGgq32J6AAAUAgIZm0QiEWlqajJUCM+MYnoAABQCAhmLGZ0iGh7omFFMDwCAQkCyr8XSTRE1Nzcn3ViytrY27THZ+BEAgDMYkbHQrl27Mk4RJQt02tvbpby8nI0fAQDIgEDGQrfeemva+7dt25Yy0Onp6ZErr7wy4XaK4AEAkIipJYu0tLTInj170j7G5/Olvf/OO++URx55hCJ4AACkQCBjsmQ5L8nMnz9fLr300rSPoQgeAADpEciYLFnOSzL/9V//JXfffbf4fD5JtksEuTAAAGRGIGOiWP2XdIqKimTZsmVSUlKS9rHf/OY3zW4eAACeo3Wy7/333y8+ny/h56KLLnK6WSllqv8iIrJs2TIJh8MZH/vXv/7VrGYBAOBZ2o/IvO9970uYqjnnHH2bPGfOnLT3/+53v5Nly5YZeiy1YgAAyEzrERmRM4FLRUVF/GfixIlONymlqqoqCQaDKeu/xIIYI48lPwYAgMy0D2Q6Oztl2rRpcuGFF8qNN94oBw4cSPv4gYEB6e/vT/ixUzgclkAgkHBbIBCQb3zjG6P2Wkr1WGrFAABgjE8lWzKjiaamJjl+/LhUV1fL4cOHZe3atXLo0CF55ZVXZPz48Ul/5/7775e1a9eOur2vr09KSkqsbnJcZ2endHV1ycSJE+Wee+5JSOwNBoMSDoelrKws4bHUigEA4Iz+/n4pLS3NeP3WOpAZ6ejRozJr1ix54IEH5Oabb076mIGBARkYGIj/v7+/X2bMmGF7IBMTCoWktbU1oXqv3++XQCAgzc3NtrcHAAA3MBrI6Js5m8SECROkqqoq7e7PxcXFUlxcbGOrUku1HHv4XkuMwAAAkDvtc2SGO378uHR3d8vUqVOdboohmZZYpwvIAABAZloHMl/+8pelra1NXn/9dXnuuefk+uuvF7/fL/X19U43zRCWWAMAYC2tA5k///nPUl9fL9XV1fLJT35SysvLZefOnTJp0iSnm2YIS6wBALCWq5J9c2E0Wcgqvb29Ul9fn3bVEgAASOTJZF83Kisrk+bmZpZYAwBgAQIZm1RWVhLAAABgMq1zZAAAANIhkAEAAK5FIAMAAFyLQAYAALgWgQwAAHAtAhkAAOBaBDIAAMC1CGQAAIBrEcgAAADXIpABAACuRSADAABci72WTBaJRKS7u5vNIQEAsAEjMiaJRqMSCoWkurpali9fLlVVVRIKhaS3t9fppgEA4FkEMiZpaGiQ1tbWhNtaW1ulvr7eoRYBAOB9BDImiEQi0tLSIoODgwm3Dw4OSktLi3R2djrUMgAAvI1AxgTd3d1p7+/q6rKpJQAAFBYCGRPMmTMn7f1z5861qSUAABQWAhkTVFVVSTAYFL/fn3C73++XYDDI6iUAACxCIGOScDgsgUAg4bZAICDhcNihFgEA4H3UkTFJWVmZNDc3S2dnp3R1dVFHBgAAGxDImKyyspIABgAAmzC1BAAAXItABgAAuBaBDAAAcC0CGQAA4FoEMgAAwLUIZAAAgGsRyAAAANcikAEAAK5FIAMAAFyLQAYAALgWgQwAAHAt9lrKUSQSke7ubjaHBADAQYzIZCkajUooFJLq6mpZvny5VFVVSSgUkt7eXqebBgBAwSGQyVJDQ4O0trYm3Nba2ir19fUOtQgAgMJFIJOFSCQiLS0tMjg4mHD74OCgtLS0SGdnp0MtAwCgMBHIZKG7uzvt/V1dXTa1BAAAiBDIZGXOnDlp7587d65NLQEAACIEMlmpqqqSYDAofr8/4Xa/3y/BYJDVSwAA2IxAJkvhcFgCgUDCbYFAQMLhsEMtAgCgcFFHJktlZWXS3NwsnZ2d0tXVRR0ZAAAcRCCTo8rKSgIYAAAcxtQSAABwLQIZAADgWgQyAADAtQhkAACAaxHIAAAA1yKQAQAArkUgAwAAXItABgAAuBaBDAAAcC0CGQAA4Fqe36JAKSUiIv39/Q63BAAAGBW7bseu46l4PpA5duyYiIjMmDHD4ZYAAIBsHTt2TEpLS1Pe71OZQh2XGxoakjfeeEPGjx8vPp/PlGP29/fLjBkz5ODBg1JSUmLKMb2OPssO/ZUd+is79Ff26LPsmNFfSik5duyYTJs2TYqKUmfCeH5EpqioSKZPn27JsUtKSjihs0SfZYf+yg79lR36K3v0WXby7a90IzExJPsCAADXIpABAACuRSCTg+LiYrnvvvukuLjY6aa4Bn2WHforO/RXduiv7NFn2bGzvzyf7AsAALyLERkAAOBaBDIAAMC1CGQAAIBrEcgAAADXIpDJwY9+9CP5h3/4BznvvPNk4cKFsmvXLqebpIX7779ffD5fws9FF10Uv//kyZOyevVqKS8vl3HjxsnHP/5xefPNNx1ssb22b98uH/3oR2XatGni8/nkN7/5TcL9Sim59957ZerUqTJ27FgJBALS2dmZ8JhoNCo33nijlJSUyIQJE+Tmm2+W48eP2/gq7JOpvz7zmc+MOt9CoVDCYwqpv9atWycLFiyQ8ePHy+TJk+W6666Tjo6OhMcY+Rs8cOCAfPjDH5bzzz9fJk+eLF/5ylfk9OnTdr4UWxjpr2uuuWbUOXbLLbckPKZQ+ktE5MEHH5RLL700XuSupqZGmpqa4vc7dX4RyGRpw4YN8sUvflHuu+8+2bNnj1x22WUSDAblL3/5i9NN08L73vc+OXz4cPxnx44d8fvuuOMOefLJJ+WJJ56QtrY2eeONN+SGG25wsLX2OnHihFx22WXyox/9KOn93/nOd+QHP/iB/PjHP5bnn39e3vWud0kwGJSTJ0/GH3PjjTfKq6++Klu2bJGnnnpKtm/fLp/73Ofsegm2ytRfIiKhUCjhfAuHwwn3F1J/tbW1yerVq2Xnzp2yZcsWOXXqlFx77bVy4sSJ+GMy/Q0ODg7Khz/8YXnnnXfkueeek0cffVTWr18v9957rxMvyVJG+ktEZNWqVQnn2He+8534fYXUXyIi06dPl29961vy4osvygsvvCBLliyRFStWyKuvvioiDp5fClm54oor1OrVq+P/HxwcVNOmTVPr1q1zsFV6uO+++9Rll12W9L6jR4+qc889Vz3xxBPx2/7whz8oEVHt7e02tVAfIqI2bdoU///Q0JCqqKhQ3/3ud+O3HT16VBUXF6twOKyUUuq1115TIqJ2794df0xTU5Py+Xzq0KFDtrXdCSP7SymlVq5cqVasWJHydwq5v5RS6i9/+YsSEdXW1qaUMvY32NjYqIqKitSRI0fij3nwwQdVSUmJGhgYsPcF2Gxkfyml1NVXX62+8IUvpPydQu6vmLKyMvXwww87en4xIpOFd955R1588UUJBALx24qKiiQQCEh7e7uDLdNHZ2enTJs2TS688EK58cYb5cCBAyIi8uKLL8qpU6cS+u6iiy6SmTNn0ncisn//fjly5EhC/5SWlsrChQvj/dPe3i4TJkyQD3zgA/HHBAIBKSoqkueff972Nutg27ZtMnnyZKmurpZbb71Venp64vcVen/19fWJiMgFF1wgIsb+Btvb2+WSSy6RKVOmxB8TDAalv78//q3bq0b2V8x///d/y8SJE+Xiiy+WO++8U95+++34fYXcX4ODg/L444/LiRMnpKamxtHzy/ObRprprbfeksHBwYQ3QURkypQp8n//938OtUofCxculPXr10t1dbUcPnxY1q5dK4sWLZJXXnlFjhw5ImPGjJEJEyYk/M6UKVPkyJEjzjRYI7E+SHZuxe47cuSITJ48OeH+c845Ry644IKC7MNQKCQ33HCDzJ49W7q7u+Wuu+6Suro6aW9vF7/fX9D9NTQ0JLfffrtcddVVcvHFF4uIGPobPHLkSNJzMHafVyXrLxGRhoYGmTVrlkybNk327dsnX/va16Sjo0M2btwoIoXZXy+//LLU1NTIyZMnZdy4cbJp0yZ573vfK3v37nXs/CKQgWnq6uri/7700ktl4cKFMmvWLPnVr34lY8eOdbBl8KJPf/rT8X9fcsklcumll8qcOXNk27ZtsnTpUgdb5rzVq1fLK6+8kpCjhtRS9dfwfKpLLrlEpk6dKkuXLpXu7m6ZM2eO3c3UQnV1tezdu1f6+vrk17/+taxcuVLa2tocbRNTS1mYOHGi+P3+UVnYb775plRUVDjUKn1NmDBBqqqqpKurSyoqKuSdd96Ro0ePJjyGvjsj1gfpzq2KiopRSeWnT5+WaDRKH4rIhRdeKBMnTpSuri4RKdz+WrNmjTz11FPyzDPPyPTp0+O3G/kbrKioSHoOxu7zolT9lczChQtFRBLOsULrrzFjxsjcuXPl8ssvl3Xr1slll10m3//+9x09vwhksjBmzBi5/PLL5emnn47fNjQ0JE8//bTU1NQ42DI9HT9+XLq7u2Xq1Kly+eWXy7nnnpvQdx0dHXLgwAH6TkRmz54tFRUVCf3T398vzz//fLx/ampq5OjRo/Liiy/GH7N161YZGhqKf8AWsj//+c/S09MjU6dOFZHC6y+llKxZs0Y2bdokW7duldmzZyfcb+RvsKamRl5++eWEAHDLli1SUlIi733ve+15ITbJ1F/J7N27V0Qk4RwrlP5KZWhoSAYGBpw9v3JOEy5Qjz/+uCouLlbr169Xr732mvrc5z6nJkyYkJCFXai+9KUvqW3btqn9+/erZ599VgUCATVx4kT1l7/8RSml1C233KJmzpyptm7dql544QVVU1OjampqHG61fY4dO6Zeeukl9dJLLykRUQ888IB66aWX1J/+9CellFLf+ta31IQJE9TmzZvVvn371IoVK9Ts2bPV3/72t/gxQqGQmjdvnnr++efVjh07VGVlpaqvr3fqJVkqXX8dO3ZMffnLX1bt7e1q//79qrW1Vc2fP19VVlaqkydPxo9RSP116623qtLSUrVt2zZ1+PDh+M/bb78df0ymv8HTp0+riy++WF177bVq7969qrm5WU2aNEndeeedTrwkS2Xqr66uLvX1r39dvfDCC2r//v1q8+bN6sILL1SLFy+OH6OQ+ksppf71X/9VtbW1qf3796t9+/apf/3Xf1U+n0/97ne/U0o5d34RyOTghz/8oZo5c6YaM2aMuuKKK9TOnTudbpIWPvWpT6mpU6eqMWPGqHe/+93qU5/6lOrq6orf/7e//U19/vOfV2VlZer8889X119/vTp8+LCDLbbXM888o0Rk1M/KlSuVUmeWYN9zzz1qypQpqri4WC1dulR1dHQkHKOnp0fV19ercePGqZKSEnXTTTepY8eOOfBqrJeuv95++2117bXXqkmTJqlzzz1XzZo1S61atWrUF4pC6q9kfSUi6mc/+1n8MUb+Bl9//XVVV1enxo4dqyZOnKi+9KUvqVOnTtn8aqyXqb8OHDigFi9erC644AJVXFys5s6dq77yla+ovr6+hOMUSn8ppdRnP/tZNWvWLDVmzBg1adIktXTp0ngQo5Rz55dPKaVyH88BAABwDjkyAADAtQhkAACAaxHIAAAA1yKQAQAArkUgAwAAXItABgAAuBaBDAAAcC0CGQAA4FoEMgAAwLUIZABoy+fzpf356Ec/Kj6fT3bu3Jn095cuXSo33HCDza0GYKdznG4AAKRy+PDh+L83bNgg9957r3R0dMRvGzdunNTW1spPf/pT+eAHP5jwu6+//ro888wz8uSTT9rWXgD2Y0QGgLYqKiriP6WlpeLz+RJuGzdunNx8882yYcMGefvttxN+d/369TJ16lQJhUIOtR6AHQhkALjajTfeKAMDA/LrX/86fptSSh599FH5zGc+I36/38HWAbAagQwAV7vgggvk+uuvl5/+9Kfx25555hl5/fXX5aabbnKwZQDsQCADwPU++9nPyvbt26W7u1tERH7605/K1VdfLXPnznW4ZQCsRiADwPWWLl0qM2fOlPXr10t/f79s3LhRbr75ZqebBcAGrFoC4HpFRUVy0003ySOPPCLvfve7ZcyYMfKJT3zC6WYBsAEjMgA84aabbpJDhw7JXXfdJfX19TJ27FinmwTABgQyADxh5syZEggEpLe3Vz772c863RwANvEppZTTjQAAAMgFIzIAAMC1CGQAAIBrEcgAAADXIpABAACuRSADAABci0AGAAC4FoEMAABwLQIZAADgWgQyAADAtQhkAACAaxHIAAAA1/r/MCbvJYO2x1EAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "adv.plot(x='TV', y='Sales', kind='scatter', c='black')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The fields `TV` and `Sales` have different units. Remember that in the week 2 assignment to make gradient descent algorithm efficient, you needed to normalize each of them: subtract the mean value of the array from each of the elements in the array and divide them by the standard deviation.\n", + "\n", + "Column-wise normalization of the dataset can be done for all of the fields at once and is implemented in the following code:" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "adv_norm = (adv - np.mean(adv))/np.std(adv)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Plotting the data, you can see that it looks similar after normalization, but the values on the axes have changed:" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjUAAAGwCAYAAABRgJRuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAA9hAAAPYQGoP6dpAABCnklEQVR4nO3deXRb5Zn48UdWiBMWO04CCQFDSSKbmZYCZgkJDgRQkJwemkxnmGJzOglkKHCgU4ZOO8ABcmgpmRYOpaVMWQoEhtakMOzjyGCWZAKBBEKAsvjaboAQmgxYxg6bAfv9/cFPwrK1XF3d5b1X3885OodouX716qL76H2f93lDSiklAAAAPlfhdQMAAADsQFADAAACgaAGAAAEAkENAAAIBIIaAAAQCAQ1AAAgEAhqAABAIIzzugFuGh4elnfffVf22msvCYVCXjcHAACYoJSSXbt2yYwZM6SiIvd4TFkFNe+++67U1tZ63QwAAGDBtm3bZP/998/5eFkFNXvttZeIfNkpVVVVHrcGAACYMTAwILW1tenreC5lFdSkppyqqqoIagAA8JlCqSMkCgMAgEAgqAEAAIFAUAMAAAKBoAYAAAQCQQ0AAAgEghoAABAIBDUAACAQCGoAAEAgENQAAIBAIKgBAACBUFbbJAAAEHSGYUhPT4/Mnj1bIpGI181xFSM1AAAEQDKZlHg8LvX19bJo0SKpq6uTeDwufX19XjfNNQQ1AAAEQEtLi3R0dGTc19HRIc3NzR61yH0ENQAA+JxhGNLe3i5DQ0MZ9w8NDUl7e7t0dXV51DJ3EdQAAOBzPT09eR/v7u52qSXeIqgBAMDnZs2alffx2bNnu9QSbxHUAADgc3V1dRKLxSQcDmfcHw6HJRaLlc0qKIIaAAACoLW1VaLRaMZ90WhUWltbPWqR+6hTAwBAANTU1EgikZCuri7p7u4uyzo1BDUAAARIJBIpu2AmheknAAAQCIzUAACADH7daoGRGgAAICL+32qBoAYAAIiI/7daIKgBAACB2GqBoAYAAARiqwWCGgAAEIitFghqAABAILZaIKgBAAAi4v+tFqhTAwAARMT/Wy0Q1AAAUKZyFdnz61YLTD8BAMqeYRiyZs0abZYtO92eYors6dY3+RDUAADKlm4VdN1qj5kie7r1jRkhpZTyuhFuGRgYkOrqaunv75eqqiqvmwMA8Fg8HpeOjo6MgnPhcFii0agkEolAtscwDKmvr8/7eCQS0apvzF6/GakBAJQl3SroutUeM0X2dOsbswhqAABlSbcKum61x0yRPd36xiyCGgBAWdKtgq5b7TFTZE+3vjGLoAYAUJZ0q6DrZnsKFdnTrW/MIqgBAJQt3SroutWeVJE9wzCkra1NDMOQRCIhNTU1rrfFTqx+AgCUPd0q6OrUHh3aYvb6TVADAPBErmq2wGgs6QYAaMmPRd3gDwQ1AABXmalmC1hBUAMAcI1fi7rBH3wT1KxcuVKOOuoo2WuvvWSfffaRJUuWSGdnp9fNAgAUwa9F3YLCT5tTWuGboGbt2rVy3nnnybPPPiuPPfaYfP7553LyySfLRx995HXTAAAm+bWom9+VSx6Tb1c/vffee7LPPvvI2rVr5bjjjjP1GlY/AYD3dNoosVz4vc8Dv/qpv79fREQmT56c8zmDg4MyMDCQcQMAeMuPRd3s5uY0UDnlMfkyqBkeHpYLLrhAjj32WPnGN76R83krV66U6urq9K22ttbFVgIAsjFTzTaovJgGKqc8Jl9OP5177rmyZs0aWb9+vey///45nzc4OCiDg4Ppfw8MDEhtbS3TTwAAT3gxDWQYhtTX1+d9XPfih4Gdfjr//PPlkUcekSeffDJvQCMiUllZKVVVVRk3AAC84NU0kF83p7TCN0GNUkrOP/98uf/+++WJJ56Qgw46yOsmAQBgmpfTQOWSxzTO6waYdd5558kf//hHefDBB2WvvfaSHTt2iIhIdXW1TJw40ePWAUD5Yg8nc7xczp7KY3Jyc0odzgPf5NSEQqGs999+++2ybNkyU8dgSTcA2CeZTEpLS4u0t7en74vFYtLa2loWSb9W+H1pdTZunAfs0p0FQQ0AfKXUX9ZBvEA7ra+vT5qbmwMVCLpxHhDUZEFQAwD2/LIOwooaLzk5DeQmt86DwK5+AgCUxo5dssup9okTIpGINDU1+TqgEdHvPCCoAYAyYteyYvZwgoh+5wFBDQCUEbt+WZdT7RPkptt5QFADAGXEzl/W5VL7BPnpdB6QKAwAZcbu1Sp+S3rVoZ5KEDl5HrD6KQuCGgAI5rJiM6ir418ENVkQ1ADAV/w2wlIq6ur4F0FNFgQ1AFCeqKvjb9SpAQDg/9OtngqcQVADAAg83eqpwBkENQCAwNOtngqcQVADACgLOtVTgTPGed0AAIC+7KjpoktdmJqaGkkkEmW36qucENQAAMawo6aLrnVhIpEIwUxAMf0EABjDjp287TgGUAzq1ABAwBU7/WNHTRfqwsBO1KkBgDKXTCYlHo9LfX29LFq0SOrq6iQej0tfX1/e19lR04W6MPACQQ0ABJTV6R87aroUOsbee+9d8Bg6MwxD1qxZI11dXV43BSMQ1ABAABmGIe3t7Rn7HImIDA0NSXt7e96LsR01XVLHyCYUCsmll15q4l3ox+roF9xBUAMAAVTq9I8dNV1+9rOfZb1fKVUwsCrEq5ESkp/1RlADAAFU6hRSqqaLYRjS1tYmhmFIIpEoain2+++/n/dxK3k1Xo6UlDL6BXcQ1ABAANm1LUAkEpGmpiZLK5Wc2G/Jy5ESkp/1R1ADAAHl9bYAdu+35PVICZti6o+gBgACyo4ppFLZGVg5MVJSTG4Om2Lqj20SACDg3N4WYHSxP7v2W7JzpMTqFg6tra3S3Nyc8To2xdQHFYUBALZwY6+neDwuHR0dGVNQ4XBYotGoJBIJ147DppjuMnv9JqgBANjCroAjn76+vjEjJcUGTmzh4D9mr99MPwEASpZK4h1tZBJvqYFCalrr+uuvFxGxPFJiJjeHoMafCGoAACVzMlCwe1qLVUylK3aTVLew+gkAUDInAwW7a9Owisk63beJIKgBAJTMqUDBqdo0Xtfw8Svdt4kgqAEADflxF2gnAgWnqvjqUMPHb7wufmgGOTUAoBE3lkU7JRUo2Lnc2en8F7dr+PiZHxKsGakBAI3oPrxvRin7RY1G/os+/JBgTVADAJrww/C+F8h/0YMfAkyCGgDQBLtAZ0f+iz50DzDJqQEATfhheN9L5L9k52bNGCfypuzESA0AaCLX8H5FRYU0NDR41Kpg8ONqskK8rBkTiURk1qxZ0t3drVWfEtQAgEayDe8PDw/L5s2btSt05ge6F4srhVdJ5Tr3KRtaAoCGurq65LTTTpOXXnrJ0Q0ig86NTTa94OWmnF70qdnrNyM1AKAhpZRs3ryZlVAlCPJqMq+SynXvU4IaANAQK6FKF+Q+9CqpXPc+JagBAA2xEqp0Qe5Dr2rG6N6nBDUAoCE/FDrTXdD70IuaMbr3KUENAGhK90JnfmBXHzq1JLyU43pVlFDn85LVTwCgOV0LnZXKzaJxVvvQqQ1G/bxxaYqb56XZ6zdBDQDAVX66oDu1fDmoS82dQlCTBUENAHjPLxd0p2rBeFljxq+oUwMA0I7udU5Gcmr5su7Lov2MoAYA4Bo/XdCtLl8ulPyr+7JoPyOoAQC4xo4LulubUxa7fNnsnki6L4v2M4IaAIBrSrmge7GRYjHLl4vZYFLnZdF+RqIwAFjg5nLkoOnr65Nvf/vbsn79+vR9ZlY/eZlgXGj5stXk36Au17eb2ev3OBfbBAC+56flyDpK9d/IgKaxsbFg/6USjEcbmWDsZFAQiUTyHt9MrlC21xc6LorD9BMAFKGYKQZkMgxDFi5cOKb/NmzYULD/dE8wJvlXDwQ1AGCSn5Yj62RkLszmzZst9Z/uQQPJv3ogqAEAk3QfLdBVttGtbPL1nx+CBpJ/veeroGbdunVyyimnyIwZMyQUCskDDzzgdZMA2MStZbql0H20YDQd+jTX6FY2hfpP96DBqw0m8RVfBTUfffSRHHrooXLDDTd43RQANvFima5VfhgtENGrTwuNbomY7z+/BA2RSESampq0OR/KivIpEVH3339/Ua/p7+9XIqL6+/udaRSAosViMRUOh5WIpG/hcFjFYjGvm5ZVMplUsVgso72xWEwlk0mvm5amU592dnZmtCPbTbf+g37MXr99W6cmFArJ/fffL0uWLMn5nMHBQRkcHEz/e2BgQGpra6lTA2jCzxv75aov4nX9Grv61M73ka2+TEVFhRx22GFy9913a/sZQx9saCkiK1eulOrq6vSttrbW6yYBGMHPibejpxh0mfIptU+deB/ZcmFSS7vdCGjsyi3SIUcJBbgybuQAMTH99Omnn6r+/v70bdu2bUw/ARopNDVhGIbXTTRNlymfUvvUyfdhGIZqa2tz7XPt7e21ZarQruPAOrPTT4EOakYjpwbQjy7BQCl0C86s9qlu76NUdp1bQThH/c7s9TvQ008A9Kf7Ml0zdJtGs9qnur2PUthVKJGCi/7iq72fPvzww4z/qbZu3SpbtmyRyZMnywEHHOBhywBYlVqm6+eN/XSrX2O1T3V7H6WwuheTU8eBO3wV1Dz//PNywgknpP994YUXiojI0qVLZdWqVR61CoAd7N7Yz81VSKn6Nbl2kPbqoldsn+r6PqywK0ALUqBXDnw1/bRgwQJRX+YBZdwIaACkeLUKKQjTaCLBeR92FUr0S8FFfMm3dWqsMLvOHYB/ZauJkhppSCQSjv99P0+jjVTM+/C6Nk8ufX190tzcLO3t7en7YrGYtLa2FlWF2K7jwDqz12+CGgCB4edifl4oNRhJJpPS0tKi/cXerkAzKAGrH5m9fvsqpwYA8iGp0xy7gpFsu293dHRIc3OzK6NiZtmVr2V33hfs56ucGgDIh6ROc/IFI2ax1Bk6IqgBEBgkdRZmVzASpJo2CA6CGgCO8WKvHCdW7wRpzx+7ghGro2JB6kvoh6AGgO283NwxVXjOMAxpa2sTwzAkkUhYSlzVZZNKO9k1RVfsqFgQ+xIacna3Br2w9xPgjqDslROU9zGaXe8rmUya3ugxqH0Jd5i9frOkG4CtgrKsOijvIxu7664UWuoc5L6EO1jSDcATQVlWHZT3kY3d+20VWuoc5L6EXghqANgqKMuqg/I+8nGr7ko59KWuVZXLDYnCAGwVlGXVQXkfOghyX5IArReCGgC2C8qmiEF5HzoIal/aUcgQ9iFRGIBjgrJXTlDeh9uyTckEqS+dSIBmGis7EoUBeM6LvXKcuCj4Zc8fXS6I+faW8ktfmmFnArRfNgfVHdNPAAKhnHMbdHvv5TIlY2cCdLn0mdOYfgLgOidGFOLxuHR0dGTsaRQOhyUajWq1Y7QTdHrv5VaTxo6+L7c+s8Ls9ZuRGgCucWpEwW87Rtu5/5Fu773cNrq0IwG63PrMSQQ1AFzj1BC7Xy4KTgR1ur33QlMye++9t0stcYcde42VQx0ftxDUAHCFkyMKfrkoOBHU6fTek8mk/Mu//EvOx0OhkFx66aWutcdNkUhEmpqaLE0TBbmOj9sIagC4wskRBScvCnZNFTkV1Ol0QcwWtI2klNJyOlAHQa3j4zaCGgCucHpEwe6Lgt1TRU4GdTpcEHMFbdnoMh2oEzumscDqJyDQdKlbkuLGKh27irvZ3VY3Vrh4WdhuzZo1smjRIlPPZTUPimX6+q3KSH9/vxIR1d/f73VTAEf19vaqWCymRCR9i8ViKplMetquZDKpZbtG6+zszGjj6JthGJaOG4vFVDgczjhWOBxWsVjM5neQX2dnp2pra8v6PvI9VuiY+frMq/eKYDB7/SaoAQJIl4tnLoZhWLpwuqWtrS3vxbmtrc3Scb0O6vIFu3YEwtnOO90DWPiD2es3009AwFDIq3RO96FX00T5ptREpOTptr6+Pmlubs4o9d/Y2Cg/+MEP5PDDD+e8g2Vmr98ENUDAFMptaGtrk6amJhdb5E86Vem1Q6FArdBriwlIgrRpJfRARWGgTOlUt8TPdFhRZKdCq6/yKXa1Uik1W4BSsEs3EDCpuiW5Rhm40JiTWmIblFGHQsFuPgTC8AtGaoAACtoog5eCMupQqEifLgX8gFKQUwMEWFBGGWCPbIm8sVgsHezmeowCcMGgW92qYpAonAVBDcqRn7/I4Ix8wS6BcPAkk0lpaWnxdcBKUJMFQQ3KSRC+yACULggr+Vj9BJQ5J3aEHs2uzR7hT379/P3abiuK2Ug1CP1CUAMEkFM7QqfYvdkj/MWvn79f210KMxupBqlfCGqAAHJyR2gRd0aBoC+/fv4tLS3y2GOPZdznh3aXwkzdKr9+ntkQ1AAB5GQBPqdHgaA3v37+GzdulPb2dhkeHs64X/d2l6rQUn6llC8/z1wIaoAAKvRFVsqqFqdHgWCOV/kPfv38zz333LyP69puO+SrW+XXzzMXKgoDAdXa2jqm7ogdBfjc3IaB5ehfGtkPU6ZM8XRVm5fbcFg9HwzDkM2bN+d9TpCrJuerjh24bVWc3CpcN2a3LgeCxDAM1dbWpgzDsO2YsVhMhcNhJSLpWzgcVrFYzJbj9/b2qlgslnH8WCymksmkLcf3i2z9MGXKFEf73gynP//RSj0f2traMl47+tbQ0OBIu/3C7c/TCrPXb4IaAEVLJpOOBh1++JJ1Q7Z+yHezM3DNx+nPf7RSz4fOzs68/bZp0yZH2u0Xbn+eVpi9flN8D/BIEKZWnKg+axiG1NfX533cr/1VjEL9kE1bW5s0NTU51KKx3Kg+bNf5EIQCdE7TuZq02eu3LTk1Q0ND8sorr8iBBx5IpVKgAD9X+h0diKVudjKTuKjbF64TCvVDNiPzH9wImp34/Eez63xwKscsSNz4PJ1mafXTBRdcILfeequIfBnQHH/88dLQ0CC1tbXy1FNP2dk+IHD8WBPCzeJcgUtctKhQP4w0clVbkAqpidh3PqSSZQ3DkLa2NjEMQxKJhPY/JFAkK3Nb++23X3oO8v7771czZsxQnZ2d6tJLL1Xz5s2zckhXkFMDrxWa23crJ6JYbue4kFPzpWz9UFFRoaZMmZIz/yGIfRfE94TimL1+Wxqpef/992X69Oki8uUc7qmnnip1dXVy5plnyiuvvGLlkEBZ8GNNCC+KrWWrq3HooYfKlVdeafvf0lm2fli4cKF0dXVlHXHwa2G8QvLVWUHpgrDnU4qloGbatGny2muvydDQkCQSCVm4cKGIiHz88cdjin0B+Iofp1a8CMRSUwUbN26UhoYGERHZvHmzHHXUUb6dSrFy4cg3ZRKJRGTWrFnS3d2dPqYfg2YzmDpyRtCmKkXE2vTTihUrVHV1tTr44IPVAQccoD799FOllFK33nqrOuaYY6wc0hVMP0EHfhtK93LKzG99lY0TNXdyHXPjxo2+nN40o7Oz0/Z6S+VmdB/66f8vx+vU3HPPPeraa69V27ZtS9+3atUq9cADD1g9pOMIaqADP9SEGM2LLz+/5h+N5kTf5Tumny5UZlCIsXTZ+rCxsdFX/3+5Vnzvk08+KfUQriGoQS5e/Ap0otKvU7wIxApVgW1ra3Psb9vFicDMTCG5IAUBQQvSvJAr4dxP/3+ZvX5bqlMzNDQkV111ldx4442yc+dOMQxDZs6cKZdddpl87Wtfk+XLl1s5LOC6QjVjnKz14aeaEPn2jnGK2fwjnYsYOlFzZ8uWLXkff++991z/rJySSnwebWTis1/fm1ty9eHo3cpH0zG/zxQrEdMVV1yhZs6cqe666y41ceJE1dPTo5RS6u677yanBr6S61fgCSecEKhfu36V71e629MSVkbznBip8du0QSmCMFrntUJ9OHrERtdRMEenn2bNmqU6OjqUUkrtueee6aDm9ddfV5MmTbJySFcQ1GCkQhcchry9l2/ay61piVKDJzvbWeicnT9/ftHH1FlQ8qq8ZOac8cOPN0eDmgkTJqg333xTKZUZ1Lz66qtqjz32sHJIVxDUYKRCv2D4ItXH6PwjNy92VoOS1MiOnTkuhc7Z1atXW32b2iKnpnSF+tAP+X2OFt/727/9W/nf//3fMfffe++9cvjhh1s5JOC6YsrQj+TXWh9+FolEpKmpKZ0/4VY9FivF7EbX/jjqqKNERGTTpk0l11gpdM4G8fuXwnulK9SHo///8jNLicKXX365LF26VLZv3y7Dw8Ny3333SWdnp9x5553yyCOP2N1GwBF1dXUSi8XG7NxbUVGRN4nOtwl0FuiahOtWEUMrib659vYSkZJ3g851zqZ2m9bpM7KLF0nqQVNWfWh1KGjdunUqGo2qvffeW02cOFEde+yxqr293erhXMH0E0bLlbNx4oknlvWQtx9qg7gxLVHsNJcb02J+rHMElMq1OjV+QlCDXEbPKZf7hcMPeQxufUbF9EWhnJebb77ZttwFP+RBAHYxe/0OKaWUvWM/zrrhhhvk6quvlh07dsihhx4q119/vRx99NGmXjswMCDV1dXS398vVVVVDrcUQaDjcK3TU0KGYUh9fX3ex3XpCxHnP6O+vj5pbm7OWctopEJ9N1KuYwAYy+z123RQU1NTI6FQyNQfTyaT5lpZpNWrV8s//dM/yY033ihz5syR6667Tu655x7p7OyUffbZp+DrCWrgZ4UKBdplzZo1smjRopyPt7W1SVNTk21/zy/MBk/xeHxMzksoFJLRX7WpPJhS82yAcmB7UHPHHXeY/uNLly41/dxizJkzR4466ij57W9/KyJfVkSsra2VH/zgB3LRRRcVfD1BDfws28XSiQuj30ZqdJNtZCcf+hMozPagxmufffaZ7L777nLvvffKkiVL0vcvXbpUPvjgA3nwwQfHvGZwcFAGBwfT/x4YGJDa2lqCGviO24GGWwFUkKVGdrZv3y5nnXVWzueV68gXUAyzQY2lOjUjffrppzIwMJBxc8L7778vQ0NDMm3atIz7p02bJjt27Mj6mpUrV0p1dXX6Vltb60jbAKe5VZclhdogpUvV/jjuuOPyPq+cSgQATrMU1Hz00Udy/vnnyz777CN77LGH1NTUZNx0cfHFF0t/f3/6tm3bNq+bBA8YhiFr1qzJWijNL9yqy5KSqmthGEbJBePKXaq2TDgczrg/HA5LLBZj6gmwkaWg5ic/+Yk88cQT8rvf/U4qKyvl97//vVxxxRUyY8YMufPOO+1uo4iITJ06VcLhsOzcuTPj/p07d8r06dOzvqayslKqqqoybigfoyu71tXVSTwel76+Pq+bVjSvLoxBqjTqJUa+AHdYCmoefvhh+c///E/5+7//exk3bpzMnz9fLr30UrnqqqvkD3/4g91tFBGR8ePHyxFHHCGPP/54+r7h4WF5/PHHZe7cuY78Tfhbrsquzc3NHrWoNFwY/YuRL8AdlrZJSCaTMnPmTBERqaqqSi/hbmxslHPPPde+1o1y4YUXytKlS+XII4+Uo48+Wq677jr56KOP5IwzznDsb8KfUnv2jDZyzx6/jT6US6lzXbdmsEMkEgnce3KamfMhyOcMimNppGbmzJmydetWERE5+OCD5U9/+pOIfDmCM2nSJNsaN9p3v/tdueaaa+Tyyy+Xww47TLZs2SKJRGJM8jDgdmKtiHu5O0GdEgrSdCFKZ+Z84JzBGFbKFV977bXq17/+tVJKqccee0xNmDBBVVZWqoqKCnXddddZOaQr2CahfLixB0+KH/ZJsqqzs9PRUvwjj++HrRngHjPng9VzxunzGvZzde+nN998U/33f/+3eumll+w4nGMIasqLWxfJIF6MnQ7Ush3frSAU+jPzo8TKD5cg/wAJOrPX76KmnzZs2CCPPPJIxn133nmnLFiwQM455xz57W9/m1HsDvCSG4m1qdydkUXqRDJzd/zI6STrbMfPx4npQujLzPSxlSnmoC0ewFhFBTU//elP5dVXX03/+5VXXpHly5dLNBqViy++WB5++GFZuXKl7Y0ErHBjxYkXuTtOczpQy3X8fChQV17M1GUqtnZTUH+AIFNRQc2WLVvkpJNOSv/77rvvljlz5sgtt9wi//qv/yq/+c1v0knDgC6cTKwt9MW6fft2331Z2h2ojU6gLnT8kcqtQJ3uhSLdap+ZukzF1m4K4g8QZFHMnFZlZaV6++230/8+9thj1ZVXXpn+99atW9Wee+5Z5EyZe8ipgROy5dSEQiHfztvblWSdK39h48aNpnNp/NRvpdA918OL9iWTyYJ/08xzUtxcPAD7OZIofMABB6i1a9cqpZQaHBxUEydOVB0dHenHX375ZVVTU2Ohue4gqIETsn2xjr75LXHYjuTnfMfI95hhGIFfmTJ69Y3uyeZets/M+WD2nNG9n3Xn5aoxR4Kac845R82dO1etW7dOXXjhhWrKlClqcHAw/fhdd92ljjzySGstdgFBDZxkGIa6+eabA/FrsJhfwNkU+lW8adMmrUcmnJJtxKOxsVHrcyZIIxylntdu0HG5uQ4jiY4ENe+9956aP3++CoVCaq+99lL33XdfxuMnnniiuuSSS4pvrUsIapCPHV8mbW1teS8AbW1tNrbYeVZHTcz2gx9HZUo5T7KNFFRUVGh9zgTtnFZKz/NOh8AhFx1GuBytU/PBBx+oL774Ysz9vb29GSM3uiGoQTZ2fpkE6VdtKYLYD6WeJ4X6RNe+CuJnqSMdAodsdPn8HalTk1JdXT0m41xEZPLkyTJ+/HgrhwQcYWa1hp21K7zaTVs3QeyHUs+TQqtvKioyv4516asgfJZ+WFWm63Jz360acyXE0gQjNeXD7K9qJ36F6DRv7+X8vE79UCo7zpNCxxidW6NTX/n1s9R5Smcknaf4/DZSQ1CDQDI7lOvkl4mX8/Y6fZnrmL9QLLvOk0Lnpe59pXv7RsvV342NjVq9D10Ch1x0mBojqMmCoKY8FPMFofuXiVU6fAkFiV3niV9HPPzIbA6TLv2v8/+zOpy3BDVZENSUh2J/Vev8ZWKF3wM1HZe0KmXveeK3EY8UOz4btz7fQt8Duv2/rkPgUIiX5y1BTRYENeWh2Iu6H75MiqHz/Hw+Ok2ZZRO086QYdnw2bn++xa420yXA9GvA6zSz1++QUkpJmRgYGJDq6mrp7++Xqqoqr5sDB8Xjceno6MhYTRAOhyUajUoikcj6mq6uLunu7pZwOCxDQ0Mye/ZsX6zsGM0wDKmvr8/7uI7vy8pn5oXUeeLX88MKOz4bLz7fbH8zl7a2NmlqanKkHfkYhiE9PT1ldT5ZYfr67UqIpQlGasqHlV/Vuo8UFMNvU2p+nzILMjdWfjn1+ZrZwsSrcyxI3zducLRODaC7mpoaSSQSYhiGtLW1iWEYkkgkpKamJudr7KxX47XW1laJRqMZ90WjUfnZz36Ws16Hl7U8fFcLo4zY8dm4/fmmzuX3338/43tg/vz52tTbCdL3jVZcCrK0wEiN/7iVVBjUkYLU/PzGjRtz/irU4RdjUPs/CPw0UlPoXNYlL8rp/tA12b4UJApnQVDjH25faP2aXGuW1R2zdWmjzkZeQPxyMSm2nU7v2m6lTaW00+tkXKe+b3T4geIUgposCGr8w+0LXJBHCqzuOeT2+9blV7RZ2S4gurfd6kXPjs8m1zF6enps+dz99P+wU2316w8DMwhqsiCo8U4xv8K8+HLy0xdisczW68j1i9Ht0Qevf0Wble0CovvFpNSLnh2fzehj5Nq5vKGhoai/47fRVrsDkCB/hylFUJMVQY37rPwy9OLLyW9fiLlkC0BKGanReT8iLxXTp7pcTHS86JnpR7PnnI7vLx+7RyaD8h2WC6ufoAUrGf6zZs3Ke8zZs2fb0jav/+Zopaw+SiaTEo/Hpb6+XhYtWiR1dXUSj8elr6+v4C7LuR6bMmWKbNiwIeN+Vmd8qdBqnpF0WblVqM1r1651qSVfMdOPZs85v+0mbmWFZj46fIdpwaUgSwuM1LirlF9OXswNezUfbUdyX6G25/tVmO2x0SM0xXx25SCIIzVWzjs32lRMP/otL8tu5NQw/QQHlTIcaveXk5m8kI0bN6qGhgbXvxBL/SIqJnjMlxMx8rGgD2XbISg5NV62uZigpphzzi95WXYLclBHUJMFQY277JjjLvXLycwoSLbnNDQ0qE2bNln6m8Wwo4+cCECczE/wy7LnQgpVq9XxYmK2wq5bn00xSex+P1/cFMSgjqAmC4Ia93k9HGrm73vZRjsCEr8sDw1qDY2RFxC/XExuueUWLUbizIzU6DjiBfcR1GRBUOM+L4dDzVzsvV4xYdffdyIws/uz8zrAxVe8Pu9HKjQlFoTAF6UjqMmCoMY7XvyCNTMKokPuiB0X+1ICkELTQVY+u9HH1Okiii/pEmRmO3fnz5+vVq9ezXmBNIKaLAhqrPNjHoSZC2kikfD8YmvniEgxAYgT00G5jrl69WrPg0dk0i2p1C9Td/CG2et3SCmlpEwMDAxIdXW19Pf3S1VVldfN8YVkMimLFy+W9evXp+9raGiQm266SY488kgPW2ZOPB6Xjo4OGRoaSt8XDofluOOOk/Hjx0t7e3vW14XDYYlGo5JIJNxqqnR1dUl3d7fMnj3blZoaufqmlPed65hz587NOIdGMwxDuzoi5cLt8w6wwuz1m6AGOSWTSamrq5Pe3t6sj8diMWltbbVcLMoNfX190tzcnBG8xGIx+fzzz2Xt2rUZF9+R/PDeSmEYhtTX1+d9vNgLXKFjzp8/X5555hlbgygA5cHs9Xuci22CzyxevDhnQCPyVaVPnS9IqaqdI3+NKqXyXnwfffRRWbhwoYutzM8wDOnp6bH1l3ShSq7d3d1F/61Cxzz//PNl9913zwgwo9GotLa2FvV3ACAXghpkZRhG3ukCEZGhoSFpb2+Xrq4u7YetI5FIuo1r1qzJ+9wvvviipL81OgixGpQkk0lpaWkZM8pkxwiSEyXVCx3z8MMPHxNg6n7elAMngmbAM86n9+iDRGHziimK5ackz87OTnXzzTc7khycLUl2ypQplhMxnV6d4sTxdVlRY4Yfk9/tFNS6QQgmVj9lQVBjnh/3tskn2xd4KBRy/II++mb2b7ixBNqJ1S+6rajJhov5l/wUgAIENVkQ1BQnFoupioqKoi7Quv76NRNwlLq3lNkg0ExQYrZ+jh397cRSWp2X53Ixp24Q/IegJguCmuIUs7eNzr9+C32B33LLLSV/iRczXTcyKLHa5o0bN2rb36VKTRHa8blkOzYXc2f2CwOcZPb6TaIwcsq2ckhEsiZ5trS0SEdHR8brdVkdVWhVzn777VdygmShJNnRCiXi1tXVSSwWy1lH5rLLLtO2v61KJpNy6qmnyhNPPJFx/4knnij33nuvLcvrnVj15UdOJIoDWnApyNICIzXO0P3Xr1vtszOnRqnc+SkbN27Uur+tisViY/KcRr5vs/JNyel+rrqJaTj4CdNPWRDU2K+3t1c1NDRoP5Ttxhd4tiCklNVPKaPzU4I4dWAmJ6lQwGF2CpSL+Zf8kNQNpBDUZEFQYz8zoxM6/Pp18wt8dBBid9JsEEcbzOQkFQrWzAYrXMwz6ZzUDaSw91MWbJNgr0Jl8Uc+T5c8hXyF3/xUhMyJfZu8ZOZcynceWdn2gSKAgH+YvX5XuNgmBEyhpMuU7u5uh1tiXiQSkaampoyLWDKZlHg8LvX19bJo0SKpq6uTeDwufX19HrY0v9bWVolGoxn3+XnLgVRidCgUyvp4LBbLG3iYSQAeLdu5AMDfCGpgmdkVP7qvpMi3cktXqZVphmFIW1ubGIYhiUTC1xtwtra2ygknnDDm/hNPPLFgsMZqHgAi7NKNEmWbBkmxezqkmD2VzE4lObFbdTGytdNP02BO6OrqkrVr14qIyPHHH2+6D4I2JQfgK6av3y7k92iDRGH75SvQZ1fyZTF7KhVbBNCrlUTZ2nniiSeqY445hgRWi0gABoKLROEsym2kxswv/lJHBbLtJN3Q0CA33XSTHHnkkZbbPlK+0aCU1C9yESnq17pXIzVm3pOISEVFhSxcuJCRhiKQAAwEDyM1WZTLSI2Z0Qq7tjVwuuZHsXsq5bvlWrLqdt0SK++J5bYAypnZ6zeJwgFkJvHVjuRYwzCkvb19zGjD0NCQtLe3S1dXl4XWZzK7wsqMXKuwsq0kOvTQQ+XKK6+07W+PZOU96bSCDAB0RVDjE4ZhyJo1awoGCmYCDbuCESvLaItV7J5K+Ywbl32rs9RKoo0bN0pDQ4OIiGzevFmOOuooR5Z2W3lPrN4BgMIIajRXbA0VM4GGXcGIG8toU/VLwuFw3ueFw2GJxWJ5n3vyySfn7bvLLrtMXnrppYz7nFjanXpPFRXm/vdrbGwkNwQAzHBpOkwLfsypKSbfo7e3VzU2NhbMzbCzzL5ueyrlW42Vr21ubz2QTCYLflap98nqHQDljr2fsvBbUFPshTbfPkyjL+Z2BSO67qnU3t5eVN95tbS7sbFRVVRUZP2b8+fPJ6ABAEWicCAUM02UK08mZd68eRlVWe0qs+9mZdvRZe3zlbkvtFR69BSbVxVpH3roIVm4cGHGfQ0NDbJp0yZZt26drysEA4DbsmdOQgvFXGgLBUAXX3xxxgUyFYzYVdMjEololfdRbJCSynPJVePGqfdm9+cAAOWMkRqN5UqSTSXFjrz4WR1pCOqmfsX0XYqXm0QG9XMAADf5Jqj5+c9/LvPmzZPdd99dJk2a5HVzXGP2QpvvIt7Y2Cjd3d221I3xk2KDlEJTaWaX1QMAvOGbbRJWrFghkyZNknfeeUduvfVW+eCDD4o+hp+3SXj00Ufl2Weflblz547JwUjp6+uT5ubmjC0LpkyZIr29vel/x2IxaW1t1T5Xw85NHUud2sm2FUSx/Vjum1QCQCkCu03C7bffrqqrq00999NPP1X9/f3p27Zt23y1+kkpa9sZpFYFzZ8/39Xy/3awY/uGzs7OnKuirChlpZhd21EAQDkL7JLuYoKaFStWZF0q66egxuoF1e26K3bRLYAotR/d3lcKAIKIJd3y5Yqf/v7+9G3btm1eN6kopWxn4MYWBnYrdfsGO/azGq2UfnRjbywAwFc8DWouuugiCYVCeW9vvPGG5eNXVlZKVVVVxs1PrF5Qk8mkXHXVVXlf6+VeQrkSbnUMIEqpX+PHwBIA/MzTOjU/+tGPZNmyZXmfM3PmTHcaoyGrF9SWlhbZsGFD1secrruST6GEW6cDCCvvuZT6NV4V9AOAsuXSdJhtismpGc1v2yQoVXxORqEcEC9L75t5LzrmEJWyFQQ5NQBQusDl1Lz99tuyZcsWefvtt2VoaEi2bNkiW7ZskQ8//NDrpjmq2ForxVYWdovZ6SGrBfCsFNszq5StILws6AcA5cY3dWqWLVsmd9xxx5j7n3zySVmwYIGpY/i5To3ZWiuGYUh9fX3ex72YelqzZo0sWrQo5+M333yz7L///un3Z6W2TLY6PbrU5WEbBACwzuz12zdBjR38HNSkmCniFo/Hc+aAJBIJt5qaoVCwNVKpgQgBBAAEi9nrt2+mn8pdMpmUeDwu9fX1smjRIqmrq5N4PC59fX1jnqvjlEeu6aFQKDTmuaUuw2YfJQAoT4zU+ISV0RfdRiyyTQ/l49VUGQBAL0w/ZeHXoEbXPBmrUsHW9u3b5ayzzsr5vLa2NmlqanKxZQAAHZm9fntapwbmOFWDxSuRSEQikYgYhpH3edRxAQAUg5waHwhqETcnl2EDAMoPQY0P5Lr4V1RUSGNjo2sX/1zbG5RCx6RmAIA/EdT4RLaL//DwsKxfvz7nKii7FLPyqlilFLYDAGAkEoV95rjjjpOnn35ahoeH0/c5XYNGx7o3AIDyweqnLPwe1HixCipoK68AAP5D8T2fMZOvYmYVlB1/x+6/CQCAGwhqPJIKLjZt2mQ6X6WYVVCjgxereTFBXXkFAAgepp9clkwmpaWlJaOqbigUkpEfQ758lUL5LdmOH4vF5PPPP5e1a9dayoshpwYA4CVyarLQIajJFiDkki1fpa+vT7797W/L+vXr0/eN3AAyVwCS7+8VyovRefdrAEDwUVFYQ4ZhmN73SGRspeDUKMzIgKaxsTEdXOQ6fqEAqlBF4tSya932kgIAYCSCGhcVSrodbXS+SktLi3R0dGTct2HDBmlubpZEIlH08XP9nVxS2xsAAKAjghoXFUq6TUnlq4wMIPKNwrS3t0tXV5fp4+f7OwAA+BWrn1yUa7uD0bJtE2BmabXZ4+f7OwAA+BVBjcuybXcQi8Vk06ZNebcJMLu0Otvxc7n++utJ9AUABAarn1xmGIb09PTIuHHj5Isvvigq6Xbq1KnS29s75v4pU6bI+++/n3FfV1eXtLa2yooVK3Ier62tTZqamop7AwAAuIyKwpoZXfzu5JNPll//+tcydepUU683DCNrQCMi0tvbK88//3zGfZFIRE477bS8x6RwHgAgSAhqXJJt5VJHR4c0Nzeben2hnJqzzz57zH25cmzC4bDEYjEShAEAgUJQ44LUyqXR9WJGrlwqpFBOzebNm7MeJ1uODQnCAIAgIqhxgR2bQtbV1UlDQ0PRx0kVzjMMI28iMgAAfkedGhfYtSnkjTfeKEcffbSl41A4DwAQdIzUuMCu3JajjjpKYrGYVFRkfmzkyAAAQFDjGrtyW1pbW2XhwoUlHwcAgKChTo3L7NoUks0lAQDlwuz1m6DGJ1JF+whiAADlhuJ7ATG6aF9dXZ3E43Hp6+vzumkAAGiFoEZzpRbtAwCgXBDUeMAwDFmzZk3Bont2FO0DAKBcENS4yMxU0siAx46ifQAAlAuK77lo8eLF8swzz2Tcl5pK+uMf/ygtLS3S3t6efqyxsTHv8diQEgCArzBS44JkMinz58+X9evXy/DwcMZjqamkxYsXj8md2bBhg0yZMoUNKQEAMIGgxgUtLS1jRmhGW79+fdbcmd7eXpk3b17G/RTbAwBgLKafHNbe3p4xpWTFxRdfLLfeeivF9gAAyIORGoekkoLj8Xje51VUVJjKnYlEItLU1ERAAwBADgQ1DslWXyabY489Vh566CGZMmVK1senTJlCIAMAgAkENQ7IVV9mpNQIzbp16+S9996T3t7erM/r7e2lHg0AACYQ1DigUH0ZEZGFCxfKQw89ZOr51KMBAKAwEoUdMGvWrLyPP/roo7Jw4ULTz6ceDQAAhTFS44C6ujqJxWI568uMDGjMPJ+cGgAACiOocUhra6tEo9GM+1L1ZbLt/ZTv+QAAoLCQUkp53Qi3DAwMSHV1tfT390tVVZUrf7OrqytdX2bKlCljtkKIxWLS2toqNTU1Y57PCA0AAOav3wQ1LorH49LR0ZGxKiocDks0GpVEIuF6ewAA8AOz12+mn1ySa5l3au8nlm0DAFAaghqXsGwbAABnEdS4hGXbAAA4i6DGJSzbBgDAWQQ1LmLZNgAAzqGisItqamokkUiwbBsAAAcQ1HggEokQzAAAYDOmnwAAQCAQ1AAAgEAgqAEAAIFAUAMAAAKBoAYAAAQCQQ0AAAgEXwQ1b775pixfvlwOOuggmThxosyaNUtWrFghn332mddNAwAAmvBFnZo33nhDhoeH5aabbpLZs2fLn//8ZznrrLPko48+kmuuucbr5gEAAA2ElFLK60ZYcfXVV8vvfvc7+ctf/pLzOYODgzI4OJj+98DAgNTW1kp/f79UVVW50UwAAFCigYEBqa6uLnj99sX0Uzb9/f0yefLkvM9ZuXKlVFdXp2+1tbUutQ4AALjNl0FNd3e3XH/99XL22Wfnfd7FF18s/f396du2bdtcaiEAAHCbp0HNRRddJKFQKO/tjTfeyHjN9u3bJR6Py6mnnipnnXVW3uNXVlZKVVVVxs1NhmHImjVrpKury9W/CwBAOfI0p+a9996T3t7evM+ZOXOmjB8/XkRE3n33XVmwYIEcc8wxsmrVKqmoKC4mMzsnV6pkMiktLS3S3t6evi8Wi0lra6vU1NQ49ncBAAgis9dv3yQKb9++XU444QQ54ogj5K677pJwOFz0MdwKauLxuHR0dMjQ0FD6vnA4LNFoVBKJhGN/FwCAIApUULN9+3ZZsGCBHHjggXLHHXdkBDTTp083fRw3ghrDMKS+vj7v45FIxJG/DQBAEJm9fvuiTs1jjz0m3d3d0t3dLfvvv3/GY7rFZD09PXkf7+7uJqgBAMABvlj9tGzZMlFKZb3pZtasWXkfnz17tkstAQCgvPgiqPGTuro6icViY3J+wuGwxGIxRmkAAHAIQY0DWltbJRqNZtwXjUaltbXVoxYBABB8vsip8ZuamhpJJBLS1dUl3d3dMnv2bEZoAABwGEGNgyKRCMEMAAAuYfoJAAAEAkENAAAIBIIaAAAQCAQ1AAAgEAhqAABAIBDUAACAQCCoAQAAgUBQAwAAAoGgBgAABAJBDQAACASCGgAAEAjs/WQDwzCkp6eHjSsBAPAQIzUlSCaTEo/Hpb6+XhYtWiR1dXUSj8elr6/P66YBAFB2CGpK0NLSIh0dHRn3dXR0SHNzs0ctAgCgfBHUWGQYhrS3t8vQ0FDG/UNDQ9Le3i5dXV0etQwAgPJEUGNRT09P3se7u7tdagkAABAhqLFs1qxZeR+fPXu2Sy0BAAAiBDWW1dXVSSwWk3A4nHF/OByWWCzGKigAAFxGUFOC1tZWiUajGfdFo1FpbW31qEUAAJQv6tSUoKamRhKJhHR1dUl3dzd1agAA8BBBjQ0ikQjBDAAAHmP6CQAABAJBDQAACASCGgAAEAgENQAAIBAIagAAQCAQ1AAAgEAgqAEAAIFAUAMAAAKBoAYAAAQCQQ0AAAiEstomQSklIiIDAwMetwQAAJiVum6nruO5lFVQs2vXLhERqa2t9bglAACgWLt27ZLq6uqcj4dUobAnQIaHh+Xdd9+VvfbaS0KhkNfNsWxgYEBqa2tl27ZtUlVV5XVzPEEf0Aci9EEK/UAfiAS7D5RSsmvXLpkxY4ZUVOTOnCmrkZqKigrZf//9vW6GbaqqqgJ34haLPqAPROiDFPqBPhAJbh/kG6FJIVEYAAAEAkENAAAIBIIaH6qsrJQVK1ZIZWWl103xDH1AH4jQByn0A30gQh+IlFmiMAAACC5GagAAQCAQ1AAAgEAgqAEAAIFAUAMAAAKBoMYHfv7zn8u8efNk9913l0mTJpl6zbJlyyQUCmXc4vG4sw11mJV+UErJ5ZdfLvvuu69MnDhRotGodHV1OdtQByWTSTn99NOlqqpKJk2aJMuXL5cPP/ww72sWLFgw5lw455xzXGpx6W644Qb52te+JhMmTJA5c+bIxo0b8z7/nnvukYMPPlgmTJgghxxyiLS1tbnUUmcV0w+rVq0a85lPmDDBxdbab926dXLKKafIjBkzJBQKyQMPPFDwNU899ZQ0NDRIZWWlzJ49W1atWuV4O51UbB889dRTY86DUCgkO3bscKfBHiCo8YHPPvtMTj31VDn33HOLel08Hpe//vWv6Vtra6tDLXSHlX745S9/Kb/5zW/kxhtvlOeee0722GMPicVi8umnnzrYUuecfvrp8uqrr8pjjz0mjzzyiKxbt06+//3vF3zdWWedlXEu/PKXv3ShtaVbvXq1XHjhhbJixQrZvHmzHHrooRKLxeT//u//sj7/mWeekebmZlm+fLm8+OKLsmTJElmyZIn8+c9/drnl9iq2H0S+rCo78jN/6623XGyx/T766CM59NBD5YYbbjD1/K1bt8q3vvUtOeGEE2TLli1ywQUXyD//8z9Le3u7wy11TrF9kNLZ2ZlxLuyzzz4OtVADCr5x++23q+rqalPPXbp0qVq8eLGj7fGK2X4YHh5W06dPV1dffXX6vg8++EBVVlaq1tZWB1vojNdee02JiNq0aVP6vjVr1qhQKKS2b9+e83XHH3+8+uEPf+hCC+139NFHq/POOy/976GhITVjxgy1cuXKrM//x3/8R/Wtb30r4745c+aos88+29F2Oq3Yfijmu8KPRETdf//9eZ/zk5/8RH3961/PuO+73/2uisViDrbMPWb64Mknn1Qiovr6+lxpkw4YqQmwp556SvbZZx+pr6+Xc889V3p7e71ukqu2bt0qO3bskGg0mr6vurpa5syZIxs2bPCwZdZs2LBBJk2aJEceeWT6vmg0KhUVFfLcc8/lfe0f/vAHmTp1qnzjG9+Qiy++WD7++GOnm1uyzz77TF544YWMz6+iokKi0WjOz2/Dhg0ZzxcRicVivvy8U6z0g4jIhx9+KAceeKDU1tbK4sWL5dVXX3WjudoI4rlg1WGHHSb77ruvLFy4UJ5++mmvm+OostrQspzE43H5zne+IwcddJD09PTIJZdcIk1NTbJhwwYJh8NeN88VqXnjadOmZdw/bdo0X84p79ixY8yw8bhx42Ty5Ml5309LS4sceOCBMmPGDHn55Zfl3//936Wzs1Puu+8+p5tckvfff1+Ghoayfn5vvPFG1tfs2LEjMJ93ipV+qK+vl9tuu02++c1vSn9/v1xzzTUyb948efXVVwO1qW8+uc6FgYEB+eSTT2TixIketcw9++67r9x4441y5JFHyuDgoPz+97+XBQsWyHPPPScNDQ1eN88RBDUeueiii+QXv/hF3ue8/vrrcvDBB1s6/mmnnZb+70MOOUS++c1vyqxZs+Spp56Sk046ydIxneB0P/iB2T6wamTOzSGHHCL77ruvnHTSSdLT0yOzZs2yfFzoa+7cuTJ37tz0v+fNmyd/8zd/IzfddJP87Gc/87BlcFN9fb3U19en/z1v3jzp6emRX/3qV/Jf//VfHrbMOQQ1HvnRj34ky5Yty/ucmTNn2vb3Zs6cKVOnTpXu7m6tghon+2H69OkiIrJz507Zd9990/fv3LlTDjvsMEvHdILZPpg+ffqYxNAvvvhCkslk+r2aMWfOHBER6e7u1jqomTp1qoTDYdm5c2fG/Tt37sz5fqdPn17U8/3ASj+Mtttuu8nhhx8u3d3dTjRRS7nOhaqqqrIYpcnl6KOPlvXr13vdDMcQ1Hhk7733lr333tu1v/fOO+9Ib29vxsVdB072w0EHHSTTp0+Xxx9/PB3EDAwMyHPPPVf0SjInme2DuXPnygcffCAvvPCCHHHEESIi8sQTT8jw8HA6UDFjy5YtIiLanQujjR8/Xo444gh5/PHHZcmSJSIiMjw8LI8//ricf/75WV8zd+5cefzxx+WCCy5I3/fYY49ljFr4jZV+GG1oaEheeeUVWbRokYMt1cvcuXPHLOf3+7lghy1btmj//35JvM5URmFvvfWWevHFF9UVV1yh9txzT/Xiiy+qF198Ue3atSv9nPr6enXfffcppZTatWuX+rd/+ze1YcMGtXXrVtXR0aEaGhpUJBJRn376qVdvo2TF9oNSSv3Hf/yHmjRpknrwwQfVyy+/rBYvXqwOOugg9cknn3jxFkoWj8fV4Ycfrp577jm1fv16FYlEVHNzc/rxd955R9XX16vnnntOKaVUd3e3+ulPf6qef/55tXXrVvXggw+qmTNnquOOO86rt1CUu+++W1VWVqpVq1ap1157TX3/+99XkyZNUjt27FBKKfW9731PXXTRRennP/3002rcuHHqmmuuUa+//rpasWKF2m233dQrr7zi1VuwRbH9cMUVV6j29nbV09OjXnjhBXXaaaepCRMmqFdffdWrt1CyXbt2pf+fFxF17bXXqhdffFG99dZbSimlLrroIvW9730v/fy//OUvavfdd1c//vGP1euvv65uuOEGFQ6HVSKR8OotlKzYPvjVr36lHnjgAdXV1aVeeeUV9cMf/lBVVFSojo4Or96C4whqfGDp0qVKRMbcnnzyyfRzRETdfvvtSimlPv74Y3XyyServffeW+22227qwAMPVGeddVb6C9Cviu0Hpb5c1n3ZZZepadOmqcrKSnXSSSepzs5O9xtvk97eXtXc3Kz23HNPVVVVpc4444yMoG7r1q0ZffL222+r4447Tk2ePFlVVlaq2bNnqx//+Meqv7/fo3dQvOuvv14dcMABavz48eroo49Wzz77bPqx448/Xi1dujTj+X/6059UXV2dGj9+vPr617+u/ud//sflFjujmH644IIL0s+dNm2aWrRokdq8ebMHrbZPanny6FvqfS9dulQdf/zxY15z2GGHqfHjx6uZM2dmfDf4UbF98Itf/ELNmjVLTZgwQU2ePFktWLBAPfHEE9403iUhpZRybVgIAADAIdSpAQAAgUBQAwAAAoGgBgAABAJBDQAACASCGgAAEAgENQAAIBAIagAAQCAQ1AAAgEAgqAEAAIFAUAPAF0KhUN7bKaecIqFQSJ599tmsrz/ppJPkO9/5jsutBuAmdukG4At//etf0/+9evVqufzyy6WzszN935577imNjY1y2223yTHHHJPx2jfffFOefPJJefjhh11rLwD3MVIDwBemT5+evlVXV0soFMq4b88995Tly5fL6tWr5eOPP8547apVq2TfffeVeDzuUesBuIGgBkBgnH766TI4OCj33ntv+j6llNxxxx2ybNkyCYfDHrYOgNMIagAExuTJk+Xv/u7v5Lbbbkvf9+STT8qbb74pZ5xxhoctA+AGghoAgXLmmWfKunXrpKenR0REbrvtNjn++ONl9uzZHrcMgNMIagAEykknnSQHHHCArFq1SgYGBuS+++6T5cuXe90sAC5g9ROAQKmoqJAzzjhDbr31Vtlvv/1k/Pjx8g//8A9eNwuACxipARA4Z5xxhmzfvl0uueQSaW5ulokTJ3rdJAAuIKgBEDgHHHCARKNR6evrkzPPPNPr5gBwSUgppbxuBAAAQKkYqQEAAIFAUAMAAAKBoAYAAAQCQQ0AAAgEghoAABAIBDUAACAQCGoAAEAgENQAAIBAIKgBAACBQFADAAACgaAGAAAEwv8Dc7IuYwH/bYEAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "adv_norm.plot(x='TV', y='Sales', kind='scatter', c='black')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Save the fields into variables `X_norm` and `Y_norm` and reshape them to row vectors:" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The shape of X_norm: (1, 200)\n", + "The shape of Y_norm: (1, 200)\n", + "I have m = 200 training examples!\n" + ] + } + ], + "source": [ + "X_norm = adv_norm['TV']\n", + "Y_norm = adv_norm['Sales']\n", + "\n", + "X_norm = np.array(X_norm).reshape((1, len(X_norm)))\n", + "Y_norm = np.array(Y_norm).reshape((1, len(Y_norm)))\n", + "\n", + "print ('The shape of X_norm: ' + str(X_norm.shape))\n", + "print ('The shape of Y_norm: ' + str(Y_norm.shape))\n", + "print ('I have m = %d training examples!' % (X_norm.shape[1]))" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "array([[ 0.96985227, -1.19737623, -1.51615499, 0.05204968, 0.3941822 ,\n", + " -1.61540845, -1.04557682, -0.31343659, -1.61657614, 0.61604287,\n", + " -0.94515567, 0.79002835, -1.4390876 , -0.57850171, 0.66625345,\n", + " 0.56466461, -0.92530498, 1.56887609, -0.90895735, 0.0030068 ,\n", + " 0.8332328 , 1.05509347, -1.5628625 , 0.94883389, -0.98952781,\n", + " 1.35285385, -0.04837147, 1.08662104, 1.18820988, -0.89260972,\n", + " 1.70316018, -0.3986778 , -0.58200477, 1.38438142, -0.59952009,\n", + " 1.67747105, 1.39956136, -0.84473452, -1.21372386, 0.94533082,\n", + " 0.64757044, 0.34981006, 1.711334 , 0.6989487 , -1.42390765,\n", + " 0.327624 , -0.66958136, 1.08428567, 0.93598932, -0.93581417,\n", + " 0.61604287, -0.54463877, 0.80987904, 0.41520058, 1.35051848,\n", + " 0.60553368, -1.63175608, -0.12660655, 0.74448853, 0.74332084,\n", + " -1.09228433, 1.33417085, 1.07727954, -0.51778195, -0.18615862,\n", + " -0.91129272, -1.34917564, -0.09040823, 1.05509347, 0.81454979,\n", + " 0.60786906, -0.43487612, -1.40405696, -0.20600931, 0.77484841,\n", + " -1.51965805, -1.39588315, -0.30993353, -1.65394214, -0.36247947,\n", + " -0.82488383, 1.08311798, -0.8377284 , -0.91829885, 0.7760161 ,\n", + " 0.53897548, -0.82605152, -0.42436693, -0.68592899, -0.43487612,\n", + " -0.14879261, -1.38303858, 0.82505898, 1.21273132, -0.46290062,\n", + " 0.18983684, 0.59035374, 0.4420574 , 1.66579418, -0.13828342,\n", + " 0.87994031, 1.74402926, 1.55486384, 0.47708803, 1.06443498,\n", + " -0.10675585, -1.42507534, -0.66140754, -1.56403019, 1.26527727,\n", + " 0.91964169, 1.10530405, 0.33463012, 0.73047627, -0.80386545,\n", + " -0.84006377, -0.09157591, -0.82488383, -0.24921376, -1.49046586,\n", + " -0.06705447, -1.49747198, 0.89862331, -0.27957364, 0.96284614,\n", + " -0.69877355, -1.62591764, -0.7805117 , 0.85541887, -1.02105537,\n", + " -1.70882347, 1.37971067, -1.61891151, 0.84958043, -1.2861205 ,\n", + " -1.15300409, -1.41806922, 1.47896413, -1.21489154, 0.4420574 ,\n", + " -0.85991446, 0.54481392, 0.85775424, -0.49559588, -0.59368165,\n", + " -0.07873135, 1.08662104, 1.12281936, -1.27327593, -1.19504085,\n", + " 1.56070228, -0.30409509, 0.59035374, 0.28325186, 0.47592034,\n", + " -1.66912209, -0.62053847, 0.03219899, -1.58037782, -0.1791525 ,\n", + " 0.29726411, -0.71628887, 0.48292647, 0.19217221, -0.34846722,\n", + " 1.02123053, -1.50798117, 0.69778102, 0.79820216, 1.60273904,\n", + " -1.1331534 , 0.20384909, -1.48813048, 0.24938891, 0.87994031,\n", + " 1.51633014, 1.18353913, 0.27040729, 1.51399477, 0.21669366,\n", + " 0.11160176, 0.83440049, -1.06075676, 1.64127273, 1.24659427,\n", + " 0.67676264, -0.08807285, 0.51445404, 1.62258973, -1.49863967,\n", + " -1.25576062, -0.83539302, -1.51615499, 0.23070591, 0.0310313 ,\n", + " -1.27094056, -0.61703541, 0.34981006, 1.59456522, 0.99320602]])" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "X_norm" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "## 2 - Implementation of the Neural Network Model for Linear Regression\n", + "\n", + "Setup the neural network in a way which will allow to extend this simple case of a model with a single perceptron and one input node to more complicated structures later." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 2.1 - Defining the Neural Network Structure" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Define two variables:\n", + "- `n_x`: the size of the input layer\n", + "- `n_y`: the size of the output layer\n", + "\n", + "using shapes of arrays `X` and `Y`." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "tags": [ + "graded" + ] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The size of the input layer is: n_x = 1\n", + "The size of the output layer is: n_y = 1\n" + ] + } + ], + "source": [ + "def layer_sizes(X, Y):\n", + " \"\"\"\n", + " Arguments:\n", + " X -- input dataset of shape (input size, number of examples)\n", + " Y -- labels of shape (output size, number of examples)\n", + " \n", + " Returns:\n", + " n_x -- the size of the input layer\n", + " n_y -- the size of the output layer\n", + " \"\"\"\n", + " n_x = X.shape[0]\n", + " n_y = Y.shape[0]\n", + " \n", + " return (n_x, n_y)\n", + "\n", + "(n_x, n_y) = layer_sizes(X_norm, Y_norm)\n", + "print(\"The size of the input layer is: n_x = \" + str(n_x))\n", + "print(\"The size of the output layer is: n_y = \" + str(n_y))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 2.2 - Initialize the Model's Parameters" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Implement the function `initialize_parameters()`, initializing the weights array of shape $(n_y \\times n_x) = (1 \\times 1)$ with random values and the bias vector of shape $(n_y \\times 1) = (1 \\times 1)$ with zeros." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "tags": [ + "graded" + ] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "W = [[0.01788628]]\n", + "b = [[0.]]\n" + ] + } + ], + "source": [ + "def initialize_parameters(n_x, n_y):\n", + " \"\"\"\n", + " Returns:\n", + " params -- python dictionary containing your parameters:\n", + " W -- weight matrix of shape (n_y, n_x)\n", + " b -- bias value set as a vector of shape (n_y, 1)\n", + " \"\"\"\n", + " \n", + " W = np.random.randn(n_y, n_x) * 0.01\n", + " b = np.zeros((n_y, 1))\n", + " \n", + " parameters = {\"W\": W,\n", + " \"b\": b}\n", + " \n", + " return parameters\n", + "\n", + "parameters = initialize_parameters(n_x, n_y)\n", + "print(\"W = \" + str(parameters[\"W\"]))\n", + "print(\"b = \" + str(parameters[\"b\"]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 2.3 - The Loop" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Implement `forward_propagation()` following the equation $(3)$ in the section [1.2](#1.2):\n", + "\\begin{align}\n", + "Z &= w X + b\\\\\n", + "\\hat{Y} &= Z,\n", + "\\end{align}" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "tags": [ + "graded" + ] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'W': array([[0.01788628]]), 'b': array([[0.]])}\n", + "Some elements of output vector Y_hat: [ 0.01734705 -0.02141661 -0.02711838 0.00093098 0.00705046]\n" + ] + } + ], + "source": [ + "def forward_propagation(X, parameters):\n", + " \"\"\"\n", + " Argument:\n", + " X -- input data of size (n_x, m)\n", + " parameters -- python dictionary containing your parameters (output of initialization function)\n", + " \n", + " Returns:\n", + " Y_hat -- The output\n", + " \"\"\"\n", + " W = parameters[\"W\"]\n", + " b = parameters[\"b\"]\n", + " \n", + " # Forward Propagation to calculate Z.\n", + " Z = np.matmul(W, X) + b\n", + " Y_hat = Z\n", + "\n", + " return Y_hat\n", + "\n", + "Y_hat = forward_propagation(X_norm, parameters)\n", + "\n", + "print(parameters)\n", + "print(\"Some elements of output vector Y_hat:\", Y_hat[0, 0:5])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Your weights were just initialized with some random values, so the model has not been trained yet. \n", + "\n", + "Define a cost function $(4)$ which will be used to train the model:\n", + "\n", + "$$\\mathcal{L}\\left(w, b\\right) = \\frac{1}{2m}\\sum_{i=1}^{m} \\left(\\hat{y}^{(i)} - y^{(i)}\\right)^2$$" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "tags": [ + "graded" + ] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "cost = 0.48616887080159726\n" + ] + } + ], + "source": [ + "def compute_cost(Y_hat, Y):\n", + " \"\"\"\n", + " Computes the cost function as a sum of squares\n", + " \n", + " Arguments:\n", + " Y_hat -- The output of the neural network of shape (n_y, number of examples)\n", + " Y -- \"true\" labels vector of shape (n_y, number of examples)\n", + " \n", + " Returns:\n", + " cost -- sum of squares scaled by 1/(2*number of examples)\n", + " \n", + " \"\"\"\n", + " # Number of examples.\n", + " m = Y_hat.shape[1]\n", + "\n", + " # Compute the cost function.\n", + " cost = np.sum((Y_hat - Y)**2)/(2*m)\n", + " \n", + " return cost\n", + "\n", + "print(\"cost = \" + str(compute_cost(Y_hat, Y_norm)))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "tags": [ + "graded" + ] + }, + "source": [ + "Calculate partial derivatives as shown in $(5)$:\n", + "\n", + "\\begin{align}\n", + "\\frac{\\partial \\mathcal{L} }{ \\partial w } &= \n", + "\\frac{1}{m}\\sum_{i=1}^{m} \\left(\\hat{y}^{(i)} - y^{(i)}\\right)x^{(i)},\\\\\n", + "\\frac{\\partial \\mathcal{L} }{ \\partial b } &= \n", + "\\frac{1}{m}\\sum_{i=1}^{m} \\left(\\hat{y}^{(i)} - y^{(i)}\\right).\n", + "\\end{align}" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "dW = [[-0.76433814]]\n", + "db = [[5.19584376e-16]]\n" + ] + } + ], + "source": [ + "def backward_propagation(Y_hat, X, Y):\n", + " \"\"\"\n", + " Implements the backward propagation, calculating gradients\n", + " \n", + " Arguments:\n", + " Y_hat -- the output of the neural network of shape (n_y, number of examples)\n", + " X -- input data of shape (n_x, number of examples)\n", + " Y -- \"true\" labels vector of shape (n_y, number of examples)\n", + " \n", + " Returns:\n", + " grads -- python dictionary containing gradients with respect to different parameters\n", + " \"\"\"\n", + " m = X.shape[1]\n", + " \n", + " # Backward propagation: calculate partial derivatives denoted as dW, db for simplicity. \n", + " dZ = Y_hat - Y\n", + " dW = 1/m * np.dot(dZ, X.T)\n", + " db = 1/m * np.sum(dZ, axis = 1, keepdims = True)\n", + " \n", + " grads = {\"dW\": dW,\n", + " \"db\": db}\n", + " \n", + " return grads\n", + "\n", + "grads = backward_propagation(Y_hat, X_norm, Y_norm)\n", + "\n", + "print(\"dW = \" + str(grads[\"dW\"]))\n", + "print(\"db = \" + str(grads[\"db\"]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Update parameters as shown in $(6)$:\n", + "\n", + "\\begin{align}\n", + "w &= w - \\alpha \\frac{\\partial \\mathcal{L} }{ \\partial w },\\\\\n", + "b &= b - \\alpha \\frac{\\partial \\mathcal{L} }{ \\partial b }.\n", + "\\end{align}\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "W updated = [[0.93509205]]\n", + "b updated = [[-6.23501251e-16]]\n" + ] + } + ], + "source": [ + "def update_parameters(parameters, grads, learning_rate=1.2):\n", + " \"\"\"\n", + " Updates parameters using the gradient descent update rule\n", + " \n", + " Arguments:\n", + " parameters -- python dictionary containing parameters \n", + " grads -- python dictionary containing gradients \n", + " learning_rate -- learning rate parameter for gradient descent\n", + " \n", + " Returns:\n", + " parameters -- python dictionary containing updated parameters \n", + " \"\"\"\n", + " # Retrieve each parameter from the dictionary \"parameters\".\n", + " W = parameters[\"W\"]\n", + " b = parameters[\"b\"]\n", + " \n", + " # Retrieve each gradient from the dictionary \"grads\".\n", + " dW = grads[\"dW\"]\n", + " db = grads[\"db\"]\n", + " \n", + " # Update rule for each parameter.\n", + " W = W - learning_rate * dW\n", + " b = b - learning_rate * db\n", + " \n", + " parameters = {\"W\": W,\n", + " \"b\": b}\n", + " \n", + " return parameters\n", + "\n", + "parameters_updated = update_parameters(parameters, grads)\n", + "\n", + "print(\"W updated = \" + str(parameters_updated[\"W\"]))\n", + "print(\"b updated = \" + str(parameters_updated[\"b\"]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 2.4 - Integrate parts 2.1, 2.2 and 2.3 in nn_model() and make predictions" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Build your neural network model in `nn_model()`." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "tags": [ + "graded" + ] + }, + "outputs": [], + "source": [ + "def nn_model(X, Y, num_iterations=10, learning_rate=1.2, print_cost=False):\n", + " \"\"\"\n", + " Arguments:\n", + " X -- dataset of shape (n_x, number of examples)\n", + " Y -- labels of shape (n_y, number of examples)\n", + " num_iterations -- number of iterations in the loop\n", + " learning_rate -- learning rate parameter for gradient descent\n", + " print_cost -- if True, print the cost every iteration\n", + " \n", + " Returns:\n", + " parameters -- parameters learnt by the model. They can then be used to make predictions.\n", + " \"\"\"\n", + " \n", + " n_x = layer_sizes(X, Y)[0]\n", + " n_y = layer_sizes(X, Y)[1]\n", + " \n", + " parameters = initialize_parameters(n_x, n_y)\n", + " \n", + " # Loop\n", + " for i in range(0, num_iterations):\n", + " \n", + " # Forward propagation. Inputs: \"X, parameters\". Outputs: \"Y_hat\".\n", + " Y_hat = forward_propagation(X, parameters)\n", + " \n", + " # Cost function. Inputs: \"Y_hat, Y\". Outputs: \"cost\".\n", + " cost = compute_cost(Y_hat, Y)\n", + " \n", + " # Backpropagation. Inputs: \"Y_hat, X, Y\". Outputs: \"grads\".\n", + " grads = backward_propagation(Y_hat, X, Y)\n", + " \n", + " # Gradient descent parameter update. Inputs: \"parameters, grads, learning_rate\". Outputs: \"parameters\".\n", + " parameters = update_parameters(parameters, grads, learning_rate)\n", + " \n", + " # Print the cost every iteration.\n", + " if print_cost:\n", + " print (\"Cost after iteration %i: %f\" %(i, cost))\n", + "\n", + " return parameters" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "tags": [ + "graded" + ] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Cost after iteration 0: 0.496595\n", + "Cost after iteration 1: 0.206164\n", + "Cost after iteration 2: 0.194547\n", + "Cost after iteration 3: 0.194082\n", + "Cost after iteration 4: 0.194063\n", + "Cost after iteration 5: 0.194063\n", + "Cost after iteration 6: 0.194062\n", + "Cost after iteration 7: 0.194062\n", + "Cost after iteration 8: 0.194062\n", + "Cost after iteration 9: 0.194062\n", + "Cost after iteration 10: 0.194062\n", + "Cost after iteration 11: 0.194062\n", + "Cost after iteration 12: 0.194062\n", + "Cost after iteration 13: 0.194062\n", + "Cost after iteration 14: 0.194062\n", + "Cost after iteration 15: 0.194062\n", + "Cost after iteration 16: 0.194062\n", + "Cost after iteration 17: 0.194062\n", + "Cost after iteration 18: 0.194062\n", + "Cost after iteration 19: 0.194062\n", + "Cost after iteration 20: 0.194062\n", + "Cost after iteration 21: 0.194062\n", + "Cost after iteration 22: 0.194062\n", + "Cost after iteration 23: 0.194062\n", + "Cost after iteration 24: 0.194062\n", + "Cost after iteration 25: 0.194062\n", + "Cost after iteration 26: 0.194062\n", + "Cost after iteration 27: 0.194062\n", + "Cost after iteration 28: 0.194062\n", + "Cost after iteration 29: 0.194062\n", + "W = [[0.78222442]]\n", + "b = [[-6.07514039e-16]]\n" + ] + } + ], + "source": [ + "parameters_simple = nn_model(X_norm, Y_norm, num_iterations=30, learning_rate=1.2, print_cost=True)\n", + "print(\"W = \" + str(parameters_simple[\"W\"]))\n", + "print(\"b = \" + str(parameters_simple[\"b\"]))\n", + "\n", + "W_simple = parameters[\"W\"]\n", + "b_simple = parameters[\"b\"]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "You can see that after a few iterations the cost function does not change anymore (the model converges).\n", + "\n", + "*Note*: This is a very simple model. In reality the models do not converge that quickly.\n", + "\n", + "The final model parameters can be used for making predictions, but don't forget about normalization and denormalization." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": { + "tags": [ + "graded" + ] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "TV marketing expenses:\n", + "[ 50 120 280]\n", + "Predictions of sales:\n", + "[ 9.40942557 12.7369904 20.34285287]\n" + ] + } + ], + "source": [ + "def predict(X, Y, parameters, X_pred):\n", + " \n", + " # Retrieve each parameter from the dictionary \"parameters\".\n", + " W = parameters[\"W\"]\n", + " b = parameters[\"b\"]\n", + " \n", + " # Use the same mean and standard deviation of the original training array X.\n", + " if isinstance(X, pd.Series):\n", + " X_mean = np.mean(X)\n", + " X_std = np.std(X)\n", + " X_pred_norm = ((X_pred - X_mean)/X_std).reshape((1, len(X_pred)))\n", + " else:\n", + " X_mean = np.array(np.mean(X)).reshape((len(X.axes[1]),1))\n", + " X_std = np.array(np.std(X)).reshape((len(X.axes[1]),1))\n", + " X_pred_norm = ((X_pred - X_mean)/X_std)\n", + " # Make predictions.\n", + " Y_pred_norm = np.matmul(W, X_pred_norm) + b\n", + " # Use the same mean and standard deviation of the original training array Y.\n", + " Y_pred = Y_pred_norm * np.std(Y) + np.mean(Y)\n", + " \n", + " return Y_pred[0]\n", + "\n", + "X_pred = np.array([50, 120, 280])\n", + "Y_pred = predict(adv[\"TV\"], adv[\"Sales\"], parameters_simple, X_pred)\n", + "print(f\"TV marketing expenses:\\n{X_pred}\")\n", + "print(f\"Predictions of sales:\\n{Y_pred}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's plot the linear regression line and some predictions. The regression line is red and the predicted points are blue." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjMAAAGwCAYAAABcnuQpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAA9hAAAPYQGoP6dpAABeRklEQVR4nO3de3yT5d0/8M/dKJVTiz0ApakUEHWe2Dwx5q9YhInTuc5YD8DjZDqcE7UFYXMOhe6Zjz6KQOecx03dXKvQBtn2qNNCilXQecKzDFgZpRQVkJaDFJp+f3/EpEmbw53kPiaf9+uVl3Lnzp0rd9Lc31zX9/peiogIiIiIiGwqw+wGEBERESWDwQwRERHZGoMZIiIisjUGM0RERGRrDGaIiIjI1hjMEBERka0xmCEiIiJbO8rsBhihu7sbO3bswODBg6EoitnNISIiIhVEBPv27cOIESOQkRG5/yUtgpkdO3agqKjI7GYQERFRAlpaWuB0OiPenxbBzODBgwH4TkZWVpbJrSEiIiI1Ojo6UFRUFLiOR5IWwYx/aCkrK4vBDBERkc3EShFhAjARERHZGoMZIiIisjUGM0RERGRrDGaIiIjI1hjMEBERka0xmCEiIiJbYzBDREREtsZghoiIiGyNwQwRERHZWlpUACYiItKC1+tFU1MT2traUFBQgJKSEjgcDrOblfYYzBAREangdrtRUVGB7du3B7Y5nU5UV1fD5XKZ2DLiMBMREVEMbrcb5eXlIYEMALS2tqK8vBxut9uklhHAYIaIiCgqr9eLiooKiEif+/zbKisr4fV6jW4afY3BDBERURRNTU19emSCiQhaWlrQ1NRkYKsoGIMZIiKiKNra2jTdj7THYIaIiCiKgoICTfcj7TGYISIiiqKkpAROpxOKooS9X1EUFBUVoaSkxOCWkR+DGSIioigcDgeqq6sBoE9A4//3smXLWG/GRAxmiIiIYnC5XKirq0NhYWHIdqfTibq6OtaZMZki4eaapZiOjg5kZ2ejvb0dWVlZZjeHiIhsihWAjaX2+s0KwERERCo5HA6UlpYm9FgGQvphMENERKQzLoWgL+bMEBER6YhLIeiPwQwREZFOuBSCMRjMEBER6YRLIRiDwQwREZFOuBSCMRjMEBER6YRLIRiDwQwREZFOuBSCMRjMEBER6YRLIRiDwQwREZHGvF4vGhsbUVtbi5ycHDz77LNcCkFHLJpHRESGM7sarp7PH6lA3tKlS5GXl4e2tjYMHToUAPD555+jsbGR1YCTxGCGiIgMZXY1XD2f318gr3ddmdbWVlxxxRWoq6tDZmYmZs6cyWrAGuJCk0REZJhIF3t//ojewy56Pr/X60VxcXHEujKKoiAnJwe7d+8Oe1+yz5+K1F6/GcwQEZEh1FzsnU4nmpubdRly0fv5GxsbMWnSpITbp/frtyO1128mABMRkSHMroar9/MnW/iO1YATx2CGiIgMYXY1XL2fX6vCd6wGHD8GM0REZAizq+Hq/fyxCuSpxWrA8TM9mLn77rtx9tlnY/DgwRg6dCh++MMfYuPGjSH7lJaWQlGUkNsNN9xgUouJiCgRZlfD1fv5YxXIUxQFubm5rAasA9ODmbVr12L27Nl4/fXX8fLLL+PIkSO44IILcODAgZD9Zs2ahba2tsDt3nvvNanFRESUCLOr4Rrx/C6XC3V1dREL5D366KO6Pn/aEov5/PPPBYCsXbs2sO28886TiooK1cc4dOiQtLe3B24tLS0CQNrb23VoMRGR/XR1dYnH45GamhrxeDzS1dVl2HPX19eL0+kUAIFbUVGR1NfXp8zzRzu/Zr9+O2lvb1d1/bbc1OzNmzdj7Nix+OCDD3DqqacC8A0zffTRRxARDB8+HJdccgnuuOMODBgwIOwxFi1ahKqqqj7bOTWbiMj8onVAalcAtsPz24Ut68x0d3fjBz/4Afbu3YtXX301sP3RRx/FyJEjMWLECLz//vv4xS9+gXPOOQdutzvscTo7O9HZ2Rn4d0dHB4qKihjMEFHaM7toHVE8bBnM/OxnP8MLL7yAV199FU6nM+J+a9asweTJk7F582aMGTMm5nFZNI+IyPyidUTxsl3RvJtuugl///vf4fF4ogYyADB+/HgAviEpIiJSx+yidakueKXsxsZGeL1es5uUNkxfaFJEcPPNN2PlypVobGzEqFGjYj5mw4YNADgXn4goHmYXrUtlVshDSmemBzOzZ89GTU0NVq1ahcGDB2Pnzp0AgOzsbPTv3x9btmxBTU0NLrroIuTm5uL999/HnDlzMHHiRJx++ukmt56IyD7MLlqXqqKtlF1eXs48JAOYnjMTqXjQE088gZkzZ6KlpQX/9V//hQ8//BAHDhxAUVERLr30UixYsEB1/gtzZoiIenJmWltb+1x4gdTLmTFixhDzkPSl9vptes9MrFiqqKgIa9euNag1RESpy180rry8HIqihHz/plrRNqOGfeLJQyotLdXseSmUZRKAiYhIf7Eq1KbCcIh/2Kd3kOEf9olU1iMRzEOyBtOHmYzAYSYiolCpWrTN6GGfxsZGTJo0KeZ+Ho+HPTMJsGWdGb0wmCEiSg9GBxfplodkNNvkzBARUXJStZclEUYP++iZh8T3VT3mzBAR2Zjb7UZxcTEmTZqE6dOnY9KkSSguLtY0L8ROzJh+rkceEt/X+HCYiYjIRMn8+uY6S32ZOeyjVU8K39cezJkJwmCGiKwomenDrG8SmT8YABB22MfKwQDf11C2W5uJiCidJDt9mOssRWbn6ed8XxPDBGAiIoN5vV5UVFSEHQYRESiKgsrKSpSVlUX89c36JtG5XC6UlZXZLoGW72tiGMwQERlMi6qxXGcpNofDYbvaLnxfE8NhJiIig2nx67ukpAROpzPi+naKoqCoqAglJSUJtZHMwfc1MQxmiIgMpsWvb399E6Dvgr1WW2fJ6/WisbERtbW1aGxshNfrNbtJlmWn99VKGMwQERlMq1/fdkh0Zb2U+NnhfbUaTs0mIjKBltOHrVoplvVSkmPV99VIrDMThMEMEVlRuDozRUVFWLZsme0v8qyXQlpgMBOEwQwRWVWq/vrmatKkBS40SURkA3acPqwG66WQkZgATEREmmO9FDISgxkiItIc66WQkTjMRERkccnk1ZiVk+Ovl1JeXg5FUcLO2GK9FNIKe2aIiCwsmTotZtd4Yb0UMgpnMxERWVQydVqsVOMlVWdskf44NTsIgxkisoJ4LurJ1GlhjRdKFWqv3xxmIiIyQLxDPvGsrK3lY4kSsmULYGLfCIMZIiKd+Yd8egcYra2tKC8vDxvQJFOnJZVqvHCRSgsTAdauBS65BDj+eOCVV0xrCoMZIiIdeb1eVFRU9MldAXrWZKqsrOxzkU6mTovax27atEnVfmYxO4GZIujqAp59FjjnHKC0FPj73wFFAdatM61JzJkhItJRomX9/Xkvra2tYQOhWDkzI0eORGtra9TndDqd2Lp1qyWneVspgZm+tn8/8Mc/AkuXAlu3+rYdcwwwcyYwZw5wwgmaPyVzZoiILCDRIR9/nRYAfQrPxarT4nA4cP3118d8zu3bt6vOmzGylyTR3izSSVsbcPvtQFERUFHhC2Ty8oBFi4Bt24CHHtIlkIkHgxkiIh0lM1yUTJ2WsWPHqnpeNcFWIjk/yWACs0V89BFw7bVAcTFw993A3r3A2LG+4GXbNmDhQiA/3+xWAmAFYCIiXfnL+scaLopU1t/lcqGsrCzu4R2t1kaK1UuiKAoqKytRVlam2ZBTKiUw244I4PEAixcDL7zQs/3cc4F583zJvhaczs9ghohIR1qU9U9kZe1kgyi/eHpJ1LRRTd4NF6k0wZEjQF2dL4h55x3fNkUBXC7g1luBCRPMbV8MHGYiItKZkWX9/VOZly9fjlmzZgV6T4LFszaSlr0kavNuuEilgfbt8yX0Hn88MH26L5Dp3x+YPRv41798AY7FAxkAgKSB9vZ2ASDt7e1mN4WI0lhXV5d4PB6pqakRj8cjXV1dmh6/vr5enE6nAAjccnNzJTc3N2RbUVGR1NfXqzqmx+MJeWykm8fjidk2RVH6PE5RFFEUpU97/Pv3fkyk/SlO27eL/PznItnZIr7BJZH8fJFf/1rkiy/Mbl2A2us3p2YTEaWAWFOZFy1ahLFjxya06naiU8R7HyPe5RXcbjcqKipCHldUVIRly5ZxWnaiPvgAuP9+oKbGN7QEACee6BtKuvpq31RrC+HaTEEYzBBRKtN7LaYVK1bgiiuuCHtcIHbNl0Rr7QBcpFITIsDq1b58mH/8o2f7xIm+pN6LLwYyrJl1ovb6zQRgIiKb0zpJN5jb7cbcuXPD3ud0OlX1kiSTd5NI8rMd6RK0HTniq9S7eDHw3nu+bRkZQHm5ryfmnHOSb7hFMJghIrI5vaYyRxq68luyZImq4R7OToou3HCa0+lEdXV1YsNpHR3AY48By5YB/mMOGABcdx1QWQmMHq1Ju63Emv1KRESkmh7BQrT6MoBviGnu3LmqqvBydlJkmhYkbGkB5s/3VeqdN88XyAwbBtx1l+++3/42JQMZgMEMEZEqVl69WY9gQcsqvMkszZDKNFu2YcMGX/Lu6NG+IaWODuAb3wD+8Aff0gO33w7k5Gj/AiyEwQwRUQxWX71Zj2BB66ErI2vt2EVSAaMI8NJLwHe/C3zrW8DTT/tWsy4tBf7v/4APP/QtRWCx2Ul6YTBDRBSF0esSJUrrYEGPoSuXy4WtW7fC4/GgpqYGHo8Hzc3NaRnIAAkGjIcPA3/6EzBuHDB1KtDQ4Fte4KqrgDff9C1FcNFFlp2dpBdOzSYiikDvKc960GpWjBb1ZSi6uKasf/ObwKOPAtXVwI4dvjsGDgRmzfKtZF1crGtbzcKp2URESdJzyrNetJrKrMWaUhSdmvWzxg8fjonPPedb4HH/ft8dBQXALbcAP/0pcOyxxjbaotKrH4qIKA7pvnoz81z0TfyOlut0BoA/i+C1zz5DRnW1L5A55RTgiSeA5mbgttsYyARhzwwRUQRq80GGDh2qc0vM43K5UFZWltDQld2r92pe/yUMf8Dof54LAcwDMNm/Q3c3MHmyb6r11Km+laypLz0WhrIaLjRJRIno6uoSp9MZdoHE4JvT6eTCh72EW/TSTucp3oUxk3LokPzzxhvlY4cjsOjjEUDqBwyQhvvu0+55bIgLTQZhAjARJco/mwlA1AJyQOw1itJFrEUvrX6eDEv8/vJL4JFH8NW996L/l18CAPYBeBRANYDtNjlfeuJCk0EYzBBRMtxuN2655Ra0trZG3Ieze3zsOAOst2QWxlRl61bfUgOPPw4cOAAA2A5fAPMYgPagXe1wvvSk9vrNBGAiohhcLheeeuqpqPtIHBVxU5mWlYPNolvi91tv+erBjBnjm2J94AD2jx6NqwGMBrAYoYEMYI/zZQVMACYiUuHzzz9XtV+qzmxSKxVmgMVbMNDrBZqagLY236zpkhJfHTsAvgTeF14A7rsPWLu258Hf/S4wbx7+tmsXnp4xI+ZzWfl8WQGDGSIiFbjyszqpcJ7U1H9xOp0oKSmB2+2rWRfcGeV0AtX3HYbrwJ+B++8HPvnEd8dRRwHTpgG33uqr4AugoLFRVZusfL6sgDkzREQqsCKuOlqcJ62ndCdyvEiJ38FJzIAL5eW+6Uchr/HryU91KIcLK4GsLF+Bu1tu8UU6vdrGz1Vkqq/fOs2mshROzSYiLfin6/aesqvLdF2ddXV1icfjkZqaGvF4PNLV1aXZsZM5T1pP6U7meOEeW1RUJPX19dLVJeJ0BmZS97kp8EqRY7t03Xu/yN69MZ8nVT5XWlN7/WYwQ0QUh2gXOLswogZMIudJ69ouWhwvUtDn8UQOZIJvHo/6ttr9c6UH1pkJwmEmItKSnSvbGlkDJp7zpPWUbl2niHd3o/bn72L6/WfG3LWmxpcmo4adP1d6sc0w0//8z//IWWedJYMGDZL8/HwpKyuTTz/9NGSfr776Sm688UbJycmRgQMHisvlkp07d6p+DvbMEBH1VDRGhErGiqJIUVGRpkNOank8nqhVlv03j8quDq2PJyIiBw+KPPKIyAkniAfnadozQ+GpvX6bXmdm7dq1mD17Nl5//XW8/PLLOHLkCC644AIc+LqQEADMmTMHf/vb37BixQqsXbsWO3bsSNtqiEREibJyDZh4p3THWgBS0yniu3YB//3fwMiRvkTef/0LJVnvwzl4L3yjWH0pClBU5JumTfozfWr2iy++GPLvJ598EkOHDsXbb7+NiRMnor29HX/4wx9QU1OD888/HwDwxBNP4Bvf+AZef/11fPvb3zaj2UREtpPsBV7PYZB4pnSrWQBSkynimzcDS5f6Vqr+6ivftuOOA+bMgeO661D98mCUl/sCl+BRO/9akMuWBdWbIX0Z01Gk3qZNmwSAfPDBByIisnr1agEgX375Zch+xx13nCxZsiTsMQ4dOiTt7e2BW0tLC4eZiCjtJTP0onfScKxFPf1DYMuXL1eV1Kv2eGGH1NatE3G5RBSlZ7zojDNEamtFjhzpdV76zmoqKvJtp+TZcjaT1+uViy++WM4999zAtr/85S/Sr1+/PvueffbZ8vOf/zzscRYuXBj2w8tghij16Tnl2O4SvcAbtYJ0rCnKK1asiCvnJ64pz11dIitXinznO6GRyUUXiaxZI9LdHeW8+nJjamp8/+VHTju2DGZuuOEGGTlypLS0tAS2JRLMsGeGKD0ZMeXY7ubPnx81GOh9roxOGo42RTmRnqWYU54PHhR56CGRsWN7Aph+/USuvVbkww81eU2UONsFM7Nnzxan0yn//ve/Q7YnMszUG2czEaU+o3oP7Kqrq0uqqqqiBgHz58/v8zhdZgWpaGu43rWamhpVbampqYl9vM8/F1m4UCQvryeIGTJE5Je/FNmxQ7PXQsmxTTDT3d0ts2fPlhEjRsi//vWvPvfv3btXjj76aKmrqwts+/TTTwWArF+/XtVzMJghSm1WnnJsBfX19VJYWBgzCAh3jhINIPSgSWC1caPIT38qcswxPUFMcbFIdbXIvn26vwaKj9rrt+mzmWbPno2amhqsWrUKgwcPxs6dOwEA2dnZ6N+/P7Kzs3Hddddh7ty5yMnJQVZWFm6++WZMmDCBM5mICEB8U45LS0uNa5gFRCqSF064c2SlhSPjWQAyhAiwbh2weDGwahUCU4/OOguYPx9wuXyLQJJtmf7uPfTQQwDQ5wvmiSeewMyZMwEAS5cuRUZGBi677DJ0dnZi6tSp+P3vf29wS4kIsGaVUk1riujArHPm9XpRUVGhKpDx632OEg4gdOBwOFBdXY3y8nIoihJ2Achly5b1nFuvF3juOV8Q8/rrPQe65BJg3jxfERj/PGqyN937iCyAw0xE2rBqgq0ZeR1qmXnO1J6XWOfIagshxkzq3b9f5He/Exk9umcoKTNTZNYskY8/NrStlBzb5MwYgcEMUfKsnGCbVE0RHZl9ztTmu6g5R1ZbCDFsUu/OnSJ33CGSk9MTxOTkiCxY4LuPbIcLTQbhQpNEydF10T6N+HNDAIQdfli+fDny8vIMG+rR4pwlOzzV2NiISZMmqdpXUZSYi0xacYgRAPDpp8CSJcCf/gR0dvq2jR4NzJ0LzJwJDBxoavMocbZZaNII7JkhSo6Vh3GCReo9mD9/vuFDPcmeMy2Gp2L1WOl5LpIpXqjqsd3dImvXilxySWiRu/HjRerqWLkuRXCYKQiDGaLkWGl6biy9L4Rqy99rLZlzpuXwVKR8F/+tqqpK8+G3ZAKxmI89ckRk+XKRs8/uCWAURaSsTKSpKWqlXrIfBjNBGMwQJccuPTO9mVl/JtFzpkebjcx3SSYQi/bYQYC8e911IqNGhSb1/vSnIp9+qvnrIGtgzkwQ5swQJcef/xFreq6ZOTPhqM0Z8Xg8mtefSfSc6dVmI/JdkskTivTY4QBuAvAzADn+jbm5wE03ATfeCAwdqulrIGtRe/3OMLBNRGRT/voeQE9CrV/Y+h4WYWb9mUTPmV5tdjgcKC0txbRp01BaWqrLexVP8cJYj/0GgMcBbAXwK/gCmU0A/jVnDrBtG7BoEQMZCmAwQ0SquFwu1NXVobCwMGS70+mMOQsmHK/Xi8bGRtTW1qKxsRFer1fL5gIwv3ptIufM7DYnI5lAzL/tPAB/A/AxgOsAZAJ4DcClAE4C8PbZZwMDBmjSXkodplcAJiL7cLlcKCsrS3q4wu12o6KiIuSXuNPpRHV1ddxBUTRWqF4b7zmzQpsTlXAg1tWFcZ98gjcBnPX1pm4AKwHcD2B9As9BaUbv5B0rYAIwkXUYXUjOatVr1Yi3zclMg9ZS3MULOzpEli4VGTkykNR7AJDfAXK8gcnaZF2czRSEwQyRNZg1u8hq1WuDRQpE1LbZaktMqArEWltFbrtNZMiQnplJ+fny0ZVXSt7X+9ol8CR9MZgJwmCGyBrMnOJtld6LYLECkVhtNnu5hEgiBWIvL10qMnOmyNFH9wQxJ5wg8sgjIgcPRn0sA5n0xKnZQTg1m8gaamtrMX369Jj71dTUYNq0aQa0yDz+5Rd6fwX7ZzqpWVrAyktMBKaC79iBb7S1YdzLL0P5xz96digp8a1c/f3vAxkZ4R9rtWUTyHBqr99MACYiw9h5po6WvF4vKioqwib4iggURUFlZSXKysoiXsTjmQatdQ0dNRzd3SjdsQNYvBh4913fxowMwOXyBTHjx0d+7NfTyInUYjBDRACM+TWs9Uwdu/6C1yIQSWQatCHnq6MDePxxYNkyoKXFt23AAODaa4HKSmDMGG2fjwgMZogIxk2V9heSKy8vh6IoYVe3Vlt8z6g260GLwnjx9nLpfr62bwd++1vgkUd8AQ0ADBsG3HwzcMMNvqq9RHrRN3XHGpgATBSZGUmkySZ5WjXxVS0tEqHjmQat6/nasEHk6qtFjjqqJ6n3pJNEHn9c5KuvEj8ukTABOAQTgInCMzOJNNEhD6snvqqh1VpX/iRiAGF7uerq6lBWVqb9+RIBGhp8+TAvvdSz/bzzfPkwF13UJ6mXKBGqr98GBFamY88MUXh2XA3bjm0OR6tifrF6uTQ9X52dIn/6k8jpp/f0wmRkiFx5pcg//5nE2SAKT+31mzkzRGnMzIUYE2XHNofjX7cpXB7LsmXLVOexxFouQZPz1d4OPPooUF0NtLb6tg0cCPzkJ0BFBTBqlKrn0IJdk75JXwxmiNKYHadK27HNkWi11lW0qcxJna+WFl8A8+ijwL59vm3DhwO33AL89KdATk5c7UyWnZO+SV/MmSFKY1rlbhjJjm02U0Ln6913gfvvB559Fujq8m07+WRfPsz06UBmpoGvwCfZIoNkT2qv38zQIkpj/qnSQM9FwS/eqdJGsWObzeD1etHY2Ijly5dj1qxZAGKcr4wM4MUXgSlTgDPOAP7yF18gc/75wPPPAx98APz4x6YEMrGKDAJAZWUlvF6v6uM1NjaitrYWjY2Nqh9HFqZz7o4lMAGYKDoj18PRao0kK6/hY/Y6UOHOTW5uruTm5vY5X+5nnhF58kmRU0/tSep1OESmTRN56y1D2x2JlknMVluYk6LjQpNBGMwQxWbEBVjrC4nZQUM4Zl8so9WUASBVVVVSU1MjTX/7m3j/539ERozoCWIGDRKZM0dk61ZD2qpWTU2NqmCmpqYm6nHsXp8oHTGYCcJghihxWvakpPqFxOzX6C+kF+liryiKfLugQLy33OILXPxBzIgRIv/7vyJffqlr+xKlZZHBaOfGX2SQrIPBTBAGM0SJ0aqXwQ4XkmSDNiu8xmgX/TMAqQHkiD+AAUROO03kqad89WMsLJ5qx5GkSn2idKP2+s0EYCIKyz97pHfl2NbWVpSXl8Ptdqs+VjwLK5rB7XajuLgYkyZNwvTp0zFp0iQUFxfb7jX2rhWjAPgegNUA3gYwDb56HDtPPdWX7Pvee8CPfgT066dbm7SgRdJ3qtQnovAYzBBRH1rPHrHyhUSroM0Kr3HTpk0AgH4AfgzgAwDPAzgfwBEAfwbwTQCPXn45MHUq0CswsDJ/kcHCwsKQ7U6nU9W07FSqT0R9MZghoj607mXQ60KS7BRbLYM2sy+Wbrcb1QsX4pcAtgL4I4BTAHQAWAxgNIAfAXgPwGOPPWbL6cgulwtbt26Fx+NBTU0NPB4PmpubVdWXKSkpgdPp7NOz46coCoqKilBSUqJ1s8kIeo93WQFzZojio9XsET8tch560yKfR8s8Cj1eo1pdmzbJ44MGyb6gfJgWQG4FJIu5IQFarYdFxmHODJGNmV3US+teBq0L3VlxaMiUYn5vvglceSUyTjwR1+3fj0EANgD4L/h6Yu6Hr2cmnHTMDUl2qIoszKDgylTsmSE7MbtOiYh+vQxaFLrTctaQHjNc1L7GaLOnos6s8npF/vpXkYkTe2YlAfICIJNVvJZEXlOqsWJ9IgqPU7ODMJghuzC7Tkm4tmjdJZ/shcQOQ0OxXmO0gDXSfStra0Uee0zkpJN6gpijjhL50Y/kn48/rjqIscI0eCK1GMwEYTBDdmCFOiW9WXHJAK3zeYzOo4hVobf3LQeQBYDsDOqFkawskZ//XKSlRURiB2VmBsVEyWDODJHNWKFOSW/JzB7Ri9b5PEbmUaiZPeU3GsADAFoA/DeAYQBaHQ50L14MtLQA//u/gNMJIHq+TjDmhlCqOsrsBhCRjxXqlITjcDhQWlpq6HNG459i29raGjYoUBQFTqczrim2LpcLZWVlaGpqQltbGwoKClBSUqL5ytuxAlYAGA9gHgAXempnvAPgPgB1Xi9ePvNMlGZlhX0NdXV1qKioCHmO/Px8zJgxA2VlZbq8JiIrYDBDZBFm1ymxC38vRHl5ORRFCQlokpk1ZETQFikQVQBcAl8QExyCPQ9fjRiPimMAxgVlRFbDYIbIIvTocUhVkXohnE4nli1bZtlhlN6B6DHwFbK7FcAJX287DOBpAEsAfKTiGL1ZrSeNyAiKhPvWTDEdHR3Izs5Ge3s7ssJ0zxJZhb9+CoCwPQ7Mdwjl9Xpt1Qvh9XpRXFyMzu3b8TMANwHI//q+LwE8BOB3AML1vfiD2ebmZku/xnRlt8+iXai9frNnhshCIvU45OXl4fe//z0DmV7s1gvh+Pe/sfaUU1CwfTv6f71tK4ClAJ4AsO/rbVoOn5H+3G532F7C6upq/s0ahLOZiCzG5XJhyZIlyMvLC2z74osvMGfOnLhWcSYLWbcOcLmAE0/E6H/8A/0BvH/00bgSwPEAfgtgSFER6uvrUV9fzwq1NqLl6vKUOA4zEVmM/8ux95+mFkNN7Ao3kNcLrFoFLF4MrF/fs/373wfmzYP33HPR9OqrYd8Lte+TXd5Pu7QzXv5hw0gz1PxDg5s3b8a6detS7vUbQfX1W79SN9bBonlkF3oWzrPCMglp4cABkd//XuT443uK3PXrJ3LddSIffaTZ09jl/bRLOxOhthp1fn5+Sr5+I7ACcBAGM2QXeqwVJGKtZRJS1mefidx5p0hubk8Qc+yxIr/6lUhbm6ZPZZf3s76+PqUrEautRp2qr98IrABMZEN6FM5TU3W2srLS8JW5U8bGjcBPfwocdxzw618Du3cDo0YBDzzgq9T7m98Aw4dr9nR2eT+9Xi+uv/76sPdZqZ3JSLTmU6q8fithMENkIXoUzrPiMgm2JwI0NQFlZcBJJwGPPgp0dmLXmDH4aNEieD/9FLjpJmDgQM2f2i7v51133YXdu3dHvN8q7UyGvzZUtCUkIkmF128lDGaILCTWl6OiKCgqKoqrcJ7eyyR4vV40NjaitrYWjY2Nqf1L0+sF6uog3/42MHEi8Ne/AgBeOuYYlADI37IFpy5ahOIxY3SbxWLGshfxvsderzewVlQsRi/PoSW1a2JFY+fXbyUMZogsJNqXY6K1RvRcJsHtdqO4uBiTJk3C9OnTMWnSJBQXF6fedNQDB4Df/Q4YOxa4/HIo//wnDgF4BMBJAKYeOoRXg3bXc1qu0cteJPIeNzU1Yc+ePYa20yyRFirNz8+P8IhQdn/9lqFr5o5FMAGY7CbcDJCioqKEEgb9M6TCJYwiiRlSdklCTUpbmy+B99hjA0m9XwCyCJB8FUmeic48i0av9zOcRN9jtYmxOTk5mp8fs3R1dYnH45GamhrxeDzS2dlp2PuUyjibKQiDGdJL7y8wLb+YtDy2/6LU+4s10cBDzynklvDxxyI/+YlvSvXXQUz3mDFy+5Ah0j/OmSvxzjxTQ+v3M5xk3mO1s/KqqqqSbqeVGfE+pToGM0EYzJAe7FA/IzggqqqqksLCQk16e/SaQm6q7m6RxkaR73+/Z2o1IPLtb4vU14unoSGuIMZ/e/rpp3UJeLXsvQsnmfc4Vu8RAMnNzbVvsBsHvd+nVMdgJgiDGdKaHYZYIgVbVVVVSV9Y1Q4j1NTUaPyqdHDkiMgzz4icdVZPAKMoIpdeKvLaa4HdEq0pkpeXp1vAq2fPYLLvcaReCf/fiRX+Royi5/uU6hjMBGEwQ1qywxCL3sFWSvTM7NsnUl0tUlzcE8Qcc4zIDTeIbNzYZ3e1rznWzUoBbzRavMfslaBkqb1+c20mojg1NjZi0qRJMffzeDymrOisdr2Y5ubmhNeH8T9Ha2tr2OJtgG82x/bt29GvX7+EnkM3bW2+gnYPPQTs3evblpfnqwtz441AhFkoal6zX+9Vr8Pdn+x7oLdYr1fta0jVdZnUSOfXrhW112/Tp2a/8soruOSSSzBixAgoioLnnnsu5P6ZM2dCUZSQ24UXXmhOY4lgTp2PeBhRVE1NfY0vvvgCY3SstxK3jz4Crr0WGDkSuPtuYO9eyNix2DhnDpYvXozG886DNycn4sPjqSkS60eTFu+B3rQqE+BwOFBaWopp06ahtLQ0bS7maVO2wCJMD2YOHDiAcePG4cEHH4y4z4UXXoi2trbArba21sAWEoUyus5HvIwKtiLV1wimZ70VVUSANWuAiy4CTj0VeOIJ4MgR4Nxzsf4Xv8DIgwdx0tKluHLmTFUXm0ivOTc3NySAaW9vV9U8qxdMKysrw6JFi3DssceGbHc6nUmt3p5qehcVrKurQ3l5eZ8fFab/PaQynYe74gJAVq5cGbLtmmuukbKysqSOy5wZ0pKRdT4SYXQ+S2dnZ58kV9PPx+HDIjU1ImecEZrUe9llIuvWJZ1T1HuWWLRZO0a8B3oIl++Sk5MjVVVVTGANEu48ORwOa/092JgtE4AjBTPZ2dmSn58vJ5xwgtxwww2ya9euqMc5dOiQtLe3B24tLS0MZtKIETMHrFw/wuhgy1LJwB0dIkuWiBx3XE8Q07+/yOzZIps2iYi2CdyxjmXXC5odZutZQaTzZJm/hxSQMsFMbW2trFq1St5//31ZuXKlfOMb35Czzz476pfAwoULw354GMykvkjTkZcvX655gGPlmRpGBltqp/AuWLBAvwBz+3aRn/9cJDu7J4gZOlTkv/9bpNePHy2Dr0RmOFk9ILDDbD0rSDSQ9d9sUbbAAlImmOlty5YtAkAaGhoi7sOemfQUz68krWp9WLl+hFHBViIXdM1qrbz/vnh/9CPxHnVUT6XeE08UefRRka++CvsQLWvkVFZWxv3arRLwRmKpnjYLS3aqfrqfP7XUBjNHwWZGjx6NvLw8bN68GZMnTw67T2ZmJjIzMw1uGZnJ6/WioqIi5pRZP38iXrJJjP6ZGlbkcrlQVlam+9RQ/0rfaqYs+yV1/kWA1auBxYuBf/wjMIthLYDFAN7bvx/LcnPhOuaYsA/XKoHb6/Xi6aefVnWspUuXYtiwYbaYnmv12XpWkejr909pLykp0bhFac6Q0EolqOiZaWlpEUVRZNWqVaqPywTg1Jdodz+7y7URrdqrZuf/8GGRP/9ZZNy4QC9MFyDPAHJ2r+NGG8bRKmFZ7WcuPz/fVp8x9syok4pDjFak9vpt+tTs/fv3Y8OGDdiwYQMAoLm5GRs2bMC2bduwf/9+zJ8/H6+//jq2bt2K1atXo6ysDMcffzymTp1qbsPJUhL5lSQ2qPVhF2qmafem+vx3dAD33w+MHg1cfTXw3nuQAQPwx0GDMBbAVQDe7HVcAKisrITX6w05lNvtxpgxY7Br166wTxVP/RS1n7kZM2ZYuiemN39PW6RaOoqioKioKO17FmKdJwB93ndOadeREZFVNJGi22uuuUYOHjwoF1xwgeTn58vRRx8tI0eOlFmzZsnOnTvjeg72zKS+ZMavUy0Rz8w8nuDnXrBgQXLnf9s2kXnzRLKyepJ6hw0TuesuaVq1Ku7eAzU5VfHks6RyD4aVZ+tZSazzpMfEg3SjSwLwtm3bkmqUWRjMpD41q/Sm0sUmEiut5J3wxf7dd0X+679EgpJ65RvfEPnDHwJJvfEm8aqZeZKfny+dnZ2qX5+az5ydhzGtPFvPSnie9KVLMDNgwAC544475MCBA0k1zmgMZtJDvHkbqZYzY7XaIHHVu+nuFvnHP0SmTOkJYACR0lKR//s/Ea835NjxBkpq91+6dGlcn4dU78Gw8mw9K+F50o8uwcxrr70m55xzjowYMUKeeOKJZNpnKAYz6SPcr6RIF1KzLvB6fOlZtTZIrIu9+5lnRJ56SuS003oCGIdD5KqrRN58M+Jx4y0MqLYnJ5GeLP4yJ9KPrnVmnnrqKXE6nXLGGWfIK6+8klADjcRgJr30DhhWrFhhiYuNnkNAVs7fCPe6TykslPevvlpkxIieIGbgQJHKSpHmZtXHVdsrEk9OVSKBLn+ZE+lD96J5Bw4ckDvuuEMGDhwol112mfz73/9O9FC6YzBDZl9s9B4C0rIQnB785/+56mrZVl4u3YMG9QQxBQUi99wjsmdP3MdV2ysSb05Vqg1BEtmV2uu3IqKyylUvBw8exDvvvIO6ujo88MAD6NevH26++WbceeedGDRoUCKH1E1HRweys7PR3t4esrItkRG8Xi+Ki4v7rKDr5y+i1dzcnPAU3sbGRkyaNCnmfh6Px5wif++845te/eyzgH+69CmnAPPmAdOmAUkUufR6vaoKA7rdbpSXlwOA6uJ+pp0vIgKg/vodVwXghx9+GG+++SbefPNNfPLJJ8jIyMCpp56KG264AePGjcMzzzyDk08+GW63G2eddVbSL4IoFTQ1NUUMZACE1FtJ9MIZqwqvKVVHRYAXX/RV6l2zpmf75Mm+IGbqVCBKjQ611FZh9tfCqaioiPp+BEv3KrdEdhFX0by77roL7e3t+NGPfgSPx4O9e/fi7bffxoMPPojrr78ea9aswQ033ICZM2fq1FxKd16vF42NjaitrUVjY2OfomhWZER5eIfDgerqagDoU8QrnkJwmujsBJ54AjjtNOCii3yBjMMBzJjh66FpaAAuvFCTQCZeLpcLW7duxdKlS1Xtr3bpAyIymdbjWzt37pSMjAytD5sU5sykBivVUImHkcm5ps6s2bNH5O67fTkw/nyYQYNE5s4V+c9/9H/+OMQ7G4qIzKF7zkyU4AivvPIKzjvvPC0PmxTmzNifP9+h98fV3+tg5RLh/pyZWENAyeTM9H4+vReXDLF1K7BsGfD448CBA75thYVARQUwaxYwZIh+z52ESDk0dvhMEaULtddvzYMZK2IwY29GJNDqLRUvnN433sDuX/4SeY2NyPC/ptNP9+XDXHkl0K+fuQ1Uwe1298mhKSoqwrJly2z3fmhJTUBseNBMaUn19VvfDiJr4DCTvRk5TKPnFO6UKK7m9Yr8/e/y+SmnhFTq/Qcg0/PypL6uzuwWxs3saftWo2Y4165DvmQ/pg0zWRF7ZuyttrYW06dPj7lfTU0Npk2blvDzhPuV7nQ6UV1drdmvdL1/zep1fO+BA9i0aBEKamuR3doKADgCoBbA/QDeh717mchHzXAugLiHfNmLQ4liz0wQ9szYmxE9M1Zb1ygRuvxa3rVLPpw2TT7LyAj0wuwF5H8BKWTibEpRsySG0+mMe9kM9uJQMnSvAGwnDGbsTe+ZJ1Zd1ygemgdjW7aI3HSTHMnMDAQx/wFkDiCDDRryI2PFs+SD2vc/FX4kkLnUXr/jqjNDZAa9a6jEU9TOirxeLyoqKsLOlPJvq6ysVFeT5403gCuuAMaOBX73OxzV2Yl3AcwAMAbAUgD7VLSJxebsR8v3rK2tTdvPJVEMDGbIFvzVWwsLC0O2O53OpHM01H6Jr1692pJfvEkHY93dwF//CkycCHz728CKFUB3N7aedBImAzgDQA2ArjjalCrF5qxWpFHP9mj5nhUUFNj+RwLZjN5dRFbAYabUocfMk3i616041p/wIpMHD4o88ojICScEhpI6AfkjIKcmOLxghyE5tayW66F3e9QM5/pzZtQM+Vp98VOyB+bMBGEwQ9HEs6KyFcf6406Q/uILkV//WiQ/vyeIGTBA7gFkRBJ5ElY8N/EIDpSrqqos9RqNyj3xP0/v5wp+HjX7iBhbUsGuWBYgNgYzQRjMUCyRvqDt0PsQGoxlCHCeAFd9/d+MnvZ++qnIjTeK9O/fUyPmuOPEe//9cuKIEQkHMf6b7WrmBAnX62GV99/oBHU19ZDU7GPVJSOsEkBYrefPqhjMBGEwk34S+cKK54JmtV+U9fX1ArgE2BaIU3y3bXICLpXt48eLKErPHWecIVJbK3LkSEKzWPwXqKqqKtMvCn6JXqQi9XpY5f03o4dDzblUs4/aXhyjWCWA4Cwv9RjMBGEwk16S+cLq6uqSBQsWqLp4WGmsv75eBOgWwNsrmPGKAq/U41LfhosuElmzRqS7O/BYtbkNVu6FSfQ9j9XrYYX33+65J1apfG2VACIVSkEYicFMEAYzqSGeX4LJfGHZbay/q0vE6QxZXSDkpsArRQN3Sdd7H4Z9vNrXu3TpUsv0wgRL5j1PprZKKvfMaCH477WhoUEaGhpM+/xYKYCw6/tpFgYzQRjM2J+aX95afWFZYaw/niETjydyIBN8i/TdaIXXm6hk3/NEeqXMypmx0/tjleEcPysFEHbvaTMai+ZRyvCvF9O7ZkVrayvKy8vhdrsBaFf8Tu8ifbG43W4UFxdj0qRJmD59OiZNmoTi4uLA6wzxr3+h7TePqzpupHI6Zr/eZCT7nsdbW8WM82G39yfS3+v27dtx2WWX4de//rXh9XrU1pIyotij2s9cqtRqMowhoZXJ2DNjX/H88tb6F48ZY/2qhky6u0VefVXkhz8UURTx4LykembMfL3JSvY9j2davtnnww7vj9ocJKN7aazUM2PHnjYzcZgpCIMZ+4rnS0iPLywjp3HGuhA4ALk+N1e6x48PiVK6vl8mzvxDoijdYYMYRREpKvLl1mj1eq0yvVWL9zzWjBsrzdjq7OyUpUuXyk033SRLly6Vzs7OuB6v9/sWTw6SGUm3VgkgrDbLy8oYzARhMGNf8fzyttoXVrwiXQgGAHIjIJuDI5TMTJFZs0Q++UREfLOZFCV09rU/kFEU3/1asVI+hFbvuR16PZI970a8b/HkIKV7AGGHz5wVMJgJwmDGvuL95R2t+J3Vf/H0vhAMBaQKkF1B0ckuQD744Q9Fdu7s8/j6+r6zmoqKtA9krDC9NVybkr1IWaW3KZxkz7tR71sis8OMGtrxeDxSWVkp+fn5lgkgrPyZswoGM0EYzNhXIr+86+vrJTc3t8++ubm5lg5m/BeCEwF5BJCvgqKSzV/3zgyI8eXf1eXLjamp8f1Xy+9GK01v7S2Vf+Ume96NfN/izUEC9J+1E+6zkZeXJ5WVlQwgbIDBTBAGM/YW7y9vXzXc8F/alu2d6e6WrjVr5KVjjgnpWlkPiAuQjKCArKGhIfAFbOQvOyslUYaTqr9ykz3vRr1vwb0fVumZsWJPIsWHwUwQBjPWotVSA+F+eVu59yCsI0dEli8XOfvsQADjBWQlIOdGuQA4nU6ZP3++obkrrI9hjmTPuxHvW7i/z4yMjKjPp/ffol7fBakaNFsVg5kgDGasI9mlBmJ9iVi99yBg3z6R3/5WZNSo0KTen/5U/vHb3yZcYl/PX5y2Obdf839enn76aVm6dKk8/fTTlrr4qL0oGtUzk2iF52i9H2Z8TuN93fF8Xq2U/J4uGMwEYTBjDUZ0+Vq+92DHDpHbbxc59tieICY3V2ThQpHPPgvs1tXVJQ0NDZKTk5NQQKPHL147zRaLtmioFS4+8VwUkz3vavJYHA5HQudITe9Hbm6uFBYWhmw3Ip9Jj7pTHLIyHoOZIAxm9BNPXRIjhn8aGhpUfYE1NDQk9Txx++gjkWuvFenXryeIOf54kYceEjlwIOxDEpkZEnxbsGCB5j0RVpveGq2NsQI+M4vfxXtRTPa8R5vlF+22fPnyqMdV+xltaGgwfGhGy54Z2w1fpxAGM0EYzOgjnl+XRg1RWCqY6e72TSm6+OLQ+dLf+Y6I2x0y1ShcUJjIukHhblr3RFh55pDaCrRmXXySuSgme97DPb53j0zvm8PhkBUrVkQ8ppV7QrXsSbTbEGsqYTAThMGM9uL9dWnUl56ZX67+gKT2z3+Wj+64Q7rPPDO0ep3LJfLaa30eFykorKqq0iSY0aPXxKpJkPH2Zhl98YmnJyOcZM978OOXLl2q+jxF+uxY/SKvVU+ilYO2VMdgJgiDGW0l8uvSqC89s75c6+vr5cQRI6QCkGb09MIc6ddP5MYbRTZtivi4aEFhbm5u3EMDat+TVBRvb5bRFx+17cvJybFMTgmAiJ8dO+RRadGTaPWgLZUxmAnCYEZbifxha/mlF+3XqRlj2//32GNyDyBfBgUxnwFyJyB5iPyrVm3ypP//kw1o0uHLNlV6ZvzvuZ4Bjdoh2Vjnyg55VFr0aFk9aEtVDGaCMJjRVqJdrlp86anJ05k/f74xF4gPPhDvNddIZ1AQ8ykgswA5RsWXnNoLW1VVVdhfluHqzMT7nsRi1eGkSOySM6MmONW7jfEGM9E+O1bOo9KKHYK2VMRgJgiDGW0l0+WazJeemjydWDNZ5s+fn9yL7+4WaWgQufDCkKTetYBcAogSx7mIdxHNcEGFf/uCBQsSfk+inW871tSw62wmLd6zeMQ7JBerHXYLfBORDkGb1TCYCcJgRlta1L2I90tPzZCM0+nUb4jp8GGRv/xF5Fvf6gliMjLkP+ecI+ck+KtWj6mjWnWD272mRrQ6M1a4+NTX16uuIaRXXk88Q14cQumRDkGblTCYCcJgRntGd7kmW3Ml4V+67e0i99/vW37aH8QMGCBy000imzcnFZAkE4CE+0JN9D3pfazOzs6UqKlh9QrAaod59OqZiWfIy+zgj9IXg5kgDGZi03O9JC1oVXNF9S/dlhaR+fNFsrJ6gphhw0R+8xuRXbsCuyXbI5JIABJt+Cfe9yTSisJmXmTThRWSSmMV07P6SvOU+hjMBGEwE92KFSskPz8/5EssJydHqqqqYn6RGtXlaljPzIYNIldfLXLUUT1BzEkniTz+uMhXX4V9iBbVWdUGIGqGf9S+J/HmbiQUFFJUVkgqDff5y83NVfX3T6Q3BjNBGMxEFm3mj5V+mcXTJR7pFvGXbne3yEsviVxwQWil3vPOE/nb30S83pjtS7aXSk0AouW0c7Wzftgzoz8rJJUyD4SsSu31WxERQYrr6OhAdnY22tvbkZWVZXZzLGPFihW44oorYu6nKArq6urgcrkMaFVkbrcb5eXlAIBEPrZ9Xsfhw8CzzwKLFwPvv+/blpEBXH45cOutwNlnx3V8r9eLpqYmtLW1oaCgACUlJXA4HBG3x6uxsRGTJk2KuZ/H40FpaakmxwpHURQ4nU40Nzcn9DqoL60+I0SpRu31+ygD20QW4vV6ceONN6raV0RQWVmJsrIyU79gXS4X6urqUFFRge3bt8f12Pz8fDz88MO+QKa9HXj0UaC6Gmht9e0wcCDwk58AFRXAqFG+i0tjY+Di8p3vfAfr1q2LerFxOBx9ggi3292nvU6nE9XV1XEHh21tbZrtp/ZYvSmKAgBYtmwZL7ZEZB0G9BKZjsNMfSWSg2KVYYXgWSpXX321qrY//fTTIv/5j8jcuSKDB/cMJQ0fLvI//yOye3fg+GoW5FNTb0Xr6c1aTuVWe6zeuVRGDn+ky9CHXev5EBmBOTNBGMz0lcjsICslfEarI9L79k1Adk6ZIuJw9AQxJ58s8sc/ihw61Oe4aquzxprurPX0Zq2XhFBzrM7OTlMCinS5wNu9ng+R3hjMBGEw01ciPTMNDQ2W+KWsNuCYCsjLQGhS7/nnizz/vC/pt5d4k2K1WKpgwYIFcZ1LLWe/WGEmTbR2pfoF3ox1xIjshsFMEAYzfcVz4fYveFhYWGj6L+VY7T4akB8B8n5QAOPNyBCZNk3krbeiHjvR6d/JLFWQyLnUcvZLMsfq6uqShoYGWbBggSxYsEAaGhqSvvCm0wVey2FDolTFYCYIg5nw1K5hE+0+o38pR7oAZAPyc0C2BwUx+xRF/vX974ts3arq2IkW5ktmqYJEz6WW+SSJFkz0r+gdfEt2Kn86XeATXbCVKJ0wmAnCYCayWLknTqcz6hoyRv9S7n0BOA6QJYB0BAUx2wF58fzzpSuoUq8aWvbMJFIXxy69DvX19TFfS7SAJlrwlE4X+HQK3IgSxWAmCIOZ6IIvLg0NDdLQ0BC40CxcuNBSX7j+C8AZgNQAciQoiHkPkKvhG2pKpD3xBiCJLlVglXOZCLXDk06nM+x5iZXYm04XeCssZ0BkdQxmgjCYSYyaX+CG/lL2eqXrr3+VVzMzQ5J6XwLkAo0uAGoDkGSWKrDEuUxQPL1XvQMOtUsxpNMF3qpJ2ERWofb6nQGTvfLKK7jkkkswYsQIKIqC5557LuR+EcGdd96JgoIC9O/fH1OmTMGmTZvMaWwa8Xq9qKioUL3/0KFD9WtMZyfwxz8Cp50Gxw9+gHM7O3EEwJ8BfBPABQBeQmhBN8BX5ba2thaNjY3wer2qnspfmK+wsDBke+8CcU6nU1VVZJfLha1bt8Lj8WDBggWq2lBQUKBqPzPEU2wveF//50nCVG72b6usrAQAVFdXA+h5P/1SsWBfpM+b2s8XEX3NiMgqmueff15+9atfidvtFgCycuXKkPvvueceyc7Olueee07ee+89+cEPfiCjRo2SryIs+hcOe2biF2/+SENDg/aN2L1b5K67fIXt/D0xgweLzJsnzz/ySMRZOFrUKOmd16FFvZVU6HVItGcm3uEjK6xXZKR0KRBIFC9bDjP1Dma6u7tl+PDhct999wW27d27VzIzM6W2tlb1cRnMxC/emT2aDo38+98it9wiMnBgTxBTWChy330ie/cGdgt3AbB6jRK7DyskmjOTSGIvL/BEpPb6bem1mZqbm7Fz505MmTIlsC07Oxvjx4/H+vXrcdVVV4V9XGdnJzo7OwP/7ujo0L2tqSbeoY5Eh0aCF9g7ob0d31y9GhluN5TubgCAjBsHZd48eC+7DE1vvIG2558PWRspeC2kWEMZiqJovsZUuAUCAURcNDDS+lJOpxPLli2z/LCCw+FAdXU1Lrvssqj7VVdXh5xjtZ+P4P3CrXVFRBSWIaGVSujVM/Paa68JANmxY0fIfpdffrlcccUVEY8TaQYOe2bUU/sLPJmhkfr6eikqLJTvA9IIhCT1vgDIZECchYUyf/58VcNGRs+ECTcUkpubK4MHD47ZVrv3OsRbZyYVhtiIyHgpMcyUaDBz6NAhaW9vD9xaWlpSOpiJdWFM9MKpZjZTokMjK2tr5SeAfBwUwBwG5ElATlMRkIQbljGyRonaJRWSPU9WFm8FYLsPsRGR8VIimNmyZYsAkHfffTdkv4kTJ8ott9yi+ripnDMTK9k1mWTYWMFMQtVed+0Sb1WVfJ6REQhi9gJyDyCFKgODSL/mjeqZiXcNJ/+NPQ/pl9hLRMlJiWDGnwC8ePHiwLb29nYmAH8tVrLr/PnzE06GVXPB7p3k2dUl4vGI1NT4/hty3d68WWT2bJH+/QNBzH8AqQRkcJxBQaTgxKihjEQrBWsRSKUCuw+xEZFxbJMAvH//fmzevDnw7+bmZmzYsAE5OTk47rjjUFlZid/85jcYO3YsRo0ahTvuuAMjRozAD3/4Q/MarbNwSaW9E1bVJLsuWbIk4WTYpqamkATVcLZv346mpiaUlpbC7QYqKoDghzidQPXsjXC99SvA7faFMAD2FBdj9tatqAPQpeJ8xNLa2gqgJzm1vLw87H4igquuuirp5N94aq1o+dhUwcReItKc/nFVdJF+5V5zzTUi4uudueOOO2TYsGGSmZkpkydPlo0bN8b1HHbqmVE7LNTQ0JBUbwZi9BTEk39SXy+iKCH5uwKIKPCKAq/U41Lfhu99T2T1avGsWaNJ2/23/Pz8kPMzf/78iPtqkZvBnhkiImPYcphJL3YJZtTWSKmvr4+6+GM8t0jJsGov2A0NHnE6+wYywQFN0cBd0rXhg8CxE1mEMdotXDn8aPsmO9SUaM5Mfn4+h1SIiOJgm+UMyEdtufe6ujqUl5djz549mjxvpPofJSUlcDqdfUrK+ymKgqKiIgAliDYaJchAy4FcNH15amCbfzjIf5xkBZ+fxsbGqMNjIoKWlhY0NTUl/HzB7Y/Hgw8+mDJl+ImIrITBjEXEylHxX4RvvPHGsAFPb4qiRL1w+oMRf5G33qIFHP5/P/bzn+Pze56K2RYA6J0qonYNpKKiIsyfPx95eXlRj+8/P42NjSrbk1zuisvlwooVK1QHJ/Pnz8fll1+e1HMSEVF4DGYsQu3F9YsvvlB9zLlz50JRlIQX7IsUcJTl56PlnHMw9ZZbUNDwJ1VtCdcBFLwIY01NDTweDw4ePBjy7+bmZtx7772BxSO1osVijuXl5aitrY26T1ZWFlasWIF777036ecjIqII9B7vsgI75Mwkk1Ta+xZc/0WLuh5dXV3iaWiQVyorZe8pp4QkxXRd/ANx5n8litIdPmdGESkq6jVNW8fz09DQYHil2UiVgKuqqpgjQ0SUBLXXb0VExZiFzXV0dCA7Oxvt7e3Iysoyuzlheb1eFBcXo7W1NewwkqIoyMvLU9Uzc/vtt+O73/1uYEq3mqneER08CDz5JLBkCbBli29bv37A1VcDc+cCJ58Mtxvwz4YObrq/Q6iuDkh2ySE158fpdKK5uRmrVq0KTM8O3tffI/Xss88iPz8/sfMRpX0Jn2MiIgpL9fVb/7jKfHbomRGJXiJfURRZvnx5XLOA1Fb6Deuzz0TuuEMkN7enm+XYY0V+9SuRtrYwbZc+s5qKinzb/ZItlhZPOfxIPVJq13kKh8XeiIiMxanZQewSzIj4aqQ4HI6Qi63D4ZD58+eLSOQLeqQAKO66Kp9+KnL99SKZmT1RyahRIg88ILJ/f9SHRqsAnMyyCsFBRFVVlephs97Bx4oVKxKuiJxM+4mIKDEMZoLYJZiJp86M2jonqnJEurtFXnlF5Ac/CO1aOecckRUrkk54Ufu6Ij2292stLCyUqqqquHpIkqk/k0z7iYgocQxmgtghmIn3Yuvvdbj99ttVBTVhK892dfmClfHjQ4OYH/zAF9x0dxv+uoJpGUQkugilEUX4iIgoPBbNsxm1dWb8xd4cDgf27NmDhx9+WNXxQ6Z+HzgA/O53wNixwOWXA2+8AWRmAtdfD3zyCbBqFVBS0pPBq5LX60VjYyNqa2vR2NgYSIpNpIid2iKCXq9XVdvUTn3vvV+i7SciIuOYvtAk+cR7sXW73SgvL1dVQA/4uq7Kzp2+IOb3vwe+/NJ3R24uMHu27zZ0aEJt97enoqIi5MLvdDojLvrYWzJBhJpFC9XWlem9X6JBEBERGYfBjEXEc7GN1mvRm6IoKB02DBOffhr485+Bw4d9d4wZ45taPXMmMGBAEi2PHFi1traqLnandxDhX54h1tTu3hWREw2CiIjIOBxmsgi1ayGVlJTE7LXwmwhglQjW7NyJjD/8wRfIfPvbQH09sHEjcOONSQcyaoaDHA6HqtcVTOsgQs3yDOEqIsfzvhARkTkYzFhEPBfbaL0RDgBXAPgngLUALvEdALj0UuC114D1630V7DQq6KYmsPJ6vRF7QwDjgohIyzM4nU7U1dXBFaayX6JBEBERGYfBjIVEutjm5eXh2WefDVxsw/VGDARwC4DNAJ4FcDYAb79+wA03AJ9+CrjdwHe+o3mbk8kVMSOICLceVHNzc9g2BD8m3iCIiIgMpN+EKuuww9TsYCtWrJD8/PyIBdr804UVRZHhgNwFyJ6gqdWfA3J/VpZ0hanUq7VE15RSu26RFmtLaYUVgImIjMW1mYLYYW0mv0jJtP7eCH9PwMvLlmH7nDmYAaDf1/v8C8ASAH8C8HR9vSE9BrHWTAoneB0lNT0rXPeIiCg9qb1+M5ixEH9gECkHRQFwRX4+as88E8qLLwa2vwrgPgB/A+AsKsKyZcsMHfrwB2AAVAc0AODxeFRNqyYiovSk9vrNnBkLiZRMexSAqwC8CeCZL77wBTKKAlx2GbxNTejyeHBVTQ3WqMj/0EOknJJYWJuFiIi0wDozFtL74j4IwE8AVAIY+fW2gwC2f/e7OOH3vweOPx4OAKUGtjESl8uFsrIyNDU1YfXq1fjNb34T8zGszUJERFpgz4yF+C/uIwDcA6AFwFL4ApnPACwAUASg5Re/AI4/XtPnDrcUQbwcDgdKS0uxaNEi1mYhIiLDMJixkJIhQ7B8wAA0A/gFgCEAPgUwC76A5i4AewDMnDkTbrdbs+d1u90oLi7GpEmTMH36dEyaNAnFxcUJPwdrsxARkZEYzJhNBGhoAC68EI5vfQuXHzyIfugpeHcygMcBdAY9pLW1FeXl5ZoENP7k3d65Osk+B2uzEBGRUTibySxHjgDPPgssXgy8955vW0YGUF4Ozxln4OoHHkBra2vEh8c7vTmcmLOnNHoOTqsmIqJEqL1+MwHYaO3twGOPQaqroXwdRHiPOQbKT36CjDlzgNGjMQnAU2edhSlTpkQ8jERZNVptAKH1ytTh+PNoiIiI9MJgxiDerVvRetttKPjrX3H0V19BAbATwG8BPHzoEAY+9xyqJ02Ca/RoAMDnn3+u6ritra1obGwMBC5ffPEF5s6dGxKkOJ1OVFdX9xna0XplaiIiIjMwmNHbhg3YVlGBgldewXFfb/oYwGIAfwFw+Otte7/OUfHnk6idtlxZWYldu3ZF3ae117H9tF6ZmoiIyAzMmdGDCPDyy8B99/mSe7/mga9S74vwLTLUW3COCoC4lwmIJlz+S6ylCLTImSEiIkoUKwCb4fBh4E9/AsaNA6ZOBRoa0AWgFsBZAM4H8ALCBzJAaI6Kw+HAtGnTNAlkeh/bj1OoiYgoFTCY0cLevcC99wKjRgHXXAN88AEwcCBaystxPIDpAN6O43BtbW1wu91YvHix5k3tnf/CKdRERGR3zJlJxn/+A1RXA489Buzf79tWUABUVADXX49XX3wR/6mri/uwQ4cOxcyZMzXrlQkWLv8leCkCTqEmIiK7YTCTjGuvBdas8f3/KacA8+YB06YBmZkA4k+c9eeoAIg6ZTpR0ZYQ4BRqIiKyKwYzyZgzx7d69bx5vhyZXnknJSUlcDqdqpJ4g3NU1E7LjteSJUvY20JERCmHOTPJ+P73A0sR9A5kAF9vx5IlS1QNFwXnqGzatEl1EyIt5hhOXl6e6n2JiIjsgj0zOnK73Zg7d27Y+5xOJ2bNmoWxY8eG5Kh4vV489thjMY/tdDqxZMmSPgXyomHxOyIiSkUMZnTiX8AxUq/M0qVLUV5e3md7rCUG/M4991xcfvnlcLlceOCBBzBnzpyYj2HxOyIiSkUcZtKB1+tFRUVFxEBGURTMnTsXXq+3z31qe09efvlleL1eOBwO3HzzzXA6nRGHnBRFiZr8S0REZGcMZnQQzwKOvantPdmzZ0/g8Sx+R0RE6YzBjA6SWcCxpKQEOTk5cT+exe+IiChdMZjRQTILODocDlRUVCT0eJfLha1bt8Lj8aCmpgYejwfNzc0MZIiIKKVxoUkdJLuAo9frxbBhw7B79+6wx+cCkERElA640KSJks1hcTgcePTRR8PexxwYIiKiUAxmdJJMDovX60VOTg4qKyv7FLpjDgwREVEoDjPpzOv1xrWAo9vtRkVFRchsqPz8fMyYMQNlZWVcAJKIiNKG2us3i+bpLJ4FHCMV2tu1axeqq6sZyBAREYXBnhkDqOmd8ScNR6pPw6RfIiJKN+yZsYhww0ZOpxPV1dVwuVyBQGf16tWqC+2p7ekhIiJKBwxmdBRp2Ki1tRXl5eWYN28eamtrVS8UCXCxSCIiot4YzOjE6/Xi+uuvD1tnxr/tvvvui/u4XCySiIgoFIMZndx1110Ri94lwp8zw8UiiYiIQrHOjA4OHz6MxYsXa3Y8FsojIiKKjMGMxtxuN5xOJ/bt26fZMVkoj4iIKDIOM2koUsJvohYsWIDJkyezvgwREVEUrDOjkVh1YuLlcDhw8OBB9OvXT5PjERER2Q0XmjRYU1OTZoEM4AuO1q1bp9nxiIiIUpXlg5lFixZBUZSQ20knnWR2s/rQo/4La8oQERHFZvlgBgBOOeUUtLW1BW6vvvqq2U3qQ239l/z8fFRVVWl6TCIionRmi2DmqKOOwvDhwwO3vLw8s5vUR0lJCZxOZ2AadThZWVl4+umncdttt0XdV1EUFBUVsaYMERGRCrYIZjZt2oQRI0Zg9OjRmDFjBrZt2xZ1/87OTnR0dITc9OZwOFBdXQ0AEYOUjo4OTJ06FWPGjMG0adPC7suaMkRERPGxfDAzfvx4PPnkk3jxxRfx0EMPobm5GSUlJVHruNx9993Izs4O3IqKigxpq8vlQl1dHQoLC6Pu19raisWLF2PevHl99mVNGSIiovjYbmr23r17MXLkSCxZsgTXXXdd2H06OzvR2dkZ+HdHRweKiop0nZodzOv1orGxEVdccQX27NkTdh//8gSbN2/GunXr0NbWhoKCAtaUISIi+praqdm2K5o3ZMgQnHDCCdi8eXPEfTIzM5GZmWlgq0I5HA44HI6IgQzgW2yypaUF69atQ2lpqXGNIyIiSjGWH2bqbf/+/diyZYvlZ/qonVbN6ddERETJsXwwM2/ePKxduxZbt27FunXrcOmll8LhcAQSaK1KbbBl9aCMiIjI6iw/zLR9+3ZMmzYNu3fvRn5+Pv7f//t/eP3115Gfn29206LyT9VubW0Nu1aTP2eG06+JiIiSY/lg5plnnjG7CQnxT9UuLy+HoighAQ2nXxMREWnH8sNMdhZpqjanXxMREWnHdlOzE2HEqtnReL1eNDU1cfo1ERFRHFJ2arYdORwOTr8mIiLSCYeZiIiIyNYYzBAREZGtMZghIiIiW2MwQ0RERLbGYIaIiIhsjcEMERER2RqDGSIiIrI1BjNERERkawxmiIiIyNZYAVhDXLaAiIjIeAxmNOJ2u1FRUYHt27cHtjmdTlRXV3NBSSIiIh1xmEkDbrcb5eXlIYEMALS2tqK8vBxut9uklhEREaU+BjNJ8nq9qKioQLjFx/3bKisr4fV6jW4aERFRWmAwk6SmpqY+PTLBRAQtLS1oamoysFVERETpg8FMktra2jTdj4iIiOLDYCZJBQUFmu5HRERE8WEwk6SSkhI4nU4oihL2fkVRUFRUhJKSEoNbRkRElB4YzCTJ4XCguroaAPoENP5/L1u2jPVmiIiIdMJgRgMulwt1dXUoLCwM2e50OlFXV8c6M0RERDpSJNyc4hTT0dGB7OxstLe3IysrS7fnYQVgIiIi7ai9frMCsIYcDgdKS0vNbgYREVFa4TATERER2RqDGSIiIrI1BjNERERkawxmiIiIyNYYzBAREZGtMZghIiIiW2MwQ0RERLbGYIaIiIhsjcEMERER2RorACeISxcQERFZA4OZBLjdblRUVGD79u2BbU6nE9XV1VxUkoiIyGAcZoqT2+1GeXl5SCADAK2trSgvL4fb7TapZUREROmJwUwcvF4vKioqEG6hcf+2yspKeL1eo5tGRESUthjMxKGpqalPj0wwEUFLSwuampoMbBUREVF6YzATh7a2Nk33IyIiouQxmIlDQUGBpvsRERFR8hjMxKGkpAROpxOKooS9X1EUFBUVoaSkxOCWERERpS8GM3FwOByorq4GgD4Bjf/fy5YtY70ZIiIiAzGYiZPL5UJdXR0KCwtDtjudTtTV1bHODBERkcEUCTfPOMV0dHQgOzsb7e3tyMrK0uSYrABMRESkL7XXb1YATpDD4UBpaanZzSAiIkp7HGYiIiIiW2MwQ0RERLbGYIaIiIhsjcEMERER2RqDGSIiIrI1BjNERERkawxmiIiIyNYYzBAREZGtMZghIiIiW0uLCsD+FRs6OjpMbgkRERGp5b9ux1p5KS2CmX379gEAioqKTG4JERERxWvfvn3Izs6OeH9aLDTZ3d2NHTt2YPDgwVAURZNjdnR0oKioCC0tLZotXpkKeF764jnpi+ckPJ6XvnhO+kqncyIi2LdvH0aMGIGMjMiZMWnRM5ORkQGn06nLsbOyslL+w5QInpe+eE764jkJj+elL56TvtLlnETrkfFjAjARERHZGoMZIiIisjUGMwnKzMzEwoULkZmZaXZTLIXnpS+ek754TsLjeemL56QvnpO+0iIBmIiIiFIXe2aIiIjI1hjMEBERka0xmCEiIiJbYzBDREREtsZgJkEPPvggiouLccwxx2D8+PH45z//aXaTDLNo0SIoihJyO+mkkwL3Hzp0CLNnz0Zubi4GDRqEyy67DJ999pmJLdbeK6+8gksuuQQjRoyAoih47rnnQu4XEdx5550oKChA//79MWXKFGzatClknz179mDGjBnIysrCkCFDcN1112H//v0GvgrtxTovM2fO7PPZufDCC0P2SbXzcvfdd+Pss8/G4MGDMXToUPzwhz/Exo0bQ/ZR8zezbds2XHzxxRgwYACGDh2K+fPno6ury8iXohk156S0tLTPZ+WGG24I2SeVzslDDz2E008/PVAIb8KECXjhhRcC96fbZyReDGYS8Oyzz2Lu3LlYuHAh3nnnHYwbNw5Tp07F559/bnbTDHPKKaegra0tcHv11VcD982ZMwd/+9vfsGLFCqxduxY7duyAy+UysbXaO3DgAMaNG4cHH3ww7P333nsvfvvb3+Lhhx/GG2+8gYEDB2Lq1Kk4dOhQYJ8ZM2bgo48+wssvv4y///3veOWVV3D99dcb9RJ0Eeu8AMCFF14Y8tmpra0NuT/VzsvatWsxe/ZsvP7663j55Zdx5MgRXHDBBThw4EBgn1h/M16vFxdffDEOHz6MdevW4amnnsKTTz6JO++804yXlDQ15wQAZs2aFfJZuffeewP3pdo5cTqduOeee/D222/jrbfewvnnn4+ysjJ89NFHANLvMxI3obidc845Mnv27MC/vV6vjBgxQu6++24TW2WchQsXyrhx48Let3fvXjn66KNlxYoVgW2ffPKJAJD169cb1EJjAZCVK1cG/t3d3S3Dhw+X++67L7Bt7969kpmZKbW1tSIi8vHHHwsAefPNNwP7vPDCC6IoirS2thrWdj31Pi8iItdcc42UlZVFfEw6nJfPP/9cAMjatWtFRN3fzPPPPy8ZGRmyc+fOwD4PPfSQZGVlSWdnp7EvQAe9z4mIyHnnnScVFRURH5Pq50RE5Nhjj5XHH3+cnxEV2DMTp8OHD+Ptt9/GlClTAtsyMjIwZcoUrF+/3sSWGWvTpk0YMWIERo8ejRkzZmDbtm0AgLfffhtHjhwJOT8nnXQSjjvuuLQ5P83Nzdi5c2fIOcjOzsb48eMD52D9+vUYMmQIzjrrrMA+U6ZMQUZGBt544w3D22ykxsZGDB06FCeeeCJ+9rOfYffu3YH70uG8tLe3AwBycnIAqPubWb9+PU477TQMGzYssM/UqVPR0dER+OVuZ73Pid9f/vIX5OXl4dRTT8Uvf/lLHDx4MHBfKp8Tr9eLZ555BgcOHMCECRP4GVEhLRaa1NKuXbvg9XpDPjAAMGzYMHz66acmtcpY48ePx5NPPokTTzwRbW1tqKqqQklJCT788EPs3LkT/fr1w5AhQ0IeM2zYMOzcudOcBhvM/zrDfUb89+3cuRNDhw4Nuf+oo45CTk5OSp+nCy+8EC6XC6NGjcKWLVtw++2343vf+x7Wr18Ph8OR8uelu7sblZWVOPfcc3HqqacCgKq/mZ07d4b9PPnvs7Nw5wQApk+fjpEjR2LEiBF4//338Ytf/AIbN26E2+0GkJrn5IMPPsCECRNw6NAhDBo0CCtXrsTJJ5+MDRs2pPVnRA0GMxS3733ve4H/P/300zF+/HiMHDkSy5cvR//+/U1sGVndVVddFfj/0047DaeffjrGjBmDxsZGTJ482cSWGWP27Nn48MMPQ3LM0l2kcxKcJ3XaaaehoKAAkydPxpYtWzBmzBijm2mIE088ERs2bEB7ezvq6upwzTXXYO3atWY3yxY4zBSnvLw8OByOPlnkn332GYYPH25Sq8w1ZMgQnHDCCdi8eTOGDx+Ow4cPY+/evSH7pNP58b/OaJ+R4cOH90kY7+rqwp49e9LmPAHA6NGjkZeXh82bNwNI7fNy00034e9//zs8Hg+cTmdgu5q/meHDh4f9PPnvs6tI5ySc8ePHA0DIZyXVzkm/fv1w/PHH48wzz8Tdd9+NcePGobq6Oq0/I2oxmIlTv379cOaZZ2L16tWBbd3d3Vi9ejUmTJhgYsvMs3//fmzZsgUFBQU488wzcfTRR4ecn40bN2Lbtm1pc35GjRqF4cOHh5yDjo4OvPHGG4FzMGHCBOzduxdvv/12YJ81a9agu7s78KWdDrZv347du3ejoKAAQGqeFxHBTTfdhJUrV2LNmjUYNWpUyP1q/mYmTJiADz74ICTQe/nll5GVlYWTTz7ZmBeioVjnJJwNGzYAQMhnJZXOSTjd3d3o7OxMy89I3MzOQLajZ555RjIzM+XJJ5+Ujz/+WK6//noZMmRISBZ5Krv11lulsbFRmpub5bXXXpMpU6ZIXl6efP755yIicsMNN8hxxx0na9askbfeeksmTJggEyZMMLnV2tq3b5+8++678u677woAWbJkibz77rvyn//8R0RE7rnnHhkyZIisWrVK3n//fSkrK5NRo0bJV199FTjGhRdeKN/61rfkjTfekFdffVXGjh0r06ZNM+slaSLaedm3b5/MmzdP1q9fL83NzdLQ0CBnnHGGjB07Vg4dOhQ4Rqqdl5/97GeSnZ0tjY2N0tbWFrgdPHgwsE+sv5muri459dRT5YILLpANGzbIiy++KPn5+fLLX/7SjJeUtFjnZPPmzfLrX/9a3nrrLWlubpZVq1bJ6NGjZeLEiYFjpNo5ue2222Tt2rXS3Nws77//vtx2222iKIq89NJLIpJ+n5F4MZhJ0AMPPCDHHXec9OvXT8455xx5/fXXzW6SYa688kopKCiQfv36SWFhoVx55ZWyefPmwP1fffWV3HjjjXLsscfKgAED5NJLL5W2tjYTW6w9j8cjAPrcrrnmGhHxTc++4447ZNiwYZKZmSmTJ0+WjRs3hhxj9+7dMm3aNBk0aJBkZWXJj3/8Y9m3b58Jr0Y70c7LwYMH5YILLpD8/Hw5+uijZeTIkTJr1qw+PwJS7byEOx8A5Iknngjso+ZvZuvWrfK9731P+vfvL3l5eXLrrbfKkSNHDH412oh1TrZt2yYTJ06UnJwcyczMlOOPP17mz58v7e3tIcdJpXNy7bXXysiRI6Vfv36Sn58vkydPDgQyIun3GYmXIiJiXD8QERERkbaYM0NERES2xmCGiIiIbI3BDBEREdkagxkiIiKyNQYzREREZGsMZoiIiMjWGMwQERGRrTGYISIiIltjMENERES2xmCGiIiIbI3BDBEREdkagxkisqXa2lr0798fbW1tgW0//vGPcfrpp6O9vd3ElhGR0bjQJBHZkojgm9/8JiZOnIgHHngACxcuxB//+Ee8/vrrKCwsNLt5RGSgo8xuABFRIhRFwV133YXy8nIMHz4cDzzwAJqamhjIEKUh9swQka2dccYZ+Oijj/DSSy/hvPPOM7s5RGQC5swQkW29+OKL+PTTT+H1ejFs2DCzm0NEJmHPDBHZ0jvvvIPS0lI88sgjePLJJ5GVlYUVK1aY3SwiMgFzZojIdrZu3YqLL74Yt99+O6ZNm4bRo0djwoQJeOedd3DGGWeY3TwiMhh7ZojIVvbs2YPvfOc7KC0txcMPPxzYfvHFF8Pr9eLFF180sXVEZAYGM0RERGRrTAAmIiIiW2MwQ0RERLbGYIaIiIhsjcEMERER2RqDGSIiIrI1BjNERERkawxmiIiIyNYYzBAREZGtMZghIiIiW2MwQ0RERLbGYIaIiIhs7f8DE9NwkqIC3cgAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "fig, ax = plt.subplots()\n", + "plt.scatter(adv[\"TV\"], adv[\"Sales\"], color=\"black\")\n", + "\n", + "plt.xlabel(\"$x$\")\n", + "plt.ylabel(\"$y$\")\n", + " \n", + "X_line = np.arange(np.min(adv[\"TV\"]),np.max(adv[\"TV\"])*1.1, 0.1)\n", + "Y_line = predict(adv[\"TV\"], adv[\"Sales\"], parameters_simple, X_line)\n", + "ax.plot(X_line, Y_line, \"r\")\n", + "ax.plot(X_pred, Y_pred, \"bo\")\n", + "plt.plot()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now let's increase the number of the input nodes to build a multiple linear regression model." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "## 3 - Multiple Linear Regression" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 3.1 - Multipe Linear Regression Model\n", + "\n", + "You can write a multiple linear regression model with two independent variables $x_1$, $x_2$ as\n", + "\n", + "$$\\hat{y} = w_1x_1 + w_2x_2 + b = Wx + b,\\tag{7}$$\n", + "\n", + "where $Wx$ is the dot product of the input vector $x = \\begin{bmatrix} x_1 & x_2\\end{bmatrix}$ and the parameters vector $W = \\begin{bmatrix} w_1 & w_2\\end{bmatrix}$, scalar parameter $b$ is the intercept. The goal of the training process is to find the \"best\" parameters $w_1$, $w_2$ and $b$ such that the differences between original values $y_i$ and predicted values $\\hat{y}_i$ are minimum for the given training examples." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 3.2 - Neural Network Model with a Single Perceptron and Two Input Nodes\n", + "\n", + "To describe the multiple regression problem, you can still use a model with one perceptron, but this time you need two input nodes, as shown in the following scheme:\n", + "\n", + "\n", + "\n", + "The perceptron output calculation for every training example $x^{(i)} = \\begin{bmatrix} x_1^{(i)} & x_2^{(i)}\\end{bmatrix}$ can be written with dot product:\n", + "\n", + "$$z^{(i)} = w_1x_1^{(i)} + w_2x_2^{(i)} + b = Wx^{(i)} + b,\\tag{8}$$\n", + "\n", + "where weights are in the vector $W = \\begin{bmatrix} w_1 & w_2\\end{bmatrix}$ and bias $b$ is a scalar. The output layer will have the same single node $\\hat{y} = z$.\n", + "\n", + "Organise all training examples in a matrix $X$ of a shape ($2 \\times m$), putting $x_1^{(i)}$ and $x_2^{(i)}$ into columns. Then matrix multiplication of $W$ ($1 \\times 2$) and $X$ ($2 \\times m$) will give a ($1 \\times m$) vector\n", + "\n", + "$$WX = \n", + "\\begin{bmatrix} w_1 & w_2\\end{bmatrix} \n", + "\\begin{bmatrix} \n", + "x_1^{(1)} & x_1^{(2)} & \\dots & x_1^{(m)} \\\\ \n", + "x_2^{(1)} & x_2^{(2)} & \\dots & x_2^{(m)} \\\\ \\end{bmatrix}\n", + "=\\begin{bmatrix} \n", + "w_1x_1^{(1)} + w_2x_2^{(1)} & \n", + "w_1x_1^{(2)} + w_2x_2^{(2)} & \\dots & \n", + "w_1x_1^{(m)} + w_2x_2^{(m)}\\end{bmatrix}.$$\n", + "\n", + "And the model can be written as\n", + "\n", + "\\begin{align}\n", + "Z &= W X + b,\\\\\n", + "\\hat{Y} &= Z,\n", + "\\tag{9}\\end{align}\n", + "\n", + "where $b$ is broadcasted to the vector of size ($1 \\times m$). These are the calculations to perform in the forward propagation step. Cost function will remain the same (see equation $(4)$ in the section [1.2](#1.2)):\n", + "\n", + "$$\\mathcal{L}\\left(w, b\\right) = \\frac{1}{2m}\\sum_{i=1}^{m} \\left(\\hat{y}^{(i)} - y^{(i)}\\right)^2.$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "To implement the gradient descent algorithm, you can calculate cost function partial derivatives as:\n", + "\n", + "\\begin{align}\n", + "\\frac{\\partial \\mathcal{L} }{ \\partial w_1 } &= \n", + "\\frac{1}{m}\\sum_{i=1}^{m} \\left(\\hat{y}^{(i)} - y^{(i)}\\right)x_1^{(i)},\\\\\n", + "\\frac{\\partial \\mathcal{L} }{ \\partial w_2 } &= \n", + "\\frac{1}{m}\\sum_{i=1}^{m} \\left(\\hat{y}^{(i)} - y^{(i)}\\right)x_2^{(i)},\\tag{10}\\\\\n", + "\\frac{\\partial \\mathcal{L} }{ \\partial b } &= \n", + "\\frac{1}{m}\\sum_{i=1}^{m} \\left(\\hat{y}^{(i)} - y^{(i)}\\right).\n", + "\\end{align}\n", + "\n", + "After performing the forward propagation as shown in $(9)$, the variable $\\hat{Y}$ will contain the predictions in the array of size ($1 \\times m$). The original values $y^{(i)}$ will be kept in the array $Y$ of the same size. Thus, $\\left(\\hat{Y} - Y\\right)$ will be a ($1 \\times m$) array containing differences $\\left(\\hat{y}^{(i)} - y^{(i)}\\right)$. Matrix $X$ of size ($2 \\times m$) has all $x_1^{(i)}$ values in the first row and $x_2^{(i)}$ in the second row. Thus, the sums in the first two equations of $(10)$ can be calculated as matrix multiplication of $\\left(\\hat{Y} - Y\\right)$ of a shape ($1 \\times m$) and $X^T$ of a shape ($m \\times 2$), resulting in the ($1 \\times 2$) array:\n", + "\n", + "$$\\frac{\\partial \\mathcal{L} }{ \\partial W } = \n", + "\\begin{bmatrix} \\frac{\\partial \\mathcal{L} }{ \\partial w_1 } & \n", + "\\frac{\\partial \\mathcal{L} }{ \\partial w_2 }\\end{bmatrix} = \\frac{1}{m}\\left(\\hat{Y} - Y\\right)X^T.\\tag{11}$$\n", + "\n", + "Similarly for $\\frac{\\partial \\mathcal{L} }{ \\partial b }$:\n", + "\n", + "$$\\frac{\\partial \\mathcal{L} }{ \\partial b } = \\frac{1}{m}\\left(\\hat{Y} - Y\\right)\\mathbf{1}.\\tag{12}$$\n", + "\n", + "where $\\mathbf{1}$ is just a ($m \\times 1$) vector of ones.\n", + "\n", + "\n", + "See how linear algebra and calculus work together to make calculations so nice and tidy! You can now update the parameters using matrix form of $W$:\n", + "\n", + "\\begin{align}\n", + "W &= W - \\alpha \\frac{\\partial \\mathcal{L} }{ \\partial W },\\\\\n", + "b &= b - \\alpha \\frac{\\partial \\mathcal{L} }{ \\partial b },\n", + "\\tag{13}\\end{align}\n", + "\n", + "where $\\alpha$ is a learning rate. Repeat the process in a loop until the cost function stops decreasing." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### 3.3 - Dataset\n", + "\n", + "Let's build a linear regression model for a Kaggle dataset [House Prices](https://www.kaggle.com/c/house-prices-advanced-regression-techniques), saved in a file `data/house_prices_train.csv`. You will use two fields - ground living area (`GrLivArea`, square feet) and rates of the overall quality of material and finish (`OverallQual`, 1-10) to predict sales price (`SalePrice`, dollars).\n", + "\n", + "To open the dataset you can use `pandas` function `read_csv`:" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [], + "source": [ + "df = pd.read_csv('data/house_prices_train.csv')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Select the required fields and save them in the variables `X_multi`, `Y_multi`:" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [], + "source": [ + "X_multi = df[['GrLivArea', 'OverallQual']]\n", + "Y_multi = df['SalePrice']" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Preview the data:" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
GrLivAreaOverallQual
017107
112626
217867
317177
421988
.........
145516476
145620736
145723407
145810785
145912565
\n", + "

1460 rows × 2 columns

\n", + "
" + ], + "text/plain": [ + " GrLivArea OverallQual\n", + "0 1710 7\n", + "1 1262 6\n", + "2 1786 7\n", + "3 1717 7\n", + "4 2198 8\n", + "... ... ...\n", + "1455 1647 6\n", + "1456 2073 6\n", + "1457 2340 7\n", + "1458 1078 5\n", + "1459 1256 5\n", + "\n", + "[1460 rows x 2 columns]" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/plain": [ + "0 208500\n", + "1 181500\n", + "2 223500\n", + "3 140000\n", + "4 250000\n", + " ... \n", + "1455 175000\n", + "1456 210000\n", + "1457 266500\n", + "1458 142125\n", + "1459 147500\n", + "Name: SalePrice, Length: 1460, dtype: int64" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "display(X_multi)\n", + "display(Y_multi)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Normalize the data:" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [], + "source": [ + "X_multi_norm = (X_multi - np.mean(X_multi))/np.std(X_multi)\n", + "Y_multi_norm = (Y_multi - np.mean(Y_multi))/np.std(Y_multi)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Convert results to the `NumPy` arrays, transpose `X_multi_norm` to get an array of a shape ($2 \\times m$) and reshape `Y_multi_norm` to bring it to the shape ($1 \\times m$):" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The shape of X: (2, 1460)\n", + "The shape of Y: (1, 1460)\n", + "I have m = 1460 training examples!\n" + ] + } + ], + "source": [ + "X_multi_norm = np.array(X_multi_norm).T\n", + "Y_multi_norm = np.array(Y_multi_norm).reshape((1, len(Y_multi_norm)))\n", + "\n", + "print ('The shape of X: ' + str(X_multi_norm.shape))\n", + "print ('The shape of Y: ' + str(Y_multi_norm.shape))\n", + "print ('I have m = %d training examples!' % (X_multi_norm.shape[1]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 3.4 - Performance of the Neural Network Model for Multiple Linear Regression\n", + "\n", + "Now... you do not need to change anything in your neural network implementation! Go through the code in section [2](#2) and see that if you pass new datasets `X_multi_norm` and `Y_multi_norm`, the input layer size $n_x$ will get equal to $2$ and the rest of the implementation will remain exactly the same, even the backward propagation!\n", + "\n", + "Train the model for $100$ iterations:" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Cost after iteration 0: 0.514220\n", + "Cost after iteration 1: 0.448627\n", + "Cost after iteration 2: 0.396224\n", + "Cost after iteration 3: 0.353227\n", + "Cost after iteration 4: 0.317639\n", + "Cost after iteration 5: 0.288102\n", + "Cost after iteration 6: 0.263566\n", + "Cost after iteration 7: 0.243179\n", + "Cost after iteration 8: 0.226237\n", + "Cost after iteration 9: 0.212158\n", + "Cost after iteration 10: 0.200457\n", + "Cost after iteration 11: 0.190734\n", + "Cost after iteration 12: 0.182654\n", + "Cost after iteration 13: 0.175939\n", + "Cost after iteration 14: 0.170359\n", + "Cost after iteration 15: 0.165721\n", + "Cost after iteration 16: 0.161867\n", + "Cost after iteration 17: 0.158665\n", + "Cost after iteration 18: 0.156003\n", + "Cost after iteration 19: 0.153792\n", + "Cost after iteration 20: 0.151953\n", + "Cost after iteration 21: 0.150426\n", + "Cost after iteration 22: 0.149157\n", + "Cost after iteration 23: 0.148102\n", + "Cost after iteration 24: 0.147225\n", + "Cost after iteration 25: 0.146496\n", + "Cost after iteration 26: 0.145891\n", + "Cost after iteration 27: 0.145388\n", + "Cost after iteration 28: 0.144970\n", + "Cost after iteration 29: 0.144622\n", + "Cost after iteration 30: 0.144334\n", + "Cost after iteration 31: 0.144094\n", + "Cost after iteration 32: 0.143894\n", + "Cost after iteration 33: 0.143728\n", + "Cost after iteration 34: 0.143591\n", + "Cost after iteration 35: 0.143476\n", + "Cost after iteration 36: 0.143381\n", + "Cost after iteration 37: 0.143302\n", + "Cost after iteration 38: 0.143236\n", + "Cost after iteration 39: 0.143182\n", + "Cost after iteration 40: 0.143136\n", + "Cost after iteration 41: 0.143099\n", + "Cost after iteration 42: 0.143067\n", + "Cost after iteration 43: 0.143041\n", + "Cost after iteration 44: 0.143020\n", + "Cost after iteration 45: 0.143002\n", + "Cost after iteration 46: 0.142987\n", + "Cost after iteration 47: 0.142974\n", + "Cost after iteration 48: 0.142964\n", + "Cost after iteration 49: 0.142956\n", + "Cost after iteration 50: 0.142948\n", + "Cost after iteration 51: 0.142943\n", + "Cost after iteration 52: 0.142938\n", + "Cost after iteration 53: 0.142934\n", + "Cost after iteration 54: 0.142930\n", + "Cost after iteration 55: 0.142927\n", + "Cost after iteration 56: 0.142925\n", + "Cost after iteration 57: 0.142923\n", + "Cost after iteration 58: 0.142921\n", + "Cost after iteration 59: 0.142920\n", + "Cost after iteration 60: 0.142919\n", + "Cost after iteration 61: 0.142918\n", + "Cost after iteration 62: 0.142917\n", + "Cost after iteration 63: 0.142917\n", + "Cost after iteration 64: 0.142916\n", + "Cost after iteration 65: 0.142916\n", + "Cost after iteration 66: 0.142915\n", + "Cost after iteration 67: 0.142915\n", + "Cost after iteration 68: 0.142915\n", + "Cost after iteration 69: 0.142914\n", + "Cost after iteration 70: 0.142914\n", + "Cost after iteration 71: 0.142914\n", + "Cost after iteration 72: 0.142914\n", + "Cost after iteration 73: 0.142914\n", + "Cost after iteration 74: 0.142914\n", + "Cost after iteration 75: 0.142914\n", + "Cost after iteration 76: 0.142914\n", + "Cost after iteration 77: 0.142914\n", + "Cost after iteration 78: 0.142914\n", + "Cost after iteration 79: 0.142914\n", + "Cost after iteration 80: 0.142914\n", + "Cost after iteration 81: 0.142914\n", + "Cost after iteration 82: 0.142913\n", + "Cost after iteration 83: 0.142913\n", + "Cost after iteration 84: 0.142913\n", + "Cost after iteration 85: 0.142913\n", + "Cost after iteration 86: 0.142913\n", + "Cost after iteration 87: 0.142913\n", + "Cost after iteration 88: 0.142913\n", + "Cost after iteration 89: 0.142913\n", + "Cost after iteration 90: 0.142913\n", + "Cost after iteration 91: 0.142913\n", + "Cost after iteration 92: 0.142913\n", + "Cost after iteration 93: 0.142913\n", + "Cost after iteration 94: 0.142913\n", + "Cost after iteration 95: 0.142913\n", + "Cost after iteration 96: 0.142913\n", + "Cost after iteration 97: 0.142913\n", + "Cost after iteration 98: 0.142913\n", + "Cost after iteration 99: 0.142913\n", + "W = [[0.3694604 0.57181575]]\n", + "b = [[1.02931362e-16]]\n" + ] + } + ], + "source": [ + "parameters_multi = nn_model(X_multi_norm, Y_multi_norm, num_iterations=100, print_cost=True)\n", + "\n", + "print(\"W = \" + str(parameters_multi[\"W\"]))\n", + "print(\"b = \" + str(parameters_multi[\"b\"]))\n", + "\n", + "W_multi = parameters_multi[\"W\"]\n", + "b_multi = parameters_multi[\"b\"]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now you are ready to make predictions:" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Ground living area, square feet:\n", + "[1710 1200 2200]\n", + "Rates of the overall quality of material and finish, 1-10:\n", + "[7 6 8]\n", + "Predictions of sales price, $:\n", + "[221371. 160039. 281587.]\n" + ] + } + ], + "source": [ + "X_pred_multi = np.array([[1710, 7], [1200, 6], [2200, 8]]).T\n", + "Y_pred_multi = predict(X_multi, Y_multi, parameters_multi, X_pred_multi)\n", + "\n", + "print(f\"Ground living area, square feet:\\n{X_pred_multi[0]}\")\n", + "print(f\"Rates of the overall quality of material and finish, 1-10:\\n{X_pred_multi[1]}\")\n", + "print(f\"Predictions of sales price, $:\\n{np.round(Y_pred_multi)}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Congrats on finishing this lab!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "collapsed_sections": [], + "name": "C1_W1_Assignment_Solution.ipynb", + "provenance": [] + }, + "coursera": { + "schema_names": [ + "AI4MC1-1" + ] + }, + "grader_version": "1", + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.8" + }, + "toc": { + "base_numbering": 1, + "nav_menu": {}, + "number_sections": true, + "sideBar": true, + "skip_h1_title": false, + "title_cell": "Table of Contents", + "title_sidebar": "Contents", + "toc_cell": false, + "toc_position": {}, + "toc_section_display": true, + "toc_window_display": false + } + }, + "nbformat": 4, + "nbformat_minor": 1 +}