04A_MLP_Optimizer_Sandbox_Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Assignment: Multilayer Perceptron (MLP) Optimizer Sandbox"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[Duke Community Standard](http://integrity.duke.edu/standard.html): By typing your name below, you are certifying that you have adhered to the Duke Community Standard in completing this assignment.**\n",
    "\n",
    "Name: [Siyuan, Liu]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multilayer Perceptron (MLP)\n",
    "\n",
    "### Imports and helper functions\n",
    "\n",
    "Let's play around with some optimizers. First some imports and helper functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "# Import data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Helper functions for creating weight variables\n",
    "def weight_variable(shape):\n",
    "    \"\"\"weight_variable generates a weight variable of a given shape.\"\"\"\n",
    "    initial = tf.truncated_normal(shape, stddev=0.1)\n",
    "    return tf.Variable(initial)\n",
    "\n",
    "def bias_variable(shape):\n",
    "    \"\"\"bias_variable generates a bias variable of a given shape.\"\"\"\n",
    "    initial = tf.constant(0.1, shape=shape)\n",
    "    return tf.Variable(initial)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Models\n",
    "\n",
    "And here's the forward pass of the computation graph definition of the completed TensorFlow MLP assignment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Model Inputs\n",
    "x = tf.placeholder(tf.float32, [None, 784])\n",
    "y_ = tf.placeholder(tf.float32, [None, 10])\n",
    "\n",
    "# Define the graph\n",
    "# First fully connected layer\n",
    "W_fc1 = weight_variable([784, 500])\n",
    "b_fc1 = bias_variable([500])\n",
    "# h_fc1 = tf.nn.sigmoid(tf.matmul(x, W_fc1) + b_fc1)\n",
    "h_fc1 = tf.nn.relu(tf.matmul(x, W_fc1) + b_fc1)\n",
    "\n",
    "# Second fully connected layer\n",
    "W_fc2 = weight_variable([500, 10])\n",
    "b_fc2 = bias_variable([10])\n",
    "y_mlp = tf.matmul(h_fc1, W_fc2) + b_fc2\n",
    "\n",
    "# Loss \n",
    "cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_mlp))\n",
    "\n",
    "# Evaluation\n",
    "correct_prediction = tf.equal(tf.argmax(y_mlp, 1), tf.argmax(y_, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimizers\n",
    "\n",
    "Instead of the optimizer being given though, let's try out a few. Here we have optimizers implementing algorithms for [Stochastic Gradient Descent](https://www.tensorflow.org/api_docs/python/tf/train/GradientDescentOptimizer) (SGD), [Stochastic Gradient Descent with Momentum](https://www.tensorflow.org/api_docs/python/tf/train/MomentumOptimizer) (momentum), and [Adaptive Moments](https://www.tensorflow.org/api_docs/python/tf/train/AdamOptimizer) (ADAM). Try out different parameter settings (e.g. learning rate) for each of them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Optimizers: Try out a few different parameters for SGD and SGD momentum\n",
    "train_step_SGD = tf.train.GradientDescentOptimizer(learning_rate=0.001).minimize(cross_entropy)\n",
    "train_step_momentum = tf.train.MomentumOptimizer(learning_rate=0.001, momentum=0.9).minimize(cross_entropy)\n",
    "train_step_ADAM = tf.train.AdamOptimizer().minimize(cross_entropy)\n",
    "\n",
    "# Op for initializing all variables\n",
    "initialize_all = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "Because we'll be repeating training a few times, let's move our training regimen into function. Note that we pass which optimization algorithm we're running as an argument. In addition to printing out the validation accuracy and final test accuracy, we'll also return the lists of accuracies at each validation step and the training losses at each iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train_MLP(train_step_optimizer, iterations=4000):\n",
    "    with tf.Session() as sess:\n",
    "        # Initialize (or reset) all variables\n",
    "        sess.run(initialize_all)\n",
    "        \n",
    "        # Initialize arrays to track losses and validation accuracies\n",
    "        valid_accs = [] \n",
    "        losses = []\n",
    "        \n",
    "        for i in range(iterations):\n",
    "            # Validate every 250th batch\n",
    "            if i % 250 == 0:\n",
    "                validation_accuracy = 0\n",
    "                for v in range(10):\n",
    "                    batch = mnist.validation.next_batch(50)\n",
    "                    validation_accuracy += (1/10) * accuracy.eval(feed_dict={x: batch[0], y_: batch[1]})\n",
    "                print('step %d, validation accuracy %g' % (i, validation_accuracy))\n",
    "                valid_accs.append(validation_accuracy)\n",
    "                \n",
    "            # Train    \n",
    "            batch = mnist.train.next_batch(50)\n",
    "            loss, _ = sess.run([cross_entropy, train_step_optimizer], feed_dict={x: batch[0], y_: batch[1]})\n",
    "            losses.append(loss)\n",
    "            \n",
    "        print('test accuracy %g' % accuracy.eval(feed_dict={x: mnist.test.images, y_: mnist.test.labels}))\n",
    "        \n",
    "    return valid_accs, losses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's train the MLP using all three optimizers and compare the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SGD:\n",
      "step 0, validation accuracy 0.072\n",
      "step 250, validation accuracy 0.382\n",
      "step 500, validation accuracy 0.576\n",
      "step 750, validation accuracy 0.636\n",
      "step 1000, validation accuracy 0.746\n",
      "step 1250, validation accuracy 0.778\n",
      "step 1500, validation accuracy 0.762\n",
      "step 1750, validation accuracy 0.788\n",
      "step 2000, validation accuracy 0.826\n",
      "step 2250, validation accuracy 0.824\n",
      "step 2500, validation accuracy 0.812\n",
      "step 2750, validation accuracy 0.856\n",
      "step 3000, validation accuracy 0.854\n",
      "step 3250, validation accuracy 0.862\n",
      "step 3500, validation accuracy 0.842\n",
      "step 3750, validation accuracy 0.856\n",
      "test accuracy 0.8675\n",
      "Momentum:\n",
      "step 0, validation accuracy 0.06\n",
      "step 250, validation accuracy 0.828\n",
      "step 500, validation accuracy 0.852\n",
      "step 750, validation accuracy 0.884\n",
      "step 1000, validation accuracy 0.896\n",
      "step 1250, validation accuracy 0.908\n",
      "step 1500, validation accuracy 0.916\n",
      "step 1750, validation accuracy 0.912\n",
      "step 2000, validation accuracy 0.914\n",
      "step 2250, validation accuracy 0.922\n",
      "step 2500, validation accuracy 0.938\n",
      "step 2750, validation accuracy 0.914\n",
      "step 3000, validation accuracy 0.938\n",
      "step 3250, validation accuracy 0.936\n",
      "step 3500, validation accuracy 0.926\n",
      "step 3750, validation accuracy 0.948\n",
      "test accuracy 0.9302\n",
      "ADAM:\n",
      "step 0, validation accuracy 0.07\n",
      "step 250, validation accuracy 0.94\n",
      "step 500, validation accuracy 0.94\n",
      "step 750, validation accuracy 0.966\n",
      "step 1000, validation accuracy 0.964\n",
      "step 1250, validation accuracy 0.968\n",
      "step 1500, validation accuracy 0.984\n",
      "step 1750, validation accuracy 0.96\n",
      "step 2000, validation accuracy 0.972\n",
      "step 2250, validation accuracy 0.974\n",
      "step 2500, validation accuracy 0.974\n",
      "step 2750, validation accuracy 0.982\n",
      "step 3000, validation accuracy 0.982\n",
      "step 3250, validation accuracy 0.98\n",
      "step 3500, validation accuracy 0.974\n",
      "step 3750, validation accuracy 0.968\n",
      "test accuracy 0.9785\n"
     ]
    }
   ],
   "source": [
    "print(\"SGD:\")\n",
    "valid_accs_SGD, losses_SGD = train_MLP(train_step_SGD)\n",
    "print(\"Momentum:\")\n",
    "valid_accs_momentum, losses_momentum = train_MLP(train_step_momentum)\n",
    "print(\"ADAM:\")\n",
    "valid_accs_ADAM, losses_ADAM = train_MLP(train_step_ADAM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualization\n",
    "\n",
    "Plotting things:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x26600422a90>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbMAAAEYCAYAAADWNhiqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXeYVOX1xz9ne1/K0mHpvStNo1Gs2Av+hGg0ahJjYomJ\nGktULNFYorEnYjcithhQAREVFKRIld5Zdpe67LJ1tkw5vz/uzLJ9Z3ensu/nee4zM3fufe9ZmJnv\nPec97zmiqhgMBoPBEM5EBNsAg8FgMBhaihEzg8FgMIQ9RswMBoPBEPYYMTMYDAZD2GPEzGAwGAxh\njxEzg8FgMIQ9RswMBoPBEPYYMTMYDAZD2GPEzGAwGAxhT1SwDWgqaWlp2qtXr2CbYTDUy+rVq4+o\nagdfjSciccD3QCzWd/YTVZ1W45jTgdnAHveuT1X1kYbGNd8lQ6jTlO9S2IlZr169WLVqVbDNMBjq\nRUT2+njIcuAMVS0WkWhgiYjMU9XlNY5brKoXejuo+S4ZQp2mfJfCTswMhtaGWgVUi90vo92bKapq\nMFTBzJkZDGGAiESKyDrgMLBAVVfUcdjJIrJeROaJyNB6xrlRRFaJyKqcnBy/2mwwBBK/iZmIvCki\nh0VkYz3vi4i8ICI73V/AE/xli8EQ7qiqU1VHAd2BcSIyrMYha4B0VR0BvAjMqmec6ao6RlXHdOjg\ns2k9gyHo+DPM+DbwEvBuPe+fB/R3b+OBf7kfDQZDPahqvogsBCYBG6vsL6zyfK6IvCIiaap6JBh2\nhjt2u53s7GzKysqCbUqrIC4uju7duxMdHd3sMfwmZqr6vYj0auCQS4B33fMBy0WkjYh0UdUD/rLJ\nYAhHRKQDYHcLWTxwNvBkjWM6A4dUVUVkHFbUJTfw1h4fZGdnk5ycTK9evRCRYJtzXKOq5Obmkp2d\nTe/evZs9TjATQLoBWVVeZ7v3GTEzGKrTBXhHRCKxROojVf1CRG4CUNV/A1cAvxcRB1AKTFXTebfZ\nlJWVGSELECJC+/btaekcblhkM4rIjcCNAOnp6UG2xmAILKq6Hhhdx/5/V3n+ElZY3+AjjJAFDl/8\nWwczm3Ef0KPK6+7ufbUwk9YGg8FgaIhgitlnwLXurMYJQIGZL2s65c5yckvN1IjBcDzy2GOPMXTo\nUEaMGMGoUaNYsWIFDoeD++67j/79+zNq1ChGjRrFY489VnlOZGQko0aNYujQoYwcOZJnnnkGl8sV\nxL8iMPgtzCgiM4HTgTQRyQamYS329IRH5gLnAzsBG3C9v2w53lBVNh7ZyKyds5i3Zx7F9mIu7385\nt4y+hbT4tGCb1yTsLju783ez7eg2tuRuIbMoE5d698WLj4onPiqehKgEEqIT6n1Mi0+je1J3oiOb\nnyllMASaZcuW8cUXX7BmzRpiY2M5cuQIFRUV3H///Rw8eJANGzYQFxdHUVERzzzzTOV58fHxrFu3\nDoDDhw9z1VVXUVhYyMMPPxysPyUg+DOb8ReNvK/Azf66/vHIkdIjfL7rc2bvnM2ugl3ERsZyVs+z\nSIlJ4eNtHzNvzzx+O+K3XDPkGmIjY4Ntbi2KK4rZdnQbW/O2si3PetyZvxO7yw5AXGQcPVN6Eh3R\nuOgoykHnQWx2GzaHDZvdRoWrot7jIySCLoldSE9OJz0lvdpj9+TuxETG+OzvNBh8wYEDB0hLSyM2\n1voup6WlYbPZeO2118jIyCAuLg6A5ORkHnrooTrH6NixI9OnT2fs2LE89NBDx/U8YFgkgLRm7E47\ni7IXMWvnLH7Y9wNOdTKyw0imnTSNc3udS3JMMgBXD76aZ1Y9w/NrnufjbR/zpxP/xLm9zm3yh7fc\nWc76nPXkluVaQlFFLKo9OmyU2ktxuBxejXu0/ChZRceSV9vGtmVQu0H8cvAvGdRuEIPaDSI9JZ2o\niOZ/JO1Ou2WXo7TSzhJ7CYdsh8gszCSzKJPMwkzm7p5Lkb2o8jxB6JLYhR4pPUhPTqdnSk96JFvP\ne6T0CMkbA0PgePjzTWzeX9j4gU1gSNcUpl1UZ5GWSs455xweeeQRBgwYwFlnncWUKVNo27Yt6enp\nJCcne32tPn364HQ6OXz4MJ06dWqp6SGLEbMQwuFykFeWx5HSIxwpPcIP+35g7p655Jfn0zG+I9cN\nvY5L+l1C79TaazF6pvTkhTNeYMWBFTy98mnu+v4uZmyZwV/G/oXhHYY3eN2swiwW71vMD/t/YOXB\nlZQ6SmsdEx0RfSx05w7fecJ83tApsROX9ruUQe0GMbDtQDomdPT5XWJ0ZDSpkamkxqY2eJyqUlBe\nYImbW+D2Fu4luyibBXsXkF+eX3msIHRK7GQJW3IPeqb0JD05nZO7nez1324wNIekpCRWr17N4sWL\nWbhwIVOmTOG+++6rdsxbb73F888/T25uLkuXLqVHjx71jHb8I+G2FGXMmDEaSpW+VZUFexfwxsY3\ncKmr0fmb+Kh4SuwllYJVdTtadhStUj82OiKaiT0mcmm/Szmp60leey1Ol5PZu2bzwpoXyC3L5YI+\nF3D7CbfTObEzAKWOUlYeXMkP+35gyb4lZBZlApCenM4p3U7hZ91+RrekbiRGJ1bOSbWm+aaC8gKy\nirKqeXOZRZlkFWWRV5YHwOIpi2kT16bO80VktaqOCaTNzaG+75LD6eIfX23ndz/vQ9vE1hl+3bJl\nC4MHDw62GdX45JNPePXVV1mzZg0ZGRnVvLNhw4bxxRdf0KtXL5KSkiguLq58b/fu3YwdO5YjR46E\ndJixrn/zpnyXjGfWArIKs3jsx8f4Yd8P9GvTj+5J3bE5bOSX5bPfsf9YWM5uw6HVw3HREdGkxaeR\nFp9G16SujOwwsvJ1+/j2pMWn0SulV6NeRl1ERkRyef/LObfXubyx4Q3e2fQO3+z9hkv6XUJ2UTYr\nD66kwlVBXGQcYzuP5erBV3NKt1NITzFr+ABSYy3vblhazfKHUFhRSHZRdr1CdjywYPMh/v3dLg4X\nlfHslaOCbU6rZdu2bURERNC/f38A1q1bx8CBAxk9ejS33HILr776KnFxcTidTioq6p4vzsnJ4aab\nbuKWW24JaSHzBUbMmkGFs4I3Nr7B6+tfJzoymnvG3cOUgVMa9Jw88zk2u42E6ARSYlL8/uFKjE7k\nthNu44oBV/Dc6uf4cNuH9E7tzZRBUzil6ymc2PlEMx/URFJiUhjSfkiwzfArZQ4nAE5XeEVtjjeK\ni4u59dZbyc/PJyoqin79+jF9+nRSU1N54IEHGDZsGMnJycTHx/OrX/2Krl27AlBaWsqoUaOw2+1E\nRUVxzTXX8Oc//znIf43/MWLWRJYfWM5jyx8jozCDSb0mcdfYu+iY0LHR87ydz/EHXZO68tRpT/Ho\nKY8a8TI0irOijDujPiSLW4JtSqvmxBNPZOnSpXW+98QTT/DEE0/U+Z7T6fSnWSGLETMvOVJ6hKdW\nPsW8PfNIT07n1bNe5eRuJwfbrCZhhMwHHNwI6z+Esx+B4zRs02f3+1wRNZsFee2BccE2x2DwCiNm\njeB0Oflw24e8uPZFyp3l/H7k7/n18F8bYWhtHNkBCx+HTZ9CbCqceB207xtsq/xChKscgCi1B9kS\ng8F7WqWY/ZTzE3N3z/Xq2LWH17IlbwsndTmJv074Kz1TevrZuuMQTymdCB9WTysvBnVBTJJvx61J\nfiYsehJ+eh+i4uHUO+DkWyG+rf+uGWQUy+PMPloSZEsMBu9plWL2n83/YcHeBSRFJzV6bJvYNjz9\n86ebtQC51ZO3G9a9D+tmQnkh/PxOGH8TRLXAq7XlwXdPwcrXwLNgOzoBYhLdW1KV54kQkwzt+0D6\nydDtBIj2cm1Y0UFY/AysegskwrL7lD9DUmsodG19zovKvFsQbzCEAq1SzIrtxQxpN4SZF84MtinH\nH+XFsHmWJWJ7fwAE+p5hzS8teBBWvWnNNw2+uGlzTk47rHwDFv3dEsZRV0HaQKgogYpi92PJsddl\nhVB4AMqLLK8KICLaErT0CZB+EvQYDwntql/HlgdL/gk/vgYuO4z+Jfz8L5DazWf/ROGCYLIZDeFD\nqxSzUnspidGJwTbj+EHVEq5178OmWWAvgXZ94cwHYcTUY0Kw8xuY/1f46Fro+TM49zHoWqtNV+2x\nt8+Hr+6H3B3Q53Q45zHoXHsNWL3Y8iBrBWQug73LYNkr8MPz1nsdBkPPkyxxy9sNS1+yxHDEFDj9\nbmjXpzn/ImGNE7B8MiNmhvChVYpZib2ELkldgm1GeONyWkkRWz6DdTPgaIYV0hs+GUb9EnqMq+15\n9TsTep8Ga96BhY/B9ImWh3XGA5BSx//HoU0w/z7YvQja94NffAgDzm16FmFCOxh4nrUB2Eth3xrI\nXAqZy2HDJ5bHCJbHOPGv0HFQU/9Fjhu2OPdxfe903tm3INimtGpEhKuvvpr33nsPAIfDQZcuXRg/\nfjxffPFFwO1Zt24d+/fv5/zzzw/4tb2hVYqZzWEznllTqLDB4c1wcD0c3AAH1ltC46nh2PvncPp9\nMPhCa56qISKjYOyvYdhkWPwPWP5vy5s75XY46RaISYDiHFj4N1jzLsSmwKQnYMyvIcpHpZWi46HX\nz6wNLGE+tMmay+sw0DfXCGOiHDaIhAgxnlkwSUxMZOPGjZSWlhIfH8+CBQvo1i144e5169axatUq\nI2ahRIm9hISohGCb0XyKc44Jy8H1UHQIEtMgqRMkdbQekzsfe56QZolITVwuKyRYUWLNdXnmnsoL\nIWebe/wNVnjP02MsLhU6j4Ax10Pn4Va4sG0zMjzj28A5f4MxN1hzaQsfg9Vvw5BLLRFzlMK4G+G0\nu2vPa/maiEjoMsK/1wgjjISFDueffz5z5szhiiuuYObMmfziF79g8eLFAOTl5XHDDTewe/duEhIS\nmD59OiNGjOChhx5iz5497N69m8zMTP75z3+yfPly5s2bR7du3fj888+Jjo5m9erV/PnPf6a4uJi0\ntDTefvttunTpwumnn8748eNZuHAh+fn5vPHGG4wfP54HH3yQ0tJSlixZwr333suWLVtISkrizjvv\nBI7VhwSYNGkSEyZMYOnSpYwdO5brr7+eadOmcfjwYWbMmMG4cb5fv9gqxazUESZzZi4XHN1zTLQ8\n4lJUpSF3ag9I7W55FrsWQnlBHQOJJXbx7cBZfixRwm5r+PqpPSzBGnqZ9dh5OLRJ9+1i4XZ9YMp7\nkLHECikufxkGTLKELq2/765j8BqTs1uDefdY3ztf0nk4nFd3BY+qTJ06lUceeYQLL7yQ9evXc8MN\nN1SK2bRp0xg9ejSzZs3i22+/5dprr61syrlr1y4WLlzI5s2bOemkk/jvf//LU089xWWXXcacOXO4\n4IILuPXWW5k9ezYdOnTgww8/5K9//StvvmmF2x0OBz/++CNz587l4Ycf5uuvv+aRRx5h1apVvPTS\nSwD19lAD2LlzJx9//DFvvvkmY8eO5f3332fJkiV89tlnPP7448yaNauF/4C1aXVi5nQ5KXWUhqZn\npmp5RDu/trbsVVDh7rslkVYIrPdp1hehywjoNKy212IvheLD7u2QezsMxQfBlmutlYpJhNikGmns\nNZ636+N/j6gqvU6B3y6CksOWV2kIGnHRUeAyHlooMGLECDIyMpg5c2at8N6SJUv473//C8AZZ5xB\nbm4uhYVW37XzzjuP6Ohohg8fjtPpZNKkSQAMHz6cjIwMtm3bxsaNGzn77LMBqwRWly7H5q0vv/xy\nwCqplZGR0WS7e/fuzfDhVuupoUOHcuaZZyIildf3B61OzDy9uhKiQ0TMygphz/ewc4GV7VfgbmDZ\nYRCMnGKF9LqMsLLuouMaHy863gr7NSf0F2wiIoyQhQDJsZFQu6Vd68ULD8qfXHzxxdx5550sWrSI\n3Nxcr87xdKeOiIggOjq6co1sREQEDocDVWXo0KEsW7aswfMjIyNxOOpebxgVFYXLUxABKCsrq3W+\n55pV7alvvJbS6sTM5rBCa0ETM1U4tNHyvHZ8DVnLrcW/McnQ5zSrwkS/s6BN622yZwgy7jijIuzL\nL6VbG9OENJjccMMNtGnThuHDh7No0aLK/aeeeiozZszggQceYNGiRaSlpZGSkuLVmAMHDiQnJ4dl\ny5Zx0kknYbfb2b59O0OH1t/9Ojk5maKiYx3ae/XqVTlHtmbNGvbs2dO8P9BHtDoxK7FbJXoSowI4\nZ1Z61Eov3+EOHxYftPZ3Gm5l8PU/G7qP8122nsHQAqRKw947PlrHBzeeFERrDN27d+e2226rtf+h\nhx7ihhtuYMSIESQkJPDOO+94PWZMTAyffPIJt912GwUFBTgcDm6//fYGxWzixIk88cQTjBo1invv\nvZfJkyfz7rvvMnToUMaPH8+AAQOa9ff5ilbXaXpT7iamfjGVF894kdN7nO47w6ricsHBn46JV/ZK\nUKeVCdhnouV59Tur7rVVhrDH152mRSQO+B6IxboB/URVp9U4RoDngfMBG3Cdqq5paNz6vktff3gt\nfypby9v7D/Fk29l8dFPrE7NQ7DR9vGM6TTcRmzuDz+fZjCW5sOtbS7x2fQMlOdb+LqPg1D9b4tVt\nTN0p8gZDw5QDZ6hqsYhEA0tEZJ6qLq9yzHlAf/c2HviX+7EZHMtn9DTqNBhCnVb3y+oRM59lM6rC\n8ldgwTSrll98O6vSRb+zoO+ZraQwrcGfqBU+KXa/jHZvNUMqlwDvuo9dLiJtRKSLqh6gqbiTBVRg\nfXZdSz0MhtCj1YmZZ84s3tvq6Q1Rmg+zb4atX8DAC6zkja6jrEW4BoMPEZFIYDXQD3hZVVfUOKQb\nkFXldbZ7XzUxE5EbgRsB0tPT67mWb2w2GAKJHxtBhSaebMYWJ4DsWwOv/hy2fwnn/h2mzoDuJxoh\nM/gFVXWq6iigOzBORJpQabnaONNVdYyqjunQoe6oQWJstHVsc401GIJAqxMzj2fW7NR8Vas9yJvn\nWiWerv8STvqDuZ01BARVzQcWApNqvLUPqLqeo7t7X5OJaH0/C4bjgFb3qa1cZ9acObOyQvj4Oph7\np9Wj63ffQ4+xvjXQYKiBiHQQkTbu5/HA2cDWGod9BlwrFhOAgmbNlxkMYUrrEzO7jbjIOCKbGg48\nsB6mnwZbPoezHoapMwNb7sngFapK9lEbX206yHNfb+em/6zmH/O3caS4PNimtYQuwEIRWQ+sBBao\n6hcicpOI3OQ+Zi6wG9gJvAb8obkXMzGG0GHWrFmICFu3WvcuGRkZxMfHM3r0aAYPHsy4ceN4++23\na5136aWXMmHChGr7HnroIUSEnTt3Vu577rnnEBFastwpVGh1CSA2u61pIUZVq5r7vLshoT1cN8dq\n5mjwOQ6ni1V7jyJAYmwUCTGRlY8JMVFERlT/mbU7XezKKWbTvkI2Hyhk837rsaDUDliR3x5tE5i/\n+SCvLd7NlLE9+O2pfejRrvmZrHklFSzfncv5wwO3RlBV1wO1upiq6r+rPFfgZp9c0ITMQ4aZM2dy\nyimnMHPmTB5++GEA+vbty9q1awHYvXs3l19+OarK9ddfD0B+fj6rV68mKSmJ3bt306fPsQazw4cP\n54MPPuD+++8H4OOPP25woXQ40erErMTRhPYvZQUw5w7Y8LGVZn/5dKv6vMHnOJwu/vjBOuZsqD8y\nFhcdQWJMFAmxkcRERpCVV0qF06oNFxsVwaAuKZw/vAtDuqYwtGsKgzonkxATxa6cYqZ/t5uZP2Yy\nY0UmF4/syu9P78uATsle2XakuJz5mw4yb8NBlu3OxelSFv9lYotEMaTxpOYH2YzWTnFxMUuWLGHh\nwoVcdNFFlWJWlT59+vDss89yxx13VIrZp59+ykUXXUSnTp344IMPuO+++yqPv/TSS5k9ezb3338/\nu3btIjU1lejo6ID9Tf6k1YmZze5FY06nA9a+C98+BqV5cMb9cModViFcg89xuZS/fLKeORsO8Oez\nBzCmZ1tKKpzYKhwUlzuwlTspqXBgq3BSUm49llY4OXNwJ4Z2TWFIlxR6pyUSFVn3/0/fDkk8ecUI\nbj+7P68v3sP7KzL539p9nDW4E3+Y2JcT0tvWOudwURnzNx1i3oYDLN+di0uhV/sEfvfzPpw/vAvd\n2x7P9QqNZ1aVJ398kq15NacoW8agdoO4e9zdDR4ze/ZsJk2axIABA2jfvj2rV6+mffv2tY474YQT\nKsOQYHlzDz74IJ06dWLy5MnVxCwlJYUePXqwceNGZs+ezZQpU3jrrbd894cFkVYpZg2GGXd9C/P/\nanVWTj8ZJj0OXWtFeAw+QlX566wNfLp2H3eeM4BbzvBfD7MuqfE8cOEQbpnYj7eXZvDOsgwuf+UQ\nE/q04/en92NQ52TmbzrInPUH+DEjD1Xo0yGRmyf247xhXRjcJbmy+nhrwHhmwWXmzJn88Y9/BKy+\nZjNnzuSWW26pdVzVkoSHDh1ix44dnHLKKYgI0dHRbNy4kWHDjq3kmDp1Kh988AHz58/nm2++MWIW\nrtgcNlJi66gsnbMdvrofdsyHtr3gyndh8MVm/sCPqCoPf76ZmT9mccvEfn4Vsqq0TYzhT2cP4Maf\n92Hmj5m8vngPv3rzx8r3+3dM4tYz+nPB8C4M6JTUqgQMjF9Wk8Y8KH+Ql5fHt99+y4YNGxARnE4n\nIsLNN9eeFl27dm1lTcOPPvqIo0eP0rt3bwAKCwuZOXMmjz32WOXxF154IXfddRdjxozxusp+ONDq\nxKzEXkLnxCo9s2x5sOgJWPUGRCfA2Y/A+JsgKrb+QQwtRlV58sttvL00g1+f0ps7zgl8xe3E2Ch+\nc2ofrjmpJ5+t28/honLOGdKJ/l7OpR3vGM8seHzyySdcc801vPrqq5X7TjvtNLKysqodl5GRwZ13\n3smtt94KWN7cl19+yUknWUlqe/bs4ayzzqomZgkJCTz55JNBr3Lva1qdmNkc7jkzRwWsfB2+exLK\nC+HE6+D0+0wtxQDxwjc7+fd3u7h6fDr3XzA4qN5PbFQk/zfG9I/zYDyz4DNz5kzuvru6Rzh58mT+\n/ve/s2vXLkaPHk1ZWRnJycncdtttXHfddWRkZLB3795qKfm9e/cmNTWVFSuqVz+bOnVqQP6OQNL6\nxMxuI6G8BF6ZAHm7rMXP5zwGnYYE27SQweHOEKwvoaKlvPrdLv759XauOLE7j14yrNWF8cIF45kF\nj4ULF9bad9ttt9XZ18xDr1692LevdtGXNWusTkDjx9fdRKFqw89wplWJmapa2YwHNoDtCFz1sdUY\n0/yYVrJsVy5/mLGaozY7sVERx9Z7uVPiE2Os10mx1uveaUmcNqADfTskeiVK7yzN4O/ztnLRyK48\nOXkEERHm3z7UkBq+2ZrMo3VmfBoMoUSrEjO7y45DHSQ47dC+Pww4J9gmhRRz1h/gTx+uI719Atf/\nrLeVDu9Ji6+SHn+kuJySCgcl5U7ySip4FOjeNp7TBnTg9IEdOblvexJja3+0PlyZybTPNnHOkE48\ne+XIWougDaGFukXtiXlb+eh3plCAIbRpVWJW2f7FUQGxHYNsTWjx9g97ePiLzZyY3pbXfzWGNgkx\nXp2XlWfj+x05LNqWw6y1+5ixIpPoSGFsr3acNqADpw3swMBOycxet597Pt3AaQM68OJVo4n2UwjT\n0HJq3mL8uCcvKHYEG1U1IfAAUXV5QXPxq5iJyCSsVu6RwOuq+kSN91OB94B0ty3/UFW/LXqobP9i\nL4dEk7EG1ofoqfnb+NeiXZwzpBMv/GI0cdHe163s0S6Bq8f35OrxPalwuFi1N4/vtuXw3fYc/j5v\nK3+ft5XOKXHkFJczoXd7Xr3mRGKjTJucsKAV/47HxcWRm5tL+/btjaD5GVUlNzeXuLi4Fo3jNzFz\nNxN8GavCdzawUkQ+U9XNVQ67GdisqheJSAdgm4jMUNUKf9hU2f7FXgpxx8/6iuZid7q4+7/r+XTN\nPq4en84jlwxrUegvJiqCk/umcXLfNO49fzAHCkr5frvltcVHR/LopcOaJJSG4CBdRkDu98E2I6h0\n796d7OxscnJygm1KqyAuLo7u3bu3aAx/embjgJ2quhtARD7Aau1eVcwUSBbr1icJyAMc/jLIZnd7\nZhWlUNfC6VZESbmD389Yw/fbc7jj7AHcckY/n9+BdkmNZ8rYdKaMrbujsSFE6TEeNr7UqrMZo6Oj\nKxceG8IDf4pZXW3ca+aGvoTVh2k/kAxMUVWXvwzyiFlCeQnEtt4w45Hicm54eyWb9hfy5OThRmwM\nBkPYE+xZ+HOBdUBXYBTwkojUcplE5EYRWSUiq1ri9lc25nS5Wq2Y7c0tYfK/lrL9UBHTrznRCJnB\nYDgu8KeYedPG/XrgU7XYCewBBtUcSFWnq+oYVR3ToUPzK3RUzplp6xSzDdkFTP7XUgpL7bz/2wmc\nObhTsE0yhDCtOcxoCD/8KWYrgf4i0ltEYoCpWCHFqmQCZwKISCdgIFa3XL9Qmc3o0lYlZi6XMmPF\nXqZOX0ZsVCSf/P5kswjWUC91zZ2GeaduQyvAb3NmquoQkVuA+Vip+W+q6iZPm3d3l9xHgbdFZANW\nIvDdqnrEXzYd88y01SSAbNxXwP2zNrIuK58JfdrxwtTRdExpWQqsoXVQ1TPbn19KWpIpvm0IXfy6\nzkxV5wJza+yr2up9PxCwMhw2u41IIojV0PfM1mXlc6iwjNMHdmjWuqyiMjvPLtjOO0szaJcYwz+n\njOTSUd3MmhlDo9QsZwXWMo7Z6/YxaVhns07QEJK0qgogNoeNhMgY66saop6Z06U8/80OXvx2B6qQ\nGh/NJaO68n8n9mBYt5RGxUhVmbPhAI9+sZnDReVcPT6du84ZRGrC8dEa3RA4qnpmf5ixhkOF5fzm\nlN7cf6Epym0IPVqXmNltJES4f9RD0DM7XFTG7R+sY+muXCaf0J0LR3Th07X7+GBlFu8u28ugzslc\ncWJ3Lh3drc6QT8aREh78bBPfb89haNcUXr1mDKN6tAnCX2IIZ+ryzA4VWnNmBwvLAm2OweAVrUrM\nSuwlJIg7RBJiYrZ01xH++ME6CkvtPHXFCK5099eaOKgjBTY7n6/fz8ers/nbnC08MW8rEwd15IoT\nu3PGoI5CSKY5AAAgAElEQVS4VPn3ot28vGgnMZERTLtoCNdM6Om3Fi4Gg8EQarQqMbM5bCR4EjhD\nRMxcLuWVRTt5dsF2eqUl8p9fj2NQ5+oh0NSEaH45oSe/nNCTHYeK+GR1Np+u3ceCzYdonxhDQmwk\nWXmlXDiiCw9cOIROJsHD0AIaCmWbdH1DqNK6xMxuI5EIiE6EiOBPYucWl/Onj37i++05XDyyK49f\nPpykOlqnVKV/p2TuPX8wd507kO935PDxqmwOFpbx+GXDObW/6ZJt8B1GuAzhROsSM4eNziGSybgy\nI49b319Lnq2Cxy4bxlXj0puUaRgVGcEZgzpxxiCz8Pl4R0R6AO8CnbA0ZrqqPl/jmNOB2ViFB8Aq\nRvBIs67XULl8o3CGEKV1iZndRkKQF0y7XMr0xbt5ev42ureN59Pfn8ywbqlBs8cQFjiAO1R1jYgk\nA6tFZEGNDhQAi1X1Ql9dVOsQNTVqZghRWpWYldhLSHA5gyZmqsqtH6xlzvoDnD+8M09MHkFKnEmZ\nNzSMqh4ADrifF4nIFqxC3jXFzDeYpYiGMKRVpbvZHDYSnY6gidnGfYXMWX+AP5zel5evOsEImaHJ\niEgvYDSwoo63TxaR9SIyT0SG1nO+10W71YiaIYxoNWLmUheljlISHBVBa8z54apMYqMi+N1pfU0l\nDkOTEZEk4L/A7apaWOPtNUC6qo4AXgRm1TWGN0W7G5wzMxhClFYjZqWOUgASHRVBqf5RZncye91+\nzhvWmdR445EZmoaIRGMJ2QxV/bTm+6paqKrF7udzgWgRSWvJNeuaHbNVOFsypMHgN1qNmHmKDMfb\ny4ISZvxy40GKyhyVi6ENBm9xd2J/A9iiqs/Wc0xn93GIyDis73Zus67XgGd21GZvzpAGg99pNQkg\nni7TiRW2oIjZR6uy6NEungl92gf82oaw52fANcAGEVnn3ncfkA6VxbuvAH4vIg6gFJiqqj5PPTQB\nSEOo0qiYiUikqoZ9bKHE4W7/EoQu05m5NpbuyuWOswcQEWF+DgxNQ1WX0IiOqOpLwEs+va4vBzMY\n/Iw3YcYdIvK0iIR1qexKzywIYvbx6ixEYPKJ3QN6XYOhOZjkJEM44o2YjQS2A6+LyHJ3am9o9k9p\nAI+YWYumA2e+06V8sjqbn/fvQNc28QG7rsHgC6JxVHttdM4QqjQqZqpapKqvqerJwN3ANOCAiLwj\nIv38bqGPsDncYqaB9cwW78jhQEEZU8aaxA9DeFA1AaST5FV7b21mPr3umcOTX24NtFkGQ4M0KmYi\nEikiF4vI/4DngGeAPsDn1OgiHcp4shkD7Zl9vCqbtgnRnDm4Y8CuaTD4AmvOrG5X7F+LdgXSFIOh\nUbzJZtwBLASeVtWlVfZ/IiI/949ZvqcyzBhAzyyvpIKvNh/kmgm9TKt5g8Fg8CPeiNkIz2LMmqjq\nbT62x28cy2YMXKHh/63dh92pXDnWJH4Ywg8FfJ/cbzD4B28SQF4WkTaeFyLSVkTe9KNNfqHUXkqs\nRFrqHQAxU1U+XpXFyO6ptZptGgyhTNVsRpPwYQgXvBGzEaqa73mhqkexCp2GFTaHjURxl5EKgJit\nzy5g68EirjSJH4YwRYElfzk92GYYDF7hjZhFiEhbzwsRaUcYVg4psZcQLxEQFQ+R/q+N+NGqLOKi\nI7hoZFe/X8tg8CXVylmZOKMhTPBGlJ4BlonIx1ipTVcAj/nVKj9gs9tIJCIgXllphZPP1u3n/GFd\nTJsXQ9jikbEzBnXk262Hg2qLwdAY3qwzexeYDBwCDgKXq+p//G2YrylxlJCgBETM5m08QFG5w4QY\nDWFJNc8sJolfjEsPnjEGg5d4FS5U1U0ikgPEAYhIuqpm+tUyH1NqLyVZA5PJ+NGqLHq2T2B873Z+\nv5bB4C9UBFAm9DGfY0Po482i6YtFZAewB/gOyADm+dkun1NiLyHB5fR7Y869uSUs353HlWN6mBp3\nhrCk2ud2yT/N59gQFniTAPIoMAHYrqq9gTOB5X61yg/YHDYSnE6/V//4eFU2EQKTTzBrywzHActf\nIcp0ejCEAd6ImV1Vc7GyGiNUdSEwxs92+ZwSewkJDrtfw4yeosKnDehA59Q4v13HEJ6ISFiWgYmL\nrtvshdtMUoghdPBGzPJFJAn4HpghIs8DJf41y/fYHDYSHeV+FbPvt+dwsNAUFTbUS1i0U2qo03RV\nHvl8s58tMRi8xxsxuwSwAX8CvgR2ARf50yhfU+GswOFykGD3r5h9tCqL9okxnDGok9+uYQhrwqqd\nUmMrzPYcKcHlMuvQDKFBg2LmDot8oaouVXWo6juq+oI77Bg2HOtl5vCbmOUWl/P1lkNcNrobMVHe\n3CMYWhth006pCVNkOw7XWbbVYAg4Df7qqqoTcIlIaoDs8QuBKDLsKSpsQoyG+gi3dkpVfa4ld0+s\n8xiT6GgIFbxZZ1YMbBCRBVSZKwunivnHPDOXX7IZVZWPVmUxOr0N/TsFrvGnIewIi3ZKdc2ZdW+b\nUM+xBkNo4I2YferewhZPY85E9U9jzq82H2L7oWL+fvlwn49tOK4Iq3ZK3syGHS4qZ8vBIi4c3oUI\nk8JvCCKNipmqvhMIQ/yJzeHxzHwfZjxcVMa9n25gaNcUs7bM0BgdRWQmcBLgApYBf1LV3cE1qzre\nZjMCXP36CgDK7E6uHGNC7Ibg0aiYicge6rhJU9U+frHID3jCjIk+7jKtqtz18XpKyh08P3WUSfww\nNMb7wMvAZe7XU4GZwPigWdQATclTPFJc7jc7DAZv8CbMWHWBdBzwf0BYFWvzl2f27rK9fLc9h0cv\nGUq/jmauzNAoCTWKdL8nIncFzZp6aG75qhW7c+neLoFubeJ9bJHB0DjeVM3PrbLtU9XngAsCYJvP\n8MyZ+TIBZMehIh6fu4WJAzvwywk9fTKm4bhnnojcIyK9RKSniPwFmCsi7dx9AutERHqIyEIR2Swi\nm0Tkj3UcIyLygojsFJH1InJCS41timemClOmL2fi04taelmDoVl4E2as+qWIwPLUwqo5Z2U2oyrE\nJrV4vHKHk9s+WEdSbBRPXTHSFGI1eMuV7sff1dg/FUs76gvdO4A7VHWNiCQDq0VkgapWLcFxHtDf\nvY0H/kUzw5dNmTOrSYXT1exzDYaW4G1zTg8OrOr5V9ZzbDVEZBLwPBAJvK6qT9RxzOlYa26igSOq\nepo3YzcFm8NGBBAXEQNRsS0e75mvtrPlQCFv/GoMHZJbPp6hdeAu1N2c8w4AB9zPi0RkC9ANqCpm\nlwDvqqoCy0WkjYh0cZ/bPMw9miGM8Cabse7Vko3grh7yMnA2kA2sFJHPqt5Nikgb4BVgkqpmikjH\n5lyrMWx2GwkSifhgvmzpziO8tng3V49P58zBpmyVwXtEJBr4PeBZU7YIeFVV7U0YoxcwGlhR461u\nQFaV19nufdXETERuBG4ESE+vu+lmfZ7ZX88fzGNzt3hrqsEQULzpZ/a4W3Q8r9uKyN+8GHscsFNV\nd6tqBfAB1t1jVa4CPvU0+lRVv5ThtjlsJBDR4l5m+bYK/vzRT/Run8hfLxjsI+sMrYh/ASdi3cC9\n4n7+L29Pdhf8/i9wu6oWNscAVZ2uqmNUdUyHDh2adO6kYZ3rfS+3uKI55hgMPsObXPLzVDXf80JV\njwLne3FefXeKVRkAtBWRRSKyWkSurWsgd0HWVSKyKicnx4tLV6fEXkKCSosyGVWVv/5vI0eKy3l+\n6mgSYsJq2tAQGoxV1V+p6rfu7XpgrDcnur26/wIzVLWuIgb7gKoLvbq79zUbbUKc8c0f9rTkUgZD\ni/FGzCJFpHJiSETiAV9NFEVh3Z1eAJwLPCAiA2oe1JK7SbDCjC2t/vHpmn3M2XCAP509gOHdw7pU\npSF4OEWkr+eFiPQBnI2dJFaG0RvAFlV9tp7DPgOudWc1TgAKmj1fZubKDGGIN+7FDOAbEXnL/fp6\nwJuqIN7cKWYDuapaApSIyPcca5PhM0rsJVZafnzzPLOsPBvTPtvEuF7tuOm0vo2fYDDUzV3AQhHZ\njSUZPbG+T43xM+AarBqp69z77gPSAVT131iFis8HdmK1bPJm3AYxzV0M4YQ3CSBPishPwFnuXY+q\n6nwvxl4J9BeR3lgiNhVrjqwqs4GXRCQKiMFKJf6nt8Z7S6mjlE4uZ7PCjA6ni9s/XIcAz04ZSaSp\nP2doBiISAZRipc4PdO/epqqNls5Q1SU04i+5sxhvbqmd0LLUfIMhWHizzqw3sEhVv3S/jheRXqqa\n0dB5quoQkVuA+Vip+W+q6iYRucn9/r9VdYuIfAmsx6pV97qqbmzZn1SbEnsJ8Q57s8TslUW7WL33\nKM9NGVVv5XCDoTFU1SUiL6vqaKzPe8jTXM/smy2HTKavIeB4M2f2MZbQeHC69zWKqs5V1QGq2ldV\nH3Pv+7c7LOI55mlVHaKqw9zVRXyOzWEjoRlitnFfAc9/s4OLR3bl0tE1c1cMhibzjYhMlhBfZd9S\n8379ziqKyrxebWAw+ARvxCzKnVoPgPt5jP9M8j0l9hISm9Fl+ov1B4gQePSSYX6yzNDK+B3WjWC5\niBSKSJGINCvFPhC0ZM5s6a6wakZvOA7wRsxyRORizwsRuQQ44j+TfItLXZQ6St1FhpuWzbg28yhD\nuqaSmhDtJ+sMrQlVTVbVCFWNUdUU92vfN9hrIfXNmXVtE88Fw7sw++afNTrG7/6z2tdmGQwN4o2Y\n3QTcJyKZIpIF3E3t2nIhS6mjFIDEJhYZdjhdrM8uYHSPNo0fbDB4gYh8482+UCUyQnj56hMY2aMN\n43qHVeMMQyvAm2zGXcAEd/UBVLVYRMJmdrd6kWHvw4zbDhVRancyOt2ImaFliEgckACkiUhbjmUm\nplC7kEDQ8SabMdKLebUKh8v0+DMEjKZ80qKAKe47ybV+ssfnVG//4r2Yrc20ip6M7tHWL3YZWhW/\nA1YDg9yPnm028FIQ7WqQhubM1IsZtdnr9plEEEPAaFDM3Gn4U0XkM2ADVgX9R7EWQIcFlY05m+iZ\nrc3Mp31iDD3amUaDhpahqs+7K+bfqap9VLW3exupqiEnZr5Ktpz22SaGP/QVFQ7TFsbgf+oNM4rI\n+8CpwFfAi8C3WIWDFwXGNN/g8cwSm+qZZR1ldHob06vM4DNU9UURORnoRZXvnqq+GzSjGqBBz8yL\nVEdbhVWpq8Jpwo0G/9PQnNkQ4CiwBasmnFNEwq7CjScBpCnZjAU2O7tzSph8Qtg4oIYwQET+A/QF\n1nGsJqMCISlmDdExJc7rY48UlZMUawpzG/xLvZ8wVR0lIoOAXwBfi8gRIFlEOqnqoYBZ2EI8CSCJ\n6r1nti7bM19mkj8MPmUMMMRdeiqsefyyYYzr3Y4HZjVesGfj/gJ6pSUGwCpDa6ZB319Vt6rqNFUd\nBPwRq8DwShFZGhDrfEBlAgiRXneZXpt5FBEYYcTM4Fs2AvU3BQsx1BNhd9TuVZYcF801E3p6NY6p\n9WgIBF4HslV1tareiVXp+x7/meRbKhNAYpLAy/mvtZn5DOyUbEIjBl+TBmwWkfki8plnC7ZRNakl\nPstfafGYe46U8OxX2zgOnFJDiNLkX2t3iOR7P9jiFyo9s5gkr453uZR1WfmcPzxsbqAN4cNDwTag\nKVTKTkVJi8e67q0f2ZtrY+q4dLq2MRnCBt9z3LseNoeNWIQoL5M/9uSWUFBqN+vLDD5DRAa5Q/bf\niUhs1bYv7kaaIYWvM3j35dtMer7B7xz3+bI2u40EFa8zGSsXS5vKHwbf8X6V58tqvNfyGJ6fUE+4\nUVr2M/H43K0+sMZgaBhv+pnFApOpvTbmEf+Z5TssMcPrTMa1mUdJjo2ibwfvwpIGgxdIPc/reh10\nas2Z+cBTM1NlBn/jTZhxNlCAVX6n0a64oUaJvYSEJqTlr83MZ2SPNkSYjtIG36H1PK/rtcFgaAbe\niFl3VZ3kd0v8hM1hI8Hp9ErMbBUOth4s5OaJ/QJgmaEV0V1EXsDywjzPcb8OvULDtTyxlt/YHSws\na/EYBkNDeCNmS0VkuKpu8Ls1fsBmt5Hk9K4x5/rsAlxq5ssMPueuKs9X1Xiv5uuQwV8uo6pS4XQR\nGxXppysYWiPeiNkpwHUisgcrzChYGfoj/GqZj7DZi+nocnqVAOJJ/hhlMhkNPkRV3wm2DS3Cizmz\nv106jPu9qAZy8hPf8qezBvDPr7ez9J4zTJq+wWd4I2bn+d0KP1JiL3G3f/FGzI7Sq30C7RJjAmCZ\nwRDaeOOZLbv3DIrKHGTm2rwe959fbwcgI7fEiJnBZzSac6uqe4E2wEXurY17X1hgZTM23v5FVVmb\nlc/odOOVGVo3tbIZD66v99guqfEM6JSMqxnpiqbMlcGXNCpmIvJHYAbQ0b29JyK3+tswX1HisHnV\n/mV/QRk5ReVmvsxgcFMpT1s+b/TYn/VLa/L4pruSwZd4sxry18B4VX1QVR8EJgC/9a9ZvsHutONQ\np7v9S8NitjbzKGA6Sxv8h4g8JSIpIhItIt+ISI6I/NKL894UkcMiUueklIicLiIFIrLOvT3YQjub\nfE5ibBRnDurYtOs0+SoGQ/14I2bCsd5LuJ+HxeewsjGnF+vM1mbmExsVwaAu3jfwNBiayDmqWghc\nCGQA/aie6VgfbwONLY9ZrKqj3JtPCho0NXDYIdm7rhQeTONbgy/xRszeAlaIyEMi8hCwHHjDr1b5\niMqK+V405lybeZQR3VOJjjzuK3wZgocn4eoC4GNVLfDmJFX9Hsjzm1U18Mxl7YtuWunWBy8a0rTr\nGC0z+BBvEkCeBa7H+jLlAder6nP+NswXeBpzJjQyZ1bucLJxf6FJ/jD4my9EZCtwIvCNiHQAfLWa\n+GQRWS8i80RkaEsGcqoViHk31bt6ph4SYpomfu41PqYtjMEn1PvpE5EUVS0UkXZYIZGMKu+1U9WA\n3Sk2lxKHu/1LI9mMWw4UUeFwMco04zT4EVW9R0SeAgpU1SkiJcAlPhh6DZCuqsUicj4wC+hf14Ei\nciNwI0B6enp9dvrApMa5f9ZGTu2fxmuL97Ds3jPokmrS9A3NpyHPzFPpezVWlQLP5nkd8ng8s0Qi\nILr+L0pl8ofJZDT4ERH5P8DuFrL7gfeAri0dV1ULVbXY/XwuEC0idaYXqup0VR2jqmM6dOhQn50t\nNckrth4s4rXFewA46e/fGg/N0CLqFTNVvdD92FtV+1TZeqtqn8CZ2Hwqw4xR8Q0G6Ndm5tM5Jc7c\nGRr8zQOqWiQipwBnYc09/6ulg4pIZ3ErkIiMw/pe57Z03Oaw5oGzm33ur95a6UNLDK0Nb9aZfePN\nvlCkMgEkOrHB49ZmHTVemSEQeLKCLwCmq+ocoNFyMyIyE6sP2kARyRaRX4vITSJyk/uQK4CNIvIT\n8AIwVYPk5rSkes7323OYtXafD60xtCYamjOLAxKANBFpy7F0/BRCsNJ3XXhS8xOi6+9NllNUTlZe\nKddM6Bkoswytl30i8ipwNvCku1egN0lYv2jk/ZeAl3xjYnC5/cN1XDo6LH5eDCFGQ1+k32HNjw1y\nP3q22YTJF6fSM2sg+WNdlqeztMlkNPidK4H5wLmqmg+0w7t1ZsHnaOAq2K1xz2EbDE2hoTmz51W1\nN3Bnlbmy3qo60n0nGPKU2EsQhfiY+lOM12YeJSpCGNY1NYCWGVojqmoDdgHnisgtQEdV/SrIZnnH\n19MCdqnLX1nK0p1H6HXPHHKLw64fsCFIeBPieFFEhonIlSJyrWcLhHEtxWa3kQBIXP1iti4rn8Fd\nUoiPMb2VDP4lrOucbvpfQC931esrAPgpOz+g1zWEL42uchSRacDpwBBgLlZLmCXAu361zAfYHDYS\nG6jL6HQpP2XlM/nE7gG2zNBK8dQ5LQEQkSexEjteDKpVIYyprG/wFm9qN10BnAkcVNXrgZFAWMTk\nbHYbCQ005txxuIiSCqfJZDQEirCtcxo0BL7efIglO44E2xJDiONN/ZlSVXWJiENEUoDDQA8/2+UT\nSiqKGxQzT2dpUynfECA8dU49MbtLCZM6p00hQmBg5xS2HChs8VgC/OZdq0ZDxhMXVHuvpNxBQkyk\nKVhsALzzzFaJSBvgNaxsxjVYoZGQx1ZR1GD7l7WZR2mbEE3P9gkBtszQGgmXOqctXaK26/HzmXvb\nKXx44wQfWWTx2U/72Xm4GIDMXBtDp81nxopMn17DEL406pmp6h/cT/8tIl8CKapaf+vZEMJmL6Jj\nA3UZ12bmM6pHG3NnZ/A7IhIJbFLVQVg3hMctnu/T+D7tWzzWu8uOLQm4beZawPLQMnKtNaTzNx3k\nl1XWiH6yOpu2CdGcObhTi69tCC/q9cxE5ISaG9a6mCj380YRkUkisk1EdorIPQ0cN9Ydxryi6X9C\n/VhzZnVXzC8otbPjcLFZX2YICKrqBLaJSN3VfQ118u3Ww3Xuj4qwBNPhrO5F3vnxT/z6HSssebiw\njEnPfc+BglL/GmkICRryzJ5xP8YBY4CfsELYI7AKDZ/U0MDuO9GXsaodZAMrReQzVd1cx3FPAj5f\nb1PisNUbZlyf7VksbZI/DAGjLbBJRH4ESjw7VfXi4JnkXy4c0YUv1h/w+biRbjFzuuoPiX60Kout\nB4t4ZeEuHr10mM9tMIQW9YqZqk4EEJFPgRNUdYP79TDgIS/GHgfsVNXd7vM+wGp3sbnGcbcC/wXG\nNtX4xrA5yklQV50JIGsz8xGBkabtiyFwPBBsA7yhfbwVHhzRYQTsqTInteETGN604MlLV53AwYKl\nrNrr26oeHjFzuFx1vv9TVj7/+Go7AP9Zvpfh3VK5cmxY5K0Zmok3CSADPUIGoKobgcFenNcNyKry\nOpsaNR1FpBtwGT6oHF4Tl7oodZXXu85sbeZR+nVIIiUu2teXNhiqISL9RORnqvpd1Q0rNT872PbV\nJEIiSIxOZGSHkdXfyG5eVfu+HeqvjdocyuxOIhrxzC55+YdqrxduqztcaTh+8CY1f72IvI7Vewng\nasBXCSDPAXe7U//rPcibhoI1KXOUoWB5ZjUqgKgqa7PyOWeImSQ2BITngHvr2F/gfu+iwJrTOILU\nkdXYvESp1ATf3jAOeuDLyuee7EaDwRvP7HpgE/BH97bZva8x9lF9PVp3976qjAE+EJEMrMXZr4jI\npTUH8qahYE0qiwzX4Zll5NrIt9lN8ochUHSqGt3w4N7XK/DmNI6IoPimi0zbBKstzCOXDCUmypuf\nHO8pqXBWFgtvCNP38/jHm9T8MuCf7q0prAT6i0hvLBGbClxVY+zenuci8jbwharOauJ16qSy/YsC\n0dXXkZnO0oYA09AHLSQ7wgqCS2vMRxVk1X1wI/zm1N50To3l0lHdePW73ezL92124bTPNjH75p/5\ndExD+NFQP7OPVPVKEdkAtW/RVHVEQwOrqsNdGXw+EAm8qaqbPA0FVfXfLTO9YSq7TEfG1uoyvS4r\nn8SYSPp3rL81jMHgQ1aJyG9V9bWqO0XkN1iFCEKOCImoHWbc+0PdBzdCdGQEl4226p+6/OAi/ZSV\nj8PpIiqyfq/PV16mIXRpyDP7o/vxwuYOrqpzsYoTV91Xp4ip6nXNvU5deDyzxKja1T12HCqmf6fk\nyowog8HP3A78T0Su5ph4jcHqMn1Z0KxqAKGOMGNpyzMSU+OjOVBQ1uJxarJq71GTzNXKaSg1/4D7\nMXBd+XxI5ZxZdG0xy8yzMaaXmS8zBAZVPQScLCITAc+Cpzmq+m0QzWoQkboSQFrOW9eP5bznF5Nv\ns/t03KnTlzf4vpkzO/5pqAJIkYgU1rEViUjLK4j6GU+YMTE6sdr+coeT/QWl9Gxn6jEaAouqLlTV\nF91byAoZuOfMcMGoq306bpfUeK6pUn4q0Ow8XMQFLyzm+a930OueOZTZnY2fZAgLGvLMwnpCqdIz\ni6n+Z2QfLUUV0tsn1nWawWCgypxZao2FxhUlENOy786lo7vx4rc76dMhkd05JY2f4AMUeH3xbv42\nZwsAm/Zb9+P5NjudU01j3uMBr/NkRaSjiKR7Nn8a5QsqsxlrVP/IzLVEzlTKNxjqp845M4A3zmnx\n2H07JJHxxAV8e8fpLR6rKTy7YHutfTWXt+YUlbN6b16ALDL4kkbFTEQuFpEdwB7gOyADmOdnu1pM\nZTZjbPWs6L3uatsmzGgw1M/h0sN8uuPT2r/2hzb69Dp/OL2vT8erjwWbD2GraDykeNkrPzD5X2HR\n4cpQA288s0eBCcB297qwM4GGZ1tDgBJHCTGqRMdVb4qdmVdKfHQkHZJjg2SZwWDwEAoRkgKbvTJi\nk320aWvg3lu+l0te/oE9RwITLjXUjzdiZlfVXCBCRCJUdSFWWnFIY6socbd/qRFmzCshvV2C6WFm\nMIQA0swSWb7iu+05jHzkK37+9MJmnX//rI38lJXPr99uXt1Kg+/wRszyRSQJ+B6YISLPU6V9Rahi\nKy+os8jw3lwb6SFwN2gweIuIvCkih0WkzhifWLzg7hu43tt+g15e3XdDNcDI7qmNH+QH/vJJ3WVm\nc4vL69z/1g97eH9FJit251bb72igFY0hMHgjZpcApcCfgC+BXYRgYdSa2CoKidfqjTldLiUzz2bm\nywzhxtvApAbePw/o795uxKddKPz8I+3Wyv6dQit5+sS/fV1r36HCMh7+fDP3/W8DU+pY15Zvq/DL\n2jyDdzS0zuxld9uKElV1qqpDVd9R1RfcYceQpqSiuJZndrionHKHKyTi9AaDt6jq90BDKXaXAO+q\nxXKgjYh08cnF62ifxP9u8kk1EIBzh3ZmTM+23HZGf5+M50vWZh7lSHE5j3y+mYwjJbXazWTl2Sqf\nZ+bZGPXIAt5bXrvGRGmFkwH3z+Phzzf53ebWTEOe2XbgHyKSISJPicjoQBnlC2z2EndjzmNfRk8m\no1ljZjjOaLR3YLMZd2PtfT/NhK/u98nwqfHRfPL7k6uF/p+4fLhPxm4qBaXVq5Jc9spSxvzta978\nYTJyZ64AACAASURBVA83vbe6lo966lO159kWbcupte++/22gwuHirR8yfGhtYHhzyR763je38QND\ngHrFTFWfV9WTgNOAXOBNEdkqItNEZEDALGwmNofN3f7lWALIXvedlAkzGlorInKjiKwSkVU5ObV/\neGsRWU+9Q0cdc0qqsOxlKDzQQhtbdHqz+cOM+ms+exs9/GbrYVxVPDinS1nkg8agZXZnUHq3PfLF\n5noboIYajc6ZqepeVX1SVUcDvwAuBbb43bIWYnOUkuiq3pgzM9dGZITQrW1Idt0wGJqLN70Dgeb1\nBqyXTf+Dh1Kh5Ij1Om83zL8PPvxli4YNVobjDzvrnz1pisDuyjkmOs98tY2jPqhDecv7aznr2e9q\nld/6dushFm71fxftcJgL9GbRdJSIXCQiM7AWS28DLve7ZS3E5iwnXqvPme3Ns9G1TRzRDbSKMBjC\nkM+Aa91ZjROAAk+hcL/hKIcV063nOdusR5fDeixveenWQZ1DKyFk68Eir4/1CN+qjDy+2VJdaKbN\n3khBqZ2DBWX8uMf7SiNLd1k3DDWzJm94exXXB2BZQBhoWYP9zM7G8sTOB34EPgBuVNWQT8sHKHFW\nWJ5ZFTHLzC2hZzszX2YIL0RkJnA6kCYi2cA0IBoqWyrNxfqe7gRseNcJvmVkLoc0d9JGzSaeLeTM\nwR1Zl53fJAEJBKsyvBUf4en5W3l54a5a77yzbG+1R4CYyAhm3jieoV1T+WbLYS4Y0YXMXBsfrMzk\n8hO60a9jcqWYCPDjnjy2HCjkklFd67z6jkNFvLFkD49dNtxnba7CQMsa7Gd2L/A+cIeq+iZ1KUDY\nnXbsuNxdpo+J1948G+cP902Sl8EQKFT1F428r8DNATLHc1WOrUFTjyEtGvGrP/2c5Lgo2ifFcs95\ng3h/RWaLxvM1f/xgnVfHXfnqMvJKKup9f2GNJJEKp6taCa22ieN5+LPNbDtUxCuLdvHWdWMr62SK\nWOMDnDGoY53j3/TeanbllPCbU3vTz8sGxKrK4h1HOLV/ml8KSmzILuBISTkTB9Ztsy9oKAHkDFV9\nPdyEDI5VzE+MiIYI608sKLWTb7Ob5A+DwReoHoun+SgGNaBTMl1SrfnslLhoRvZo08gZoUlDQgZW\nGn9DFJbasTuPebs7DxdX8cy8Fxpv/ls8c3Afrszi2jd/5H9r65xqbfGc2UUvLeH6t/wbDj0uJ48q\niwxHxlXuM9XyDYZm0r6uNWBVxcz9w+vjO/pbJvbz6Xjhwk3vrWF3lVqPVf9Z6xObQQ/Mo7jcQb7t\nmJB+uDKLDdkF9V5n9rp9DHrgS3YeLiLrqPX7uD/fqk3pdCmzqlxr9d6jVDh8G072NcelmFW2f4k6\nJmZ789xrzMycmcHgNZmFmdQ5Y2LLpTLM6KyRreejObSzh3TirnMH+mSscOZvc7ZQ7haS+/63oc5j\nyuwuJjz+DaMeWVC57/Ule7jopSUcLiyr43gnX206BMCWA8fmJj0O2Fs/7OH2D4+FVadMX87f5mxu\n8d/iTxqaMwtbKhtzRh3zwva6PTNTl9Fg8J7bF93Op/Ht6n7zyA7r8f3/g4cKjv0S5u702fVvntiP\nmyf2o9c9cwCYdtEQHv48tH9Ug0VxuZVNWnPOa9zj33DFid2JFCEtOYbp3+/G7jx2g1JX2DOnqPY6\nwqZkXwaD49ozS4xOqtyXmWsjLSmGpNjjUr8NBr+gqjB1Rt1vFu0PmB3p7rnu607uxZVjugfsuqFM\nU6axPlmdzYersnh54a5qQgbw9Pxtlc8rI8e+MDDAHJdiVumZxRwTs73u1i8Gg6GJJHmRgVaQ7dfS\nHV/efiprHjgbEeGpK0b67TrhxMyVvsv29CwjKCxzcLiwzG+LpHcc8t9yi+NTzNwJIIkx1at/9DQ1\nGQ2GJuH1j1q+f9PoE2KiaJcY8//tnXl4FFW6uN9TvWbfEwJJICgIQhCEQR1BHccFEBWXGRlXxOXq\nRXScn4p3HBG94qBXHa8yV0EHt5GACgMouIDKKIqyyyoSIGwCCWHJnk7S5/dHdTrd6e6kk3S6O53z\nPk+eVJ06VfVVd1d9db7zLR16js7Gqys949iAdqW9mv31HoY/84XP7VJKJr61lu/3lGC3S4+MJA28\n9u/dnPXk5x7tjy0KbKVyVyJamUVb9BpJNXX1HC6tViMzhaKVSH8NTlK230W/tgp2f9m+Y/iJBRsm\n6oJyrs5IYYn38IFnlu3gy5+KGD/7ex5ZsJl+j3/qtd+MT37ySNzc0USkMquo1Yey0dYkAA4cr0JK\n5ZavUPjLoNRBrduh7DD83zntO+myh+Dda6Co41O/7rRO4HPzwx1+ns7K8u1HPdp+OlLG69/sda5/\nuP4gADtdMrWs3FlEpc33S8KavcedSu6TLYfd8li2l4hUZpXVpxBSYrXqQZf7HW75SpkpFH7S2umv\nBXc0v/3UIbC34LJf/LP+v9p3bFQgydU8H9iB4jRxiGu1rzv0+IXWGzlNeI87CyaXv/Q1+0oqKCgq\nZ8Kbaxn392+d27zFpj3ocPm/970N/PaFfwdMjohUZhU1J4mSEs2RMd/plq9izBQKv6iq04Nn/TYz\nNsexAvjbmfDtS37u0LImTY3V589GnJ7KHSNy2fPMmHYIGHiWmx/hRfNrHXb8qwx6SquxmmfF61Dw\n9NIdzhHZz0cbR1t9//KJR98DLWRAaSsR6adeVVPqlmR4X0kl0WaD8wZQKBTNs+uEHkNmq3dklLj2\ndVh4l/8HWPM6DHf0P+VwDtmzEkb+KSDyLXtgJAVF5fz6tNSAHC8QpHKKGFHFPtkNTXS0c3t4Oc8v\n336UoT2TvG77eLN7CIdE92MINJE5MrOVES0bC3PuP15JTnJ0hyTQVCgiGac346Dft27HZQ/p/ze/\nr/9BY4kY32fz3vzLJg/nkvQ4q4ci++L/XdiiWHk9Elrs01bWWe/l35bAKGt/kSGq/eaNGZ/85LX9\n8SYejFJK7nx7XcDPH5HKrLK2Qq8y7TQzVqj5MoWiDbiZGQdc07qdty/RR3M/5uvr+76FT6ZAhUsR\nzAV3QkETV/C6arA5TFHb/gWzL4QtH7Ysqx+DlThrRBqjwpqm82a7iyv4ZtexgJ8nQpVZJdEOM6Pd\nLjlwokrFmCkUbcDukmfxz+lp/GdGK6pTv3+LZ9sPr8H/9IatC/T1LR/APx21fhu00TtXwTOZ8Hzf\nxpRZxd7f+l2J96KoVpof5D8Ni9zapozq55f48ZTzn4bFCMI7wW64U2ELvEnRGxGpzCrqqohxVJk+\nUlqNrc6uYswUijZwtLLR4++jwk/5JjoqMAf+cCJs/GfzfcqP4lEzrRnS4xsTi2eJYm41fEYv7SiP\nmHQzZ6H1Rh4+9jj3DvAvvmy6aQ6PmOYzQnM3k50h9vOz5Ra6E/jRhb+IMJszCwciUplV1Vc7R2b7\nVOkXhaJdfLT7o4458GKXeqIlu+GIl4zwDbrsx3l6Hz+Za3qap0xve7QPqVkDfx/udZ9pxrfc3Olj\n0LPNNw2uvsnwBWZRzyWG9X7L440+4iCni4PtOkZnRMPOY8Z/kkFgExdHpDKrsNv0kZk5tjHGTLnl\nKxRt4onvnnBbn5zeAR6Er5wN9Z6Z2p2xZ6WH4LURje1SQp3vIpgJosLnNl9MMH7u5k7v6lxxqbaO\nmw3LHe067XW9WG55hBWWR9p5FMgSRaQSnNg8f4ijkkLrjfxW867sz9F2cJdxGf9jmhXQ80akMqu0\n1xIljKAZ2FdSiVETdE+0tryjQqFokZUxQbRybHm/cbnWJT7pu5fh6TR3ZxIXWuvld3E/38mUBZLX\nzS/ytOlNskRxWHkQSmCV5Y+ss97rse0sUcBV2rce7UbqGCZanoNsDRdqP/KYUTcb93GMNicZFze7\nj5HAzqVFnGuPlJJKWUeMZgJg3/FKeiRFYTREpN52o7a2loMHD1Jd7VmMTxF4rFYrWVlZmEymUIvS\nodTaa9l5fGfLHYPJj/P0/+VH4OdPoOwIXPAQt57Xk3dW7/NQOJ8/eAG86vtwI05PhUL3Nm8jsFWW\nB5hbd7Gj3Z95K4k/Y7iR2mYqpJUNsi8m6rjZsJx36i+jHkOz+1mF79HpYstUAJZUn+/W/ohxPncb\nl3JFzXS2ydyWL8EP3jY/C8D0upudn4s9yGOliFNmVXVVSCDaYAH0bPldxfnj4MGDxMXF0atXLxVT\n18FIKSkpKeHgwYPk5gbmgRDOXP/R9aEWwTcNc2+pfXjq8t/w0HAL8bPcc/71aqU383PXD+KSjzcC\nnkrL4PBu9EeZCaSHYr1cW8MGex+3tnfNM3Q5q+dyl2Epj5jmU4uRf9ZfwlliNz/K070ef5JxiUdb\nN0r43jrZua5hd1Ms/YQexJ4iSjsk9lpzHDTYLioRN1xpqGUWY9DNil0pxqy6upqUlBSlyIKAEIKU\nlBQ1Cg4174xrXH7/VpiRTXyBp8OK2ej9UWcyCDIpYeKKwc62s7IT+f2w7BZP7c9d1rSPmVpmmV8i\n3zzd2TZWW+3WJ1boqcTiqeT3hpUstkzlMm1ti+ducFQZaXB3pNliaSFvZgfRoMRjqQxKhYLIU2YN\n5V+M0ZystFFaXdelnD+UIgse6rMOAyqKPNu2LvR7959j/4PPrnYfQyye5G6WazoCa1g3UscV2vf0\nE/sZIAq9Hl9rEqPWsG+2KHa2TTW969ZHuiydLvRUUD3FUYaKnWy03M1fjO9yhjjgca5hmm4Kvs3g\nXkcsRnhxrOlAhHNkpt8fW6138qbp2Q4/b4eaGYUQo4D/BQzAG1LKGU223wRMQX/RKAPulVL+2J5z\nVtTqXkzRpujGBMNdZGQWLkyfPp25c+diMBjQNI1Zs2YxdOhQpk6dygcffEBMjP5y8bvf/Y7HHnsM\nAIPBQF5eHrW1tRiNRm699VYefPBBNC3i3rcUHc1RLy7+PhA1pcRvn9tsn5F909zm04xCd1z4sym/\n5eP73OLdCHeaOORUAsKlV7o4yQPGhSSJcu40eibv1fvrvQdqhV63Bw7JWO17PrcPw4bnfLG3vJQj\nDNugg8ubdZgyE0IYgL8DlwIHgbVCiCVSyu0u3fYCF0opTwghRgOzgXYVRWowM0ab49h3XMWYBZvV\nq1fz8ccfs2HDBiwWC8eOHcNms/GXv/yFI0eOsGXLFqxWK2VlZbzwwgvO/aKioti0SS8NUVRUxI03\n3khpaSlPPvlkqC4lbPDjpfAiYDH6/QSwUEr5VFCFDCbTApxf8UDzmefPLP/Bbb01JrOmo7qZplc8\n2tPFSefyF5aHeblON51q2J2K7S7jMr/P6Y3uHOM76/1MsLW/httIbQszza/wRt1onq7zzPLSdGTW\nQKH1Rr6uz+MCg/8vG62hI197hwMFUso9UkobMA+42rWDlPI7KeUJx+r3QFZ7T+ocmZnj2F+iL3cV\nB5Bw4PDhw6SmpmKx6A44qampJCYm8vrrr/PKK69gtepzmXFxcUybNs3rMdLT05k9ezYzZ85sTHTb\nRXF5KRwNnAn8QQhxppeu30gpBzv+OlyRdVSCp49io9nZnHeoa6B1a2jN72jTXKhtnAsdeszdxXx4\niv/zpD9bb2OwKGCC4VMu0H7kUj8CrV1HZhdqm/0+1/lNMpW4MlgrAOD3hn8zVNNj967QfvDZvynn\natu50aDn0IxHHyR0E80HPUvpOS51VWSBNtJ3pJmxB+Bq2D1I86OuOwDv4+dWUGnTFViMJYF9JZWk\nxVmINkec02aLPPnRNrb/UhrQY57ZPZ4nrhzQbJ/LLruMp556ir59+3LJJZdwww03kJSURE5ODnFx\ncX6fq3fv3tTX11NUVERGRkZ7Re/MOF8KAYQQDS+F25vdq4NZGR3FxZVVzfbZYzKSW1vXqofWn9P0\ngOwte/d779BSCixfvHK2/30X3QtHt/ncnHFyU7O7F864AqY1rr9gepXTtMNufczCd4zVA8Z/AWAQ\ndvprPj4HL0wyLuHtusu9bmtQkCO0rc45tMsNa3msbiJ1TdRAtjjKYZnCULGLV81/Y2jNa8wzPw3A\nvPrfOMeUBuzEU04psW77N4xcW3p9EAEukxMWExJCiN+gK7MpPrbfLYRYJ4RYV1xc7K2Lk8pqfaAX\nbUlg3/FKeqpRWVCJjY1l/fr1zJ49m7S0NG644QZWrlzp1ufNN99k8ODBZGdnc+CA50S2wg1vL4U9\nvPT7tRBisxDiEyFE828cAeCUpvFMcpJPg9saq4Wrs7qzIC5MnK+O72ld/9Uz236uGvewgKaKzF8e\nMPrvyNLAGqv3keur5v8FIF40Bp4nigpeNrlfZzKlfGN5kALrrcy3/DfJopyLtY3O7fcYljgV42jD\nWjZb7yaNE87tFmy844g5C7Z7VEcOWQ4Brv6tWY42N4QQg4A3gNFSSq/h/FLK2ejzaQwbNqxZdd6g\nzGKsSewvqeTXp6e0SfjOTksjqI7EYDBw0UUXcdFFF5GXl8esWbPYv38/ZWVlxMXFcfvtt3P77bcz\ncOBA6uu9v6Hu2bMHg8FAerrvzAwKJxuAHClluRBiDLAI6OOtoxDibuBugJycnDafcGqafl/tNRt5\n/YjnC+Yeh6lwh9kM6NaS9RYL1Zrg/KrOG85QLgSxLZksV/41OMIEgDGGNU7HjKnGd+jrxUsyTTSm\nysoSxeyVmW7bu4lGZfai6f/8PnegkyV35MhsLdBHCJErhDAD4wG3CD8hRA6wELhFSvlzIE5aUa1P\nphrNSRwpre5SbvnhwM6dO9m1a5dzfdOmTZxxxhnccccd3Hfffc64rPr6emw279kLiouLueeee7jv\nvvuU+7sfL4VSylIpZbljeRlgEkJ4TaAopZwtpRwmpRyWltaKci4+2BQT3+z2hm+vFpjQPYN7urm/\nnPwtKYG83JxOUWRli9nMeb2yWdFS5YD2jOpCwCdm3SA20fip7nXYhBmmN9zWmzp2nKU1JoAeqTXO\nicWLCka3Yl6uvXTYyExKWSeEuA/4DN0La46UcpsQ4h7H9teAqUAK8H+Oh1adlHJYe85bWXMKk5Sc\nrNPNi8qTMbiUl5czefJkTp48idFo5PTTT2f27NkkJCTw+OOPM3DgQOLi4oiKiuK2226je/fuAFRV\nVTF48GCna/4tt9zCn/4U3Kq9YYrzpRBdiY0HbnTtIIToBhyVUkohxHD0l1TvSQsDjdDTLRUbNLaZ\nLVxU5X0e7doejW/zJzSNOQnxPHDiJG8l6MrQTnDnPE5qGtF2O+ZW7LPVovf+PsrKJS3MF3Ym+msH\n6EHz0zcNuIYLNPC06U3nsu6qr382A7R9TvNmMOhQzwjHW+KyJm2vuSzfCdwZyHNW2sqIttv5pdoE\nSBVjFmSGDh3Kd99953XbjBkzmDFjhtdtvsyNXR0/XwqvB+4VQtShP0nGy3a6gaZHp1NU6SUg2Ru9\nRnKtfQ8nDQan40bDyQtMJuxAobnRQ/HZlCSWxsaQWl/vfMcfkpvDy0f9e6AGgpE9szinqpo3jvh5\njRHOt9YH/Oo3yrCWPxi/8rk9VQTW6aw1RJybX2VtOTF2yYFKI1CrHEAUnR4/XgpnAgG1bXWL6eaX\nMpNI1o2axsnPbgfABphpVGbro6zc16Q6tc1hOn4+JQmji869388q1t9GWTlu0LiyvLLlzs3wQ1Tr\nKml07SARnSRR3nInPzlH+wlzACOpw8KbMZBU1lYSLe0UlmnEWowkx7TGkKBQKAC/4/vs0s6WY43z\nJDd37+bRp7nq1HVtmBO9p1u604XfGwtjY8jLzaGkHdlj3oqPY0ms9/n2Lj+LG0BcA8bbS8Qps4q6\nSqLtkoJTgpzkaOVAoFC0gXMy/UvEU2uvRRONj5EdjnklL/GyTkQHB8IviNPjng6Y2m54eiElicfS\nuqYndDA5T/Mdz9daIk6ZVdZXEy3t/HxCOX8oFG3lvsH3hVqEsKM5Ba1oG+MSCgJ2rIhTZhX1NcSg\nse9kjXL+UCjaiEFrviikK8+ve95t/dc5WZxoZv/OrhMCnLiiS9Orbm/Lnfwk4pRZlb2WKIzU1ksV\nY6ZQhIAyg8asJN/JgD/zMRfljSohyMvN4fUEz3i2YT2z+GtyEpU+phI6u9JsDXtMxk7poNLDppSZ\nTyrstViEbitXZsbgI4Tg5ptvdq7X1dWRlpbG2LFjQyLPpk2bWLasfRnHFcGnQgi+t1oodThxzIuP\n9ehTo2nMTYjjidRkt/YSgz4qXB1l5ZDR/xFmZ+XbKCtXZ3Xno1a8JEQiEafMKrFjlnpMi8qWH3xi\nYmLYunUrVY7g2eXLl9Ojh7dUgsFBKbPOyaNpKdyVmUGxoVEZVQrB9V68Jfc7HD0OGA3clJnBIcf6\n35MSuSKru1vfGclJfBrj+Vwo0TTKfYzwJLDbxZnE3xHfbZnpPJec6GfvttOYOqyZagNdgIhSZrX2\nWmxINGnCZBB0T2wh7YyiQxgzZgxLly4FID8/nz/84Q/ObcePH2fcuHEMGjSIc889l82b9RIX06ZN\n47bbbmPkyJH07NmThQsX8sgjj5CXl8eoUaOordXjUdavX8+FF17I0KFDufzyyzl8WE/ietFFFzFl\nyhSGDx9O3759+eabb7DZbEydOpX58+czePBg5s+fz7Rp03j++cY5noEDB1JYWEhhYSH9+vVjwoQJ\n9O3bl5tuuokVK1Zw/vnn06dPH9asWROsj08BrHQonNWOWLAio5G/JSey0+IZatOQXumBjDQ2Wy1u\n2+qbKKj3EuJ4ON3Trf+inlmMyu7u0Q6wJDaGcVndWRXVuufJBquVd72YRwNNwxU2TTPV1YiooOnK\nWkcQZb2JrKRoDFoX/nI/eRSOBLgIXrc8GO09g4cr48eP56mnnmLs2LFs3ryZiRMn8s033wDwxBNP\nMGTIEBYtWsSXX37Jrbfe6izKuXv3br766iu2b9/Oeeedx4IFC3juuee45pprWLp0KVdccQWTJ09m\n8eLFpKWlMX/+fB577DHmzJkD6CbNNWvWsGzZMp588klWrFjBU089xbp165g5U48p9lVDDaCgoIAP\nPviAOXPm8Ktf/Yq5c+eyatUqlixZwjPPPMOiRYva+QEqWsvLLiObefHeSwjtsJjZZjZT6+Nhnpeb\nw4PHT5BV23xRzVMGA7VApUt82vdWC9vNugJd1VJOxgBxShNsN5s5r7qmVft5mzOrEWCWXWP+MCKV\nWV2dWZkYQ8igQYMoLCwkPz+fMWPGuG1btWoVCxYsAODiiy+mpKSE0lI9Bc7o0aMxmUzk5eVRX1/P\nqFGjAMjLy6OwsJCdO3eydetWLr30UkBPgZWZ2Zjz79prrwX0lFqFhYWtljs3N5e8vDwABgwYwG9/\n+1uEEM7zdzXmXD6HiZ9NDLUYfjG+Rzd62Xxnk5iZlMglFS1nDHkwI41/uyituzIziLa3Lg2yBAbl\nNlYkyMvNYemBX8ipa7lCdYHJxNTUZLZYLawuPNBshv56YIvF7FLZ2Z1TmsaInllMPn6Su0+FLs1U\nsIgsZVan/1irbSbl/OHHCKojueqqq3jooYdYuXIlJSX+5bxtqE6taRomk8kZ8K5pGnV1dUgpGTBg\nAKtXr252f4PBQJ2PB4fRaMTu8nBqyOLvun/DOV3l8XW8SCbe3PEmskBiayZBQj3eq2O/nhDPXS4P\n+n97GX1VNskkcrgNTiUro6O4tbSM76KsFBqN3FjmmRZqn9HINVmNL2d1QjRbIfuNxHhmJiUyqlwv\nsdM0Du6YQZf749iYoCmz9RYLNkGrR5WBIKLmzCps+g+kps6qRmYhZuLEiTzxxBPOkU4DI0eO5L33\n3gNg5cqVpKamEh/v30PzjDPOoLi42KnMamtr2bat+QwCcXFxlJWVOdd79erFhg0bANiwYQN79wbO\nNVgRWn5pJuOHXQivIQEvJyf6dO33xaroKJ5PTqS6GYeRppRrGhssFv6jWzp/beJ92cDYJnN227zM\nD57SBA0puWcm6SbYIodyPWbwrmSDaWKc0D2DuzNDUxk+opRZZdVxAOrsUfRM6dpuqqEmKyuL+++/\n36N92rRprF+/nkGDBvHoo4/y9ttv+31Ms9nMhx9+yJQpUzjrrLMYPHiwzwz9DfzmN79h+/btTgeQ\n6667juPHjzNgwABmzpxJ3759W31tisjinF7ZLXdyoVYI3k6I55oenp6VAIVelOqrSQnc1r11D/l7\nuqVzVY9MbnIohwohGNEzmxeaeEgaHNpzhRcvza5ERJkZK6p0c1aNPUaZGUNEebmn+aSh6jRAcnKy\nV0eKpo4Zrsdx3TZ48GC+/vprj/1XrlzpXE5NTXXOcSUnJ7N27Vq3vp9//rlX2bdu3epcfuutt5zL\nvXr1ctvWVciIDs0bdmfhoMm7K/x4L+EDLeHLiL3Xxd2+zGHufDchniscpkXwHJEUGzR2m0yk1OuG\nVX8qOn9vtXC6rZZUP+cHK4SgVggSm+n/XnwsA2psDK7xLMJbC/wxI41JJ05ypl9nbJnIGplV6+W7\nbfUxysyoULSTRGsitw+8PdRidDqq2pCtf64PT01f/C05ybmsuSireuDinCzuysxwJlz2x8x4V2YG\nt7fCPHhZdndG9sxymlrfiY8jz8XpBWBGSjK3NFHs78bHMbRnNnvMJr6OjuLx1MAlc44oZVZVrZcT\nsFgSsZoiP/Jfoeho/jRUVftuCRt6kHdrso18HWXlpIvS+ya65dpqdhetVGJo3He/sXH0NthFobyX\noCvIArOZQqNvI1yDKnQtoFojaLbSWKljfu5XvbIp0TTeTmheGb8fF8ud3dJ5LiUJm2vIVCtygLZE\nRCmzippTACTEpodYEoUicugRG7oMLuHO88mJrI6ysjQ2hlHZPTjm56hsUrd0RvbMYkivbA4YDXzv\nR0D259GN1qYCc6NzyCE/St00DSbfYTbxfHIiEncvzxE5PZDAsF45jPERRN6UF5MTKWpGWQL8d2qy\nWzHUWYmO3J2W1o1ImyOilFmlTXc/TU5UykyhCBRLr1kaahHClrcT4t0yb8xO9J1g2Rt1QjAmM+rh\nTQAADTRJREFU27+XhRdSklru5IOtZjP1wIroKCTw+x6ZvJ0QT5UQbsrslMHA3xxekkeMRubHxTI9\nJYlig29VsSTOM2/mwhbyRC7vAGeViHIAKaspI9puJy3dv/LrCoWiZVpTDqYrsjCu8cGd34K5ra08\n0s5CofkJcXSvq+OFlCSeLTrmbBfAK0nu3pFvJjaGyjztCCMoMhgYUGPjleRENuzd3+L5nnCR96lm\nlLAIYOBARI3MTlSWEm2XpKUpZaZQBJJnRjwTahHClq+C4BL/SQAy4h91zOmVNIlHc1VevijVNF5x\nhAT4iq/zxQetdG5pKxGlzEptFcRIOz3UyCykLFq0CCEEP/30EwCFhYVERUUxZMgQ+vfvz/Dhw91c\n3xsYN24c5557rlvbtGnTEEJQUNBYkfall15CCMG6des69DoUjVx52pWhFkHRTv7pSHr8Xs+BjY3n\n/9Gvfde5zHdVZA8LqFyBIqKUWUVdNVY75KR1fNkFhW/y8/MZMWIE+fn5zrbTTjuNjRs3smPHDubN\nm8dLL73Em2++6dx+8uRJ1q9fz6lTp9izZ4/b8fLy8pg3b55z/YMPPmDAgAEdfyEKr8y6ZBY397+Z\nvNS8ljsrwo5D5Yecy8MPLWj1/pcajgZOGIvnfFtbiShlVllfg0UKEqO7dl2fUFJeXs6qVav4xz/+\n4aaAXOnduzcvvvgiL7/8srNt4cKFXHnllYwfP95jv3HjxrF48WJAz6yfkJBAaqpnGQ9Fx3L/kPuZ\nPmI6v+7xa6YMn8I5meeEWiRFZ8cauIFHRDmAVEsb8XbNmaC2K/Psmmf56fhPAT1mv+R+TBk+pdk+\nixcvZtSoUfTt25eUlBTWr19PSorn5PXZZ5/tNEOCPpqbOnUqGRkZXHfddfz5z392bouPjyc7O5ut\nW7eyePFibrjhBrdRnSI43DXoLrf1SYMncWXvK5n85WT2l3k6BcSaYnn83MdJiUrhzs/vDJaYik7E\nzhM7A3asiBqZVVOPVUSUfu505OfnM378eECva+ZqanRFumQDP3r0KLt27WLEiBH07dsXk8nkkT6q\nYcS2aNEirrnmmo67gDBECDFKCLFTCFEghHjUy3YhhHjZsX2zEOLsYMhl1Iz0TuzN0muXsuW2xtp5\n745+l2//8C2rb1zNmN5jOCfzHJ447wmP/T+/7nMWXNV6M5dC4Y2IevJrmiTOaGm5YxegpRFUR3D8\n+HG+/PJLtmzZghCC+vp6hBBMmjTJo+/GjRvp378/AO+//z4nTpwgNzcXgNLSUvLz85k+fbqz/9ix\nY3n44YcZNmyY31n2IwEhhAH4O3ApcBBYK4RYIqXc7tJtNNDH8XcO8Krjf1CZd8U8DJqBfsn9PLZd\n3/d6+qf053D5YR5c+SAAmbGZZJLJVaddxZLdS4ItriLCiChl9nlNLDIhp+WOig7hww8/5JZbbmHW\nrFnOtgsvvJADBw649SssLOShhx5i8uTJgD6a+/TTTznvvPMA2Lt3L5dccombMouOjubZZ5/tilnu\nhwMFUso9AEKIecDVgKsyuxp4R+rD3e+FEIlCiEwp5eFgCjogtXmnnAEpAxiQMoB/XfUvzIbGDBbT\nR0xn+ojG73rn8Z1M+mISRyt1R4OJAyey88ROvj30rdvxesT2cHNmaOD87ufz7S/f0j+5PzuO72jP\nJSk6ERGlzKguRWR0nbf2cCM/P58pU9xHhNdddx1//etf2b17N0OGDKG6upq4uDjuv/9+JkyYQGFh\nIfv27XNzyc/NzSUhIYEffvjB7VgN5ssuRg/A9W3gIJ6jLm99egBBVWb+cnrS6c1uPyP5DFb8bgUf\n7f6IUlspN/a7ESEEO0p2MGfrHCYPmczyfcu5+vSrSY1K5VD5IQzCgK3ehkCQHd9Y0uWTvZ/w/Lrn\nmTBgAjf1vwlNaGwq2kS9rGdI+hAEAolkY9FGhmYMBaDMVsaKfSuorKvkaMVRzAYzszbP4syUM+mf\n3J+b+9/MDR/fgM3umQ0e4No+17Jw10K3tn7J/dzmsKON0YzOHc2CXV3bzHp2euAs4kI2U8k0HBk2\nbJj0GV+05UOIzYDckcEVKkzYsWOH03SnCA7ePnMhxHopZUCCcYQQ1wOjpJR3OtZvAc6RUt7n0udj\nYIaUcpVj/QtgipTS40YRQtwN3A2Qk5MzdN++fYEQUwFU1FYQZYxCE95dEWz1NqQjra/F0DgdcqL6\nBElWPUuGlBKJ5JfyXzBpJtKj06mX9djqbUSboqm311NVV0WsOZZfyn8BoNZeS1pUGnZpx+KYZjFp\n7h7dNfU1FFUWsfP4Tnon9qZ3Qm/2l+6nuKoYKSUJlgQyYjKwGqwYhIGa+hqiTdFU1FZQVVdFUWUR\nmtCwGCxsPbaV87qfhyY0Ei2J7Dqxi8zYTMpsZZysPolRM1JqK2Xvqb30SerDkPQhnKo5xeGKw2hC\no6SqhOKqYhItiVyQdUGzn2lr7qXIGpnlXR9qCRSKQHMIcK0emeVoa20fAKSUs4HZoL8YBk5MRYyp\n+SwdrqZVVxoUGYAQAoEgKy7L2WYURoya/qg2aAZizXpsVvdY/xIBg648s+OyyY5r/JnkxOeQE+99\nWiZa07OaxJhiiDHFkBrVGAqTm5Dr1veM5DMAiDfHuyWl/lW3XzmXEywJJFgceSvbnmKyWSLKm1Gh\niEDWAn2EELlCCDMwHmjqLbEEuNXh1XgucCrY82UKRaiJrJGZQhFhSCnrhBD3AZ8BBmCOlHKbEOIe\nx/bXgGXAGKAAqARURU1Fl0MpswhDSqmCxoNEsOabpZTL0BWWa9trLssS8Ix/UCi6EMrMGEFYrVZK\nSkqC9pDtykgpKSkpwWptuUKwQqHoeNTILILIysri4MGDFBcXh1qULoHVaiUrK6vljgqFosNRyiyC\nMJlMziwaCoVC0ZVQZkaFQqFQdHqUMlMoFApFp0cpM4VCoVB0ejpdOishRDHQXA6eVOBYkMTpaNS1\nhCctXUtPKWVasIRpKy3cS+H8fYWrbOEqF4SvbAG7lzqdMmsJIcS6QOXFCzXqWsKTSLoWX4TzNYar\nbOEqF4SvbIGUS5kZFQqFQtHpUcpMoVAoFJ2eSFRms0MtQABR1xKeRNK1+CKcrzFcZQtXuSB8ZQuY\nXBE3Z6ZQKBSKrkckjswUCoVC0cWIGGUmhBglhNgphCgQQjwaannaixCiUAixRQixSQjho7R2eCKE\nmCOEKBJCbHVpSxZCLBdC7HL876ASfYHFx7VME0Iccnw3m4QQY0IpY6AJ9r3U2t+LEOK/HLLtFEJc\n7tI+1HHPFAghXhbtLB8hhMgWQnwlhNguhNgmhHggjGSzCiHWCCF+dMj2ZLjI5jimQQix0VEFPThy\nSSk7/R96nafdQG/ADPwInBlqudp5TYVAaqjlaKPsFwBnA1td2p4DHnUsPwo8G2o523Et04CHQi1b\nB11v0O+l1vxegDMdMlmAXIesBse2NcC5gAA+AUa3U65M4GzHchzws+P84SCbAGIdyybgB8fxQy6b\n45h/AuYCHwfr+4yUkdlwoEBKuUdKaQPmAVeHWKYui5Tya+B4k+argbcdy28D44IqVBvxcS2RTNDv\npVb+Xq4G5kkpa6SUe9ELkg4XQmQC8VLK76X+JHyHdv7GpJSHpZQbHMtlwA6gR5jIJqWU5Y5Vk+NP\nhoNsQogs4ArgDZfmDpcrUpRZD+CAy/pBR1tnRgIrhBDrhRB3h1qYAJAhpTzsWD4CZIRSmAAwWQix\n2WEi6xQmUz8Jl3vJ1+/Fl3w9HMtN2wOCEKIXMAR9BBQWsjlMeZuAImC5lDJcZHsJeASwu7R1uFyR\noswikRFSysHAaGCSEOKCUAsUKBxvWp3ZjfZVdDPcYOAw8EJoxYlsQv17EULEAguAP0opS123hVI2\nKWW94xmRhT6aGRhq2YQQY4EiKeV6X306Sq5IUWaHgGyX9SxHW6dFSnnI8b8I+Be6+aczc9RhOsDx\nvyjE8rQZKeVRx4PEDrxO5/9uXAmXe8nX78WXfIccy03b24UQwoSuyN6TUi4MJ9kakFKeBL4CRoWB\nbOcDVwkhCtFN1BcLIf4ZDLkiRZmtBfoIIXKFEGZgPLAkxDK1GSFEjBAirmEZuAzY2vxeYc8S4DbH\n8m3A4hDK0i4abkoH19D5vxtXwuVe8vV7WQKMF0JYhBC5QB9gjcOEVSqEONfh9XYr7fyNOY7zD2CH\nlPLFMJMtTQiR6FiOAi4Ffgq1bFLK/5JSZkkpe6H/dr6UUt4cFLna67USLn/AGHRvo93AY6GWp53X\n0hvdw+dHYFtnux4gH938Votu674DSAG+AHYBK4DkUMvZjmt5F9gCbHbcjJmhljPA1xzUe6m1vxfg\nMYdsO3HxcAOGob9Y7AZm4kgK0Q65RqCbwzYDmxx/Y8JEtkHARodsW4GpjvaQy+Zy3Ito9GbscLlU\nBhCFQqFQdHoixcyoUCgUii6MUmYKhUKh6PQoZaZQKBSKTo9SZgqFQqHo9ChlplAoFIpOj1JmCoVC\noej0KGWmUCgUik6PUmYKhUKh6PT8fx0zyKTKXWYrAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x266005902b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 2)\n",
    "fig.tight_layout()\n",
    "\n",
    "ax[0].plot(valid_accs_SGD)\n",
    "ax[0].plot(valid_accs_momentum)\n",
    "ax[0].plot(valid_accs_ADAM)\n",
    "\n",
    "ax[0].set_ylabel('Validation Accuracy')\n",
    "ax[0].legend(['SGD', 'Momentum', 'ADAM'], loc='lower right')\n",
    "\n",
    "ax[1].plot(losses_SGD)\n",
    "ax[1].plot(losses_momentum)\n",
    "ax[1].plot(losses_ADAM)\n",
    "\n",
    "ax[1].set_ylabel('Cross Entropy')\n",
    "ax[1].legend(['SGD', 'Momentum', 'ADAM'], loc='upper right')\n",
    "# ax[1].set_ylim([0,1.5])  # <- Use this to change y-axis limits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Short Answer\n",
    "\n",
    "How do SGD, SGD with momentum, and ADAM compare in performance? Ease of tuning parameters?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "Compared with SGD, SGD with momentum and ADAM are much better optimizers. With the same learning rate = 0.001 (momentum = 0.9 for SGD-momentum and default setting for ADAM), ADAM has the fastest converge speed, SGD-momentum ranks at second place and SGD has the lowest converge speed. As for the final cross entropy (or accuracy on validation set), the ADAM and SGD-momentum can achieve relatively higher accuracy than SGD can at the end of training phase. \n",
    "\n",
    "As for the parameter tuning, it is difficult to adjust the eps, beta1, beta2 for ADAM, in most cases I would like to use the default setting, which have good performance. Though SGD-momentum have one more parameter than classic SGD, I think it is very worthy to give SGD-momentum a try, we can set the proper training rate and then adjust momentum across validation set. Overall I think both ADAM and SGD-momentum are very good optimize algorithms.\n",
    "\n",
    "For the CNN experiment, I got the same conclusion. I did it in CNN_Optimizer.ipynb\n",
    "BTW: Though CNN have more complex structure than MLP, the difference among 3 optimizers seem to be smaller than that of MLPs. Besides, I set the training rate=0.1 for SGD-momentum in MLP and CNN, the MLP converges after step 250, however, the accuracy of CNN on the validation set remained around 10% after several steps. I find out that lr=0.1 is too large for CNN, but it is acceptable for MLP. Does it mean CNNs converges faster than MLPs? Or it is relavent to the size of the networks?\n",
    "\n",
    "Thank you!\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutional Neural Network (CNN): *Optional*\n",
    "\n",
    "Adapt the MLP code above to train a CNN instead (*Hint: you can adapt the code from the 01D_MLP_CNN_Assignment_Solutions.ipynb for the CNN just like I did for the MLP*), and again compare the optimizers. The more complex nature of the CNN parameter space means that the differences between optimizers should be much more significant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}