t81_558_class_07_2_Keras_gan.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4mB_sFildiDh"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_07_2_Keras_gan.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "303iHntmdiDj"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 7: Generative Adversarial Networks**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d5n2iv9udiDk"
   },
   "source": [
    "# Module 7 Material\n",
    "\n",
    "* Part 7.1: Introduction to GANS for Image and Data Generation [[Video]](https://www.youtube.com/watch?v=0QnCH6tlZgc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_07_1_gan_intro.ipynb)\n",
    "* **Part 7.2: Implementing a GAN in Keras** [[Video]](https://www.youtube.com/watch?v=T-MCludVNn4&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_07_2_Keras_gan.ipynb)\n",
    "* Part 7.3: Face Generation with StyleGAN and Python [[Video]](https://www.youtube.com/watch?v=s1UQPK2KoBY&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_07_3_style_gan.ipynb)\n",
    "* Part 7.4: GANS for Semi-Supervised Learning in Keras [[Video]](https://www.youtube.com/watch?v=ZPewmEu7644&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_07_4_gan_semi_supervised.ipynb)\n",
    "* Part 7.5: An Overview of GAN Research [[Video]](https://www.youtube.com/watch?v=cvCvZKvlvq4&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_07_5_gan_research.ipynb)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zgMWBuf61OmL"
   },
   "outputs": [],
   "source": [
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return \"{}:{:>02}:{:>05.2f}\".format(h, m, s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1yqlUD4sdiDk"
   },
   "source": [
    "# Part 7.2: Implementing DCGANs in Keras\n",
    "\n",
    "Paper that described the type of DCGAN that we will create in this module. [[Cite:radford2015unsupervised]](https://arxiv.org/abs/1511.06434) This paper implements a DCGAN as follows:\n",
    "\n",
    "* No pre-processing was applied to training images besides scaling to the range of the tanh activation function [-1, 1]. \n",
    "* All models were trained with mini-batch stochastic gradient descent (SGD) with a mini-batch size of 128. \n",
    "* All weights were initialized from a zero-centered Normal distribution with standard deviation 0.02. \n",
    "* In the LeakyReLU, the slope of the leak was set to 0.2 in all models.\n",
    "* we used the Adam optimizer(Kingma & Ba, 2014) with tuned hyperparameters. We found the suggested learning rate of 0.001, to be too high, using 0.0002 instead. \n",
    "* Additionally, we found leaving the momentum term $\\beta{1}$ at the suggested value of 0.9 resulted in training oscillation and instability while reducing it to 0.5 helped stabilize training.\n",
    "\n",
    "The paper also provides the following architecture guidelines for stable Deep Convolutional GANs:\n",
    "\n",
    "* Replace any pooling layers with strided convolutions (discriminator) and fractional-strided convolutions (generator).\n",
    "* Use batchnorm in both the generator and the discriminator.\n",
    "* Remove fully connected hidden layers for deeper architectures.\n",
    "* Use ReLU activation in generator for all layers except for the output, which uses Tanh.\n",
    "* Use LeakyReLU activation in the discriminator for all layers.\n",
    "\n",
    "While creating the material for this module I used a number of Internet resources, some of the most helpful were:\n",
    "\n",
    "* [Deep Convolutional Generative Adversarial Network (TensorFlow 2.0 example code)](https://www.tensorflow.org/tutorials/generative/dcgan)\n",
    "* [Keep Calm and train a GAN. Pitfalls and Tips on training Generative Adversarial Networks](https://medium.com/@utk.is.here/keep-calm-and-train-a-gan-pitfalls-and-tips-on-training-generative-adversarial-networks-edd529764aa9)\n",
    "* [Collection of Keras implementations of Generative Adversarial Networks GANs](https://github.com/eriklindernoren/Keras-GAN)\n",
    "* [dcgan-facegenerator](https://github.com/platonovsimeon/dcgan-facegenerator), [Semi-Paywalled Article by GitHub Author](https://medium.com/datadriveninvestor/generating-human-faces-with-keras-3ccd54c17f16)\n",
    "\n",
    "The program created next will generate faces similar to these.  While these faces are not perfect, they demonstrate how we can construct and train a GAN on or own.  Later we will see how to import very advanced weights from nVidia to produce high resolution, realistic looking faces. Figure 7.GAN-GRID shows images from GAN training.\n",
    "\n",
    "**Figure 7.GAN-GRID: GAN Neural Network Training**\n",
    "![GAN](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/gan-3.png \"GAN Images\")\n",
    "\n",
    "As discussed in the previous module, the GAN is made up of two different neural networks: the discriminator and the generator.  The generator generates the images, while the discriminator detects if a face is real or was generated.  These two neural networks work as shown in Figure 7.GAN-EVAL:\n",
    "\n",
    "**Figure 7.GAN-EVAL: Evaluating GANs**\n",
    "![GAN](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/gan_fig_1.png \"GAN\")\n",
    "\n",
    "The discriminator accepts an image as its input and produces number that is the probability of the input image being real.  The generator accepts a random seed vector and generates an image from that random vector seed. An unlimited number of new images can be created by providing additional seeds."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SpCjlQyEdiDo"
   },
   "source": [
    "I suggest running this code with a GPU, it will be very slow on a CPU alone.  The following code mounts your Google drive for use with Google CoLab.  If you are not using CoLab, the following code will not work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 156
    },
    "id": "Y8_-1h5ddiDp",
    "outputId": "c623348e-940c-48b5-be88-c5998b0e8314"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
      "\n",
      "Enter your authorization code:\n",
      "··········\n",
      "Mounted at /content/drive\n",
      "Note: using Google CoLab\n",
      "TensorFlow 2.x selected.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "try:\n",
    "    from google.colab import drive\n",
    "    drive.mount('/content/drive', force_remount=True)\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "    %tensorflow_version 2.x\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BeVAWGHOdiDl"
   },
   "source": [
    "The following packages will be used to implement a basic GAN system in Python/Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KubxTY1mdiDm"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Input, Reshape, Dropout, Dense \n",
    "from tensorflow.keras.layers import Flatten, BatchNormalization\n",
    "from tensorflow.keras.layers import Activation, ZeroPadding2D\n",
    "from tensorflow.keras.layers import LeakyReLU\n",
    "from tensorflow.keras.layers import UpSampling2D, Conv2D\n",
    "from tensorflow.keras.models import Sequential, Model, load_model\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "import numpy as np\n",
    "from PIL import Image\n",
    "from tqdm import tqdm\n",
    "import os \n",
    "import time\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X9uwIRYJdiDr"
   },
   "source": [
    "These are the constants that define how the GANs will be created for this example.  The higher the resolution, the more memory that will be needed.  Higher resolution will also result in longer run times.  For Google CoLab (with GPU) 128x128 resolution is as high as can be used (due to memory).  Note that the resolution is specified as a multiple of 32.  So **GENERATE_RES** of 1 is 32, 2 is 64, etc.\n",
    "\n",
    "To run this you will need training data.  The training data can be any collection of images.  I have used various sources of data for this example over the years.  Data sources sometimes become unavailable for copyright reasons. I have one sample source listed below.  Simply unzip and combine to a common directory.  This directory should be uploaded to Google Drive (if you are using CoLab). The constant **DATA_PATH** defines where these images are stored.\n",
    "\n",
    "One sample dataset of faces can be found here:\n",
    "\n",
    "* [Kaggle Faces Data New](https://www.kaggle.com/gasgallo/faces-data-new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "tb_XblE7diDr",
    "outputId": "78debcc4-f20c-4578-ebe6-91b40d8b6f3f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Will generate 96px square images.\n"
     ]
    }
   ],
   "source": [
    "# Generation resolution - Must be square \n",
    "# Training data is also scaled to this.\n",
    "# Note GENERATE_RES 4 or higher  \n",
    "# will blow Google CoLab's memory and have not\n",
    "# been tested extensivly.\n",
    "GENERATE_RES = 3 # Generation resolution factor \n",
    "# (1=32, 2=64, 3=96, 4=128, etc.)\n",
    "GENERATE_SQUARE = 32 * GENERATE_RES # rows/cols (should be square)\n",
    "IMAGE_CHANNELS = 3\n",
    "\n",
    "# Preview image \n",
    "PREVIEW_ROWS = 4\n",
    "PREVIEW_COLS = 7\n",
    "PREVIEW_MARGIN = 16\n",
    "\n",
    "# Size vector to generate images from\n",
    "SEED_SIZE = 100\n",
    "\n",
    "# Configuration\n",
    "DATA_PATH = '/content/drive/My Drive/projects/faces'\n",
    "EPOCHS = 50\n",
    "BATCH_SIZE = 32\n",
    "BUFFER_SIZE = 60000\n",
    "\n",
    "print(f\"Will generate {GENERATE_SQUARE}px square images.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oDTfFQjTdiDu"
   },
   "source": [
    "Next we will load and preprocess the images.  This can take awhile.  Google CoLab took around an hour to process.  Because of this we store the processed file as a binary.  This way we can simply reload the processed training data and quickly use it.  It is most efficient to only perform this operation once.  The dimensions of the image are encoded into the filename of the binary file because we need to regenerate it if these change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "id": "dJ69ALfSdiDv",
    "outputId": "59daec8c-7d81-4764-ff40-b9c4931011eb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking for file: /content/drive/My Drive/projects/faces/training_data_96_96.npy\n",
      "Loading previous training pickle...\n"
     ]
    }
   ],
   "source": [
    "# Image set has 11,682 images.  Can take over an hour \n",
    "# for initial preprocessing.\n",
    "# Because of this time needed, save a Numpy preprocessed file.\n",
    "# Note, that file is large enough to cause problems for \n",
    "# sume verisons of Pickle,\n",
    "# so Numpy binary files are used.\n",
    "training_binary_path = os.path.join(DATA_PATH,\n",
    "        f'training_data_{GENERATE_SQUARE}_{GENERATE_SQUARE}.npy')\n",
    "\n",
    "print(f\"Looking for file: {training_binary_path}\")\n",
    "\n",
    "if not os.path.isfile(training_binary_path):\n",
    "  start = time.time()\n",
    "  print(\"Loading training images...\")\n",
    "\n",
    "  training_data = []\n",
    "  faces_path = os.path.join(DATA_PATH,'face_images')\n",
    "  for filename in tqdm(os.listdir(faces_path)):\n",
    "      path = os.path.join(faces_path,filename)\n",
    "      image = Image.open(path).resize((GENERATE_SQUARE,\n",
    "            GENERATE_SQUARE),Image.ANTIALIAS)\n",
    "      training_data.append(np.asarray(image))\n",
    "  training_data = np.reshape(training_data,(-1,GENERATE_SQUARE,\n",
    "            GENERATE_SQUARE,IMAGE_CHANNELS))\n",
    "  training_data = training_data.astype(np.float32)\n",
    "  training_data = training_data / 127.5 - 1.\n",
    "\n",
    "\n",
    "  print(\"Saving training image binary...\")\n",
    "  np.save(training_binary_path,training_data)\n",
    "  elapsed = time.time()-start\n",
    "  print (f'Image preprocess time: {hms_string(elapsed)}')\n",
    "else:\n",
    "  print(\"Loading previous training pickle...\")\n",
    "  training_data = np.load(training_binary_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y9kO_iSRyixQ"
   },
   "source": [
    "We will use a TensorFlow **Dataset** object to actually hold the images.  This allows the data to be quickly shuffled int divided into the appropriate batch sizes for training.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BXl0JohJBx69"
   },
   "outputs": [],
   "source": [
    "# Batch and shuffle the data\n",
    "train_dataset = tf.data.Dataset.from_tensor_slices(training_data) \\\n",
    "    .shuffle(BUFFER_SIZE).batch(BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2dATyXqQdiDw"
   },
   "source": [
    "The code below creates the generator and discriminator."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zB_aX4ChdiD0"
   },
   "source": [
    "Next we actually build the discriminator and the generator.  Both will be trained with the Adam optimizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Ulou-BZPybzT"
   },
   "outputs": [],
   "source": [
    "def build_generator(seed_size, channels):\n",
    "    model = Sequential()\n",
    "\n",
    "    model.add(Dense(4*4*256,activation=\"relu\",input_dim=seed_size))\n",
    "    model.add(Reshape((4,4,256)))\n",
    "\n",
    "    model.add(UpSampling2D())\n",
    "    model.add(Conv2D(256,kernel_size=3,padding=\"same\"))\n",
    "    model.add(BatchNormalization(momentum=0.8))\n",
    "    model.add(Activation(\"relu\"))\n",
    "\n",
    "    model.add(UpSampling2D())\n",
    "    model.add(Conv2D(256,kernel_size=3,padding=\"same\"))\n",
    "    model.add(BatchNormalization(momentum=0.8))\n",
    "    model.add(Activation(\"relu\"))\n",
    "   \n",
    "    # Output resolution, additional upsampling\n",
    "    model.add(UpSampling2D())\n",
    "    model.add(Conv2D(128,kernel_size=3,padding=\"same\"))\n",
    "    model.add(BatchNormalization(momentum=0.8))\n",
    "    model.add(Activation(\"relu\"))\n",
    "\n",
    "    if GENERATE_RES>1:\n",
    "      model.add(UpSampling2D(size=(GENERATE_RES,GENERATE_RES)))\n",
    "      model.add(Conv2D(128,kernel_size=3,padding=\"same\"))\n",
    "      model.add(BatchNormalization(momentum=0.8))\n",
    "      model.add(Activation(\"relu\"))\n",
    "\n",
    "    # Final CNN layer\n",
    "    model.add(Conv2D(channels,kernel_size=3,padding=\"same\"))\n",
    "    model.add(Activation(\"tanh\"))\n",
    "\n",
    "    return model\n",
    "\n",
    "\n",
    "def build_discriminator(image_shape):\n",
    "    model = Sequential()\n",
    "\n",
    "    model.add(Conv2D(32, kernel_size=3, strides=2, input_shape=image_shape, \n",
    "                     padding=\"same\"))\n",
    "    model.add(LeakyReLU(alpha=0.2))\n",
    "\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Conv2D(64, kernel_size=3, strides=2, padding=\"same\"))\n",
    "    model.add(ZeroPadding2D(padding=((0,1),(0,1))))\n",
    "    model.add(BatchNormalization(momentum=0.8))\n",
    "    model.add(LeakyReLU(alpha=0.2))\n",
    "\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Conv2D(128, kernel_size=3, strides=2, padding=\"same\"))\n",
    "    model.add(BatchNormalization(momentum=0.8))\n",
    "    model.add(LeakyReLU(alpha=0.2))\n",
    "\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Conv2D(256, kernel_size=3, strides=1, padding=\"same\"))\n",
    "    model.add(BatchNormalization(momentum=0.8))\n",
    "    model.add(LeakyReLU(alpha=0.2))\n",
    "\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Conv2D(512, kernel_size=3, strides=1, padding=\"same\"))\n",
    "    model.add(BatchNormalization(momentum=0.8))\n",
    "    model.add(LeakyReLU(alpha=0.2))\n",
    "\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2kxKk7uDOnPS"
   },
   "source": [
    "As we progress through training images will be produced to show the progress.  These images will contain a number of rendered faces that show how good the generator has become.  These faces will be "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UKnCeDut2cp0"
   },
   "outputs": [],
   "source": [
    "def save_images(cnt,noise):\n",
    "  image_array = np.full(( \n",
    "      PREVIEW_MARGIN + (PREVIEW_ROWS * (GENERATE_SQUARE+PREVIEW_MARGIN)), \n",
    "      PREVIEW_MARGIN + (PREVIEW_COLS * (GENERATE_SQUARE+PREVIEW_MARGIN)), IMAGE_CHANNELS), \n",
    "      255, dtype=np.uint8)\n",
    "  \n",
    "  generated_images = generator.predict(noise)\n",
    "\n",
    "  generated_images = 0.5 * generated_images + 0.5\n",
    "\n",
    "  image_count = 0\n",
    "  for row in range(PREVIEW_ROWS):\n",
    "      for col in range(PREVIEW_COLS):\n",
    "        r = row * (GENERATE_SQUARE+16) + PREVIEW_MARGIN\n",
    "        c = col * (GENERATE_SQUARE+16) + PREVIEW_MARGIN\n",
    "        image_array[r:r+GENERATE_SQUARE,c:c+GENERATE_SQUARE] \\\n",
    "            = generated_images[image_count] * 255\n",
    "        image_count += 1\n",
    "\n",
    "          \n",
    "  output_path = os.path.join(DATA_PATH,'output')\n",
    "  if not os.path.exists(output_path):\n",
    "    os.makedirs(output_path)\n",
    "  \n",
    "  filename = os.path.join(output_path,f\"train-{cnt}.png\")\n",
    "  im = Image.fromarray(image_array)\n",
    "  im.save(filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XiUbj3W4Oo3U"
   },
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 285
    },
    "id": "gL5byGhNzOzd",
    "outputId": "ad1df425-b190-4404-b716-5998f6ea48fb"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7fd92007ad68>"
      ]
     },
     "execution_count": 9,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD7CAYAAACscuKmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOy9WbAtaXYetP7M3PN0xjvUvbeG7i6p\nB0lGUkuARThkCwIbDPIDoTAQCtkI+gkwGALJPPFChB1BIPvJQYcNISJMCGMcIUV4AIVCMmZqVBq6\nS9Wtrq6u4d66w5n3PnvOkYf1ffmv3PvUrWpX65Stk+tln7Nz+vPP3P+avvUtVxSF1FJLLX/4Jfik\nB1BLLbVcj9Q/9lpquSFS/9hrqeWGSP1jr6WWGyL1j72WWm6I1D/2Wmq5IfKxfuzOuT/pnPumc+4t\n59zPf7cGVUsttXz3xf2T5tmdc6GIvCki/4qIvC8ivyki/3ZRFF//7g2vllpq+W5J9DGO/VEReaso\nirdFRJxzvyQiPykiH/hjj7q9ojHak7xhvnRYbAqnn4H+71Lnd8mwC+yQIpTKviIiLtS/uXY5f3j5\nneT4kv8XZqfwAxa93PyN/V169a66kfvio2HGmLnKrkX136qUNtcV4+I2js2cyOUfcG5zmq1NPJ8z\nO131HYXzyA88hyK74oaCjed7lWy+AxvjrQzafs93hY/1queaVcfqT3DFta4ax+Z3W+cxX+bVTS7f\n3uXK54O55nteXuGKfcqr8jdgh1GIpOfnks3nV072x/mx3xORR+b/90Xkn9/cyTn3JRH5kohINNyV\nl3/2L8ryjr+rookHtta7KTq6rXHuh9a80LEnA903GeqMFT1/ntZgLSIiWRriun4W8lzPna0wQ4n+\n7xI/J0Ufv2D+oPkC8xgRCVZ6XOt0w/ux7w1254KwvO9XhuaZbuSPK+ctBts/qLyN6/M+zCVdW++7\nWOJezcIYzXGvmNeA2+yPfeNyaQ/zaRYmaek1gkYum5IvdeCuqftE+EwWZhXHqcKu3n8e497tOLi+\n43juI2IWDi4sEcYR+4kIenruqKHH89nniZmsS0wyTl1gUbcLL9+DvKnXsPNZRBv780dnJtFhjMES\n7xVey2hpFw0MY4XbMr+8tKvnakzx7hVX7NPD+4DrZ73tsbpU5PEv/FX5IPk4P/aPJEVRfFlEviwi\n0rn7oAjS6gDLVY2r4ZwvsN+F2/gjj24tRUQkz/zLES+a+scEL1zkH0Y41Yu0V3rdaI5LJ/4a8Q6u\ni++4sNh9ooUeP3pbx5F0txfQHMPwq7qZYv4A1tgXQ83a25pkvYMJWG9MkPgFKJr6+6c0Jnquduwq\n18qaZhg8rHwB9Yv1vlmEZeOHk/sxhjMsKH3MZy/WDT0/xlF/tTU2keoi/OLwQkREupEenxf+RzqO\nO/hOr3G66ImIyDL2C0qvFVe+u5w2t8bqqB4xncFa/29O/LWKUrPix2rWt2SI9wC3yLlLh2YnLARb\nC4N5rHmjarHyBy4ikuxjQWxizrnQt8w1aHmuuaDoeVr3Z+Uu8bt9eZ58nADdYxF5YP6/j+9qqaWW\nfwrl4/zYf1NEXnXOveKca4rInxWRX/nuDKuWWmr5bss/sRlfFEXqnPsPReR/E/WI/vuiKN543jEu\nE4lmVX8pj+h/qylDXzk98GZOetLSfQa6T6ettnXg4q1rzFu6b7Pl/YB5o6t/jPV2aaI3pn6tS/o6\nju5TfBc47GPueXNpvCJoxH3iAf3AYmsbD4x39Zr5obkPmJ0P7p5XLpXk/uIrmK3jhpptzgQXY/gR\nAW4/mnMc/lx0TehGlIEg600wLgAzPuh7fyZPmpV99voLERGZLNvlPp2G7j9sqTn/raNDERHpd715\n/3g2EhGRVqjnudO7LLctUxvFFenifOvEv7J7Hb3uuqnfhYHO5/ii5w+kG0RvBM8jsT47rOUM5nPj\ncjt4l+BZNS70ROHcP4+si+fY4nPQbTaOG8BDou9t3QCHOFUGN6Az0jlqNfwZAtzb+VOdM5r4/c66\n3Odk0PngQLN8TJ+9KIq/LyJ//+Oco5Zaarke+QMP0FkJEpH+00zSnolwv69DSHv6WQZLbGwCyiCD\nhp8d6qpfmPhUGeDDCroyC1wTQZnWOYIjCIg1vSKRCEGn3lOs0tF2FLs51229R6pRJNf/g6XXeus7\nAx1rO6h82uvypDNElvNzrxGbE/189uCOWEkOTKQQqjiEpSL3luUmahlB5iDDfNhAIzUYg0UNxHiy\nlh9rnugcM5qf2yg45jPrIHh2qZp0fdnyQ8QYVyki93hA58fDcp+oo5ornail8LC75wc51u8KWA8M\ngkUz/9C/cagWW0DrkAG2sbcKWieYh05R+WSwVcS8O/jM/OOQbFDNh/nzmBe0rX+HyGBkTu85b/lr\nBJi/MohnMkFyXrXGVnOdz9XIP7RmB39vxIRv9XyAbrbfEhdtZ0/KMXzgllpqqeUPlVyrZndFIeE6\nlyL0q3MOZcBVLR5UV2ARkeZY16TVIVbQQ1X19w7G5T6XK69VRERCk7ueLmARHMF3h++/NH5bb1e1\n47NnuqoG8PXax349jJCimt9WX5npGJu3TjfScanREuFGiCHDkK0VQz+aKTSfc/WPKoBW6BwBfzDp\n+m0p98c16dKZYVGrFdiHMQwbX+CYaFUVDRNnwX6NnmqbVhMauuufa6ep237w8H0REXkjvCsiIseZ\nn8/DXQ2IPD1Vf75Y+nukJRAN9DzppQk6UNZIVfF/aM/otrd04kTnJm9XsQT2XoO4asVlO95XjoAT\noKUiSMW5eNusbGAe8jXGalQpraECz66CadgYR4j8fLb273T6Aq+Le7zUz99veQswW0RSpB+sv2vN\nXkstN0SuVbMXTiRvuAoskL4k/W9qNgup5d9c+bjKn3c75T7zqarQgAgpo26J7Arhx+bwqSITUV1O\n9LsmotcBtH7nxMJdq/fTnsC/N4s8UVO9Z6rG1zv+Rkq4bxmX4E3748OkinwLEiLpTPS3UwV6hFfg\nVxob36V+qoSnSvrQcpzzthkI0GRZDxbOvteWHYBZPrV7JiIiL3Q00HC69qCOzw+eiojInxp8TURE\nXh/eFxGR/2v31XKfL/QVlvE/xgq8LAw+lBmHBiLSRUevOU8GfozwkT3KTu8nNH5rikwHswoNIC0t\nICtbEikDrdn1mn3Q1/tmhD+ERi4W/vhoB6CgfANcY5B84Wpjm0HX8ba7z6DRodDtb2DR1C9b53pO\nxpvilX+wDbcRC9iQWrPXUssNkfrHXkstN0SuOUAnEqRFJWUWI7bEdBDBB5kBB7BQg2kQBm3ujybl\nPo9gC61Xavuka3Nr6+qaRkBCsTJ4bwRQGjDjGaCyKavOGYAmIY8pKv+L+JRdEKNQITduAINtqACh\n+VaY2CLTWTThCEBa73vTNAMAqAj0wgywiZhiig28ti2qKFNMAIHQxAzMPLEohEU6o5434+kiDRpq\nEg8j9RlSMxEr3MCzTFNt760PRETk3ZlPr+00NIUZp1VQjIhIvEJKNtFzpgjeRQuTykSqMGA9BbOO\n/lbFzfS4NsAw6wNWkvhn34BJXRbNzX1UdXIL84n0WIp3pzExbsBa90939GXh3AcGVUNXK1FMTFnc\nJeLf+dLNw7NKTa1B0aymhAna4u9GBIG9D8bU1Jq9llpuilxzgM5J1gqqsNONcr5yNazAGbG6Iu3Q\nbOkK+nLfQ0pZObVIdQXuNzyMcLLWIMbD810RERkiwDQfepWawBJIdvST5awus6k3BPgIMx3SCthe\nTjMEVJKuP749QSloBwCLjXJYEZEMlo6vxNLPMj0kInmptbGPLRvNq59CrWEyVykspOEdTX2luMdP\n7fpU5qcGGnw7XmnQ7YdGtppZ5Yvdt0VEZD/UMsJvxbfLbT/U1pTb9zQ0sLUfvC4iImeJh7Lea2nV\n28t7VWiwiMiTQC2CbjOpfh/uln8PdtQyWMGaSwDqaXT9MfxrDSso2EUwzVReJi1MJME1ZlsD5bdx\nFxPJSlfzPPj8EsB2mWazwJ32Oaso9bvu0fY701joQ1vtlWQC5bZ4j1WMAIjhUS0P/T7NyXYQ2Uqt\n2Wup5YbItWp2caqxLAMHiwcIMgjgPw3eNesQ+Rvgv83muur/4+BT5S6zM6hEEgmY1Aj9Vq64q4H6\nWC7evkb3CYAaLI02MzS/C78PyoU1yYHRBCxwiFBkk+wZgAYIOXivGcg3WB+u90ifH2nGHmuk/Tg4\nRzn9cZMyC07g/yItx1r5fH97yY9QgMKa8cj4zNToLFYJjPlwp615n15frac2BtcNvDVFuchU+57l\n6qt/c3J7ax9e/8KkkRYrfQAER7GwpjPwHnkbablWpNcfo1hoZ7go9yng265QQLPb1diDLSwaz/Td\niWHdFefeDIpRI8+4AK0x648XiMGwOIb+eOTDHGV8J4JV2B7755E1q3GiaIlinb6xHmhJbGTWbNrV\n5VubK1Jr9lpquSFyrZo9a4hM74eyMrRU3XsK5B/Bjz55siMiIuHSgFHo27JMENqy3/aapHNXV34C\nMy5NRDWDT5qhuKIx1ONsxL7IySCyURJpZihDYQMZc3JEX/OZgbICYJEA3vryK8flNhZ6tAEQidd6\nj/27fnmejFXLFAucE7RQztBjFfQfYZmEQ4/DJeMOswq0Pio0SrBEhpi/TqTP4G7HVwYNoZYImPl8\n90m5bVXouNtwVr+VKNz1Ny4/W+4z7r0rIiJ3Ij3+JFVr7GTuffZRS79jfGWn6VXhjxw+FBGR8xhF\nIZnOhy2j5bPOCGZhRsaUwTIekYKyahbqO3A581ZEDjBNMYfPbbn0AMllxiKjNWegzyycoUbvPoM1\nZjFKKKKKcJ72iX/myVDHlDUB/JmXI/PXwLbmpHruzolU5YPrYGrNXkstN0XqH3sttdwQud569kxT\nEHnDm6TzQgNBi6GahO1HMBHPvdm52ieAAOdB8O1s7LHYCQMpHVYemZp5Yqi5tMHcK2uERSRG+qbY\nqDqzwIaS4ZOAH6YErdnnGGjU67/3dN9vw2H3d9S0PZ7p+NsmvTQnU2tSXYcLQ6AZtYEXh1vR63p3\nZh4A4AErlUSHYoJ4PNNhR833g5bajbPUB6aWiFAuM52Qb69uldtOYx33Raom9ihU8/v3TfBtnmoa\n7DNddWPOse9y7a/xrTM1/xmgs6CaF4btyjamWd8O/Hwy45mRPXiqY10bhpckZj29/k/zPTNVdGWq\nDcHA0LLClqkujPEpzPBzg79HELSBwFoLwbciMKCtFd9BBO/e8+5d2FfXLbkDFpqIGHmTLyX8n9wO\nW8SmGMdzInS1Zq+llhsi1wuqCUXioZP1roEBIv00BEBieoCV2JSSlVxtxD6Ar26n7wM6pIoj99zS\nmZWbWQtUM7GO+l7fwG2nGhgcd3SVXZyoBnB9ryU6PVRMQZNESPnMAh/s6SP4loEv/vbIk9jNUJ/8\n6eGpiHhNZgNj/89cz5UEqqWaoGkOQ7+Es+qMASkLG359AF66pMrD3+p77d8Db1kPQKQv9LT6rO28\nhcEgXILoqKV5prZ9paXRoRjoHgtkut9WwMzdhn42EF29s+PvlRqZbDaN0Adu5+C5W4Ixh5x0hNaK\niMSoKc+Qkm0/1W3xygcBGVhLB5g/fLQuTBUhXhVWE1ptySqyTShrYDKZjH1Sw0eA8a5H/hrNafU8\n7b7nIJAm7m1EdiA9ISGxIt6qLSHWDCCbyrgwKWq4bC211HLdoJpcJFwVJXBExDeFiIfVGvNoYQ4D\n6IB+aIE0zLjtV3B2EGF3kMpl4ZORuZYAC8JoRUSmSOkQWEE/znaEWea6TwTrIaOvbthBFvNWZTyP\nk51yW4J04u+E93T8AHOcdHzsYTnWazROsdrvbK/H655u6wFgkluUEju4LMPKMY0r5iVy1TzNEas0\nxGv0NcyprqHZOVpryuxWU/fPoGbOjEY9aWvdOTX6k7XOw5Nzcw3MNTkIIuNr39mZVu6NVkAz8vsw\npZpc6HzwnamwBRGQxWIXaua1KUSBQUOgTGNmC6T4XVVlVjpV4e+QrL38VVkGI4KbyPgTmeeTovtO\nAk7DdLtBCfFKJRNRszo+Pbc8V2rNXkstN0TqH3sttdwQuXZsfNZ0FbI/NtzrAs11fgAz9tLbJKXZ\nzwAKTOTMEhSiqd6SFEUW984OrzC3pw29Rs+kvEgplAO51hqz/tkEE7E2FnArCuKrr2jMGMKcHg18\nEJFhtFeGmkb6FqiRek1vIp+TjrhNe5OlWObcoF0iNnxoOKh29jSNNm3pPQ56uu1W31MOv4yKNqbF\nGJjbaZ2W+3xj+YKIiCxgL14aXiv2YdtF07xJhv9b3veaI+o1A8cS3QEbaMx4H2z6YVJvKeb2fALX\ngOZ8fIV+KivRtjfxXStpz5CCTHvGlYSHw5Rb74kl3twwqXFY+9w0cIBbGS1xblQ10hwXEYnAbb5u\nbufGXMw6eNZaICBtzHIiIbPWxrE2UBjU2PhaaqlFPgnCyWYVbx4gNTIF2MGtNvHBPjXSYkAMC+bK\ngHNKummAT2wzgXSEVRg5khQAGhvYWs2bles3AaqJd/0+rFZjyitE6i0zDC1lms9tE1+WVMNYf3c6\nqvX/ud33y326DVVP7w80kBVecZ4Xhpq+aiJaM0v9cv/pPdXObxYKWNlBlZdNa/1gX3Hnh5EGwR40\nVNMvcn+eabNd2edR4hlmXkAx9X6o1sJxogG7dmiCZ5ijN2cKxnl4qaWCq4VPibYAahphjP2GV820\nPliJx/E3TM5rD5bEZaJj/b30JRER6dz1VszhQF+kNayoPqyot77tKZjDS7DhgICzeWnSci2Caqpp\nOZTiY/8M2wDKWetO8dC/F6x6I/DGrfy9Fm2w4KB5ChmUsooVAM3eZl07ADxT03k3cBVuhE2pNXst\ntdwQuV6fHdIcm/pvuGRMS3FLPDQH4MsEcNVKk3tIvtEbu0KvjHwL/S6CJ544nxYLTlE/DXYRrpCR\nqTUvU20A5zShqZv7BniDOMBBVzXKIvGaLEHlVYBVuhNh34bXRASm0B9nmjBLvZZg3fftrmrd9yc+\nncU68PWZ7rMA7LTb8xPyP61/VEREXh6q9vxX97Qf5xOTJnxroRq5BetharpdHC01rfa7DaWH/p4+\nILFrDxQ5aOv9jxBP+BduvysiIr8d+i7f9wdqIew1VUO/v/DX7yCP9b07RyIictjUOXq88vvQeths\nArmcWQtFz2PBOCJSdW5d9TMyvcP4aLon4B4A7LX91D+zIgJct4d3aKbPMJr6cURjvcfklqZZs0e+\nu3nwyosiItKcoLkpfPbG3MStyE7EMBHThYmJFxWFuOKDUTW1Zq+llhsi16vZA/XZk6EpLgEfWnek\nftsiVe2w3vNLL7m8yL9V8rIZdlgyuzQmVZ9bxMcICDksOeQMGKZsu9sg+AF10LvGCcIp+yg8IcAj\n2DY0JMnDre+oiajJYuzTMCHVGJHdJnxURupnphVQG9fdh8963PSNEy6n0K5gQT3cU/9+1PKa/W5X\nv7vXVs26Aw65lQn/7jT0eRygZ/XabJujYIaMNe2A0Xyfedhr6jnHSadyj6lhiJklrcp8kEdQROTT\nbYXifnulsYcREDNvZYflPmUBTdlzeftBnD9RqydCE8wZ3rfWqX8+rTP4yEg4tM/986BvzGdPzR4f\negARG2KWDSJTMvOaiP8aEXewDhepea+apBLGYSlr302LqnX13shiszYFX41FXgH7bEqt2Wup5YZI\n/WOvpZYbItceoCtctbc1+2T98AtKVfx6pGCOyXJn61jWZhO7nHpIeVnnm7IBg7WiaYKtq1/klwaU\nk1ZdhNIkMyAO4vZnBfp+oVmFJWMkXnveUVPX1qqTGJFyp60mcsuAoJtIX7Hn+YN9NbUX623gM68R\nm75lJUUyKatYX2/GuIar8HCp6bA9zPk35nfLfZ4uhzhOU1TW/H7nVGvKnw11n1dHanKfLP0DIaX3\ns5m6GLOBmuzL2N/HGhTOb0/1fJmprPvGQsfCtBrN+lHDuwqstZ851COgws9ZU7bg+4D7RyDX9r5j\nqT88Fglj29+vmk5rHOtOLjUpr5AlcQgSz3WM6QPvcgiAM9FEt+Utg47Bcc3HCrtyU3WBup0Xyl0u\nXzbtgMUH5tg0QgQuqPtgO77W7LXUckPk+jV7UCXhZ5VaC2AJQibZoklEJEGgo3kO8j9WIpkgXGOK\ntNz8irQclAkDdIznWPaXHNfzrCWyPY4OcZV6HGGeSWKCPaA8jgALTYzWXS1V232toSt2i4G2pk/j\nrKCtYqQix0tVQcvpNoHme5Fq5pMjn3orCQejakVbO0xkUxqYCAbmWoYfmZbADoJnb08Oto9HELEF\na8SClPqRPqRlrGCcc1TEzRdeoy0BkeZTaJuKNgY4T3Hc31t9n4hUtf/RVC2JBOnJcIr6dvN+8XmW\nwd0LVlfaqjdoe1TLxX3TfTUmrTO/U4sziL1mL63ADFVrbb3HvLP982KaLuj6NGW606nsE+WA3Zp+\n94TJ+lhuFVwjIpK2XB2gq6WWWq5bsxdaj9uY+jVmdaEa6/We+mjsy+1Mj+wC6YUAwJKSpcNQdJGo\nnwT7tsUUIY8llJCZDltXn1X3YfGNqWUomwCkANHMJ8zlmfNAo4ZDDSKwKYGIyByacKet31Frvr/0\nLY2ogUltTZhs1PaaeQiADNNpRy2jZR6Bg26EFBEsiwsDeLEFKyK+JdOx6a9OiZG3zIzKYDELz/Nw\nruN/+MxDak/7es7ZhV53Casmm3mf/QytoZoAEC0Dv+2ipdruyaXGBWgV2cKi2UT3KXuvJwRUmRgG\nmzpAs5NSu+kJhMp4D7J8vuZcpIz3lDTm0LZB079gnJrGFDGccFu9Fh3EFdAiqph5PHi4wDOnvz3W\nlGbnsdf47Vd2MDb9v4TvZiY9l1ZbgW3Kh2p259wD59yvO+e+7px7wzn3F/D9nnPuV51z38Ln7oed\nq5Zaavnk5KNo9lRE/rOiKH7bOTcQkd9yzv2qiPw5Efm1oij+snPu50Xk50Xk5553oiIQSbu+5ZOI\nj3YTXrrXUW2xHJomEVg634s0arvE/25pCw3Ytkm3JR5nUrYmpp9WWgZ9rxHbuytcS/9fw+Jwbb9P\nXlJ6IsINLZ7HhskWpa3U6Httr0Uboa7uLGD5dFeLVtamMigKVNvdu6uVFgcdUxHEscK3vdfV6O3b\nDc+4GqO9NRtT0p8/a/r7uBxBy3Bem7pOTxZekzB2wmj+5dQ0VcBcn/bBd4d9cgPvzFDk4liQBOvD\ntuUSqIcVMg2EH4uInC6r/jhbN9usBNsoF222UUbRyoXtdKkfIUtMUdjUPfIxjeYULcFbADktTLYI\nfyc9QGJRiGItHVqOKfZp9HTM8zt+rN0uS5b1o/fAR9rXu/quBWtActv6f2Ggr7RQfMNPlrxaK6T4\neD57URRPi6L4bfw9FZFviMg9EflJEflF7PaLIvJnPuxctdRSyycn31GAzjn3soj8oIh8RURuF0Xx\nFJueich2xz495kvOudecc69li20tVUsttVyPfOQAnXOuLyL/q4j8J0VRXDqTvC+KonDu6tBAURRf\nFpEvi4i07z0oSnADJFzoevPuuQZ3DgeahrJAEZINdoc0tWGOG6rdRYEAFKuDhj6gFTZRX0wcMay0\nl1/yjbK+uK813q+daQXSGD3gRx2PKX8CgkWH8fQQKJteehOXRJPTlZrKDMaJ+NQUg15kcbFVbzTp\n2dOM82Dr0VktR+ALGyGIiBSgvibuv9FV0FJkUnFk6Dmf65wRMMO0od2HYllkZgsdW7+5rhzfGHn2\nw6JkYcS9bzRisOdcwUWwlX0J5p202QwKdlseP79k6jQgwwv+tR1vsXsCZhoGdS1Ns0MqMx6Cqcb0\nTm8f6/PrrFgGqfvmbT/n6z19jsTIszmEDRLTvSxTZ4l3WRa3dFC9J0hFJgjYGb6Gst9hXv204LE8\ndB+/SYRzriH6Q/9bRVH8XXx95Jy7i+13ReT4g46vpZZaPnn5UM3uVIX/TRH5RlEU/63Z9Csi8jMi\n8pfx+csferVCq9MsbxYbQBCYwuouG4hZb0BFqT1Do60Kagyu9qZ9UsF0GNom7aJ/97948E65z4st\nre2e7+j1gx09z2e6R+U+/29P+8G/PdaA2K2eauR3Mn8tMqP8S7e+LSIiX+z5a3SdrtzTXDXZWaZB\nqFuRb5xwu6FBt6e31Ir44ug9ERE5SnyBPyvIWG12uvAVWAnGMkH1GxsoJJcezEKmuZitkfB/0fFa\nk9qbDSVYJy8iEk/0XOMRUl+8tum8m7KVEtKsBKxYxbME7bYgZZZl/nVcgVq8bNeElN284cFFDaRC\nwzO9R9891WtmYonIEJOggrIx9+9O91jvO2/oeNonfh7CxzpbBeCxxVyfbzjwacrOSgFHZWUbPqOl\nT0WWIBy2rDr2VmWYKLV4WS1X9rUy9wEDM99o+7Te8TPanDw/QPdRzPgfE5GfFpHXnXO/i+/+S9Ef\n+d92zv2siLwnIj/1Ec5VSy21fELyoT/2oij+T/lgT+AnvpOLuUIkjEUM6UnJDBPPdMk6hdYg9FFE\nJG9XGyoGPcBUp17bROARy5B6KtrbIQRaBFw4P932nscPt1WDPlxDazdV277aelbu80Z0r3I+FpRY\nxlT65ay/HgRL2ZQeGP/5eZz6POGbSy08ebpQTf6aKK/aymg9+vy81umFP77kxVtBay+q/cVFRJaC\nGAOtIMxLbO4jPkfMAOlCCxyKJnrOpx3EMBiHMTBVNmMo3xxYWuHKtERCO62sXe2hLiIS0g+HX58z\nddfwY0zANZBgbCHYhnPD3UaQFEEo5fljw5bUBlgLcSim4EREoh64EWdIobL2PDMp2abuzzhWCH65\naOYtBJegSGeJ7zr+R8A0GllmipWq8WDt4yasbQciV5ozVmz5+Uy7rhIn2JQaLltLLTdErr39U7D2\nCkXEr7A5YIQenmiMCfouKHEtNYnZp/zritLUnGoG22ZgTp1kHkLKopA+qmzoF4emhw+ZTQlqsUyn\n5T5h9Tg2SLTyKFbroQvNfpoOt/Zp4txkop0mXhNQo6/Bq8ZWxSIiGSK4jpDRDe6yijCBAj+/MNqb\noBzyrNtWRjyM4BrXpK9pnvlr03cAACAASURBVBm0FJmIyoduzsMsQunrG/jyGM0yE7TDYvGRbVC5\nBJiJ407ADmtIcksu+M0IuSUSysDzTsxUMjCsxV09mUP0PD89wzVNVuGe8vWV0fMVxpgZ+DGzV2j7\nVJhoPKP4yVAH0NlTaGyy699PZhNYmNO8QJZiZXjq4uLjwWVrqaWWPxxS/9hrqeWGyLW3f8ob1SYR\nFOLlWVlmLeR0R83Ezp4Gu3b7Giy5aHozZ4mUTBvAjkrQDGZeSqAJaujZtkhEZA5TvxGwNhv7GtZ9\nYtoJJqGpXZh8x6iJhgfIldj6a6bcpnnVjThNvLn37ZmmcUgXzT7lBOmIeLrqCXq5B0vjshB0EVdB\nHDZAl4QwJYlJwfGJ6WnfRv/ytL8dm2WNQdnUIKzSeIuI0HtxNO2jbXeAeHcG9pypVkuCVmVsBdo4\n2br+AJ1u2eKJxKSxqb1gXUTaYfBtY1ziQVIliaOpdehirvIdPKNnDMYZM36sKdiCPddxjJjqtzzA\ne002m9AEoCN+Yn+y4BhsPMEzJTY+Z72Hvdfn5N2k1uy11HJj5HrbPwVK12sb8BEcUGp0KNLMpM5Y\n3UYNOoOWYy9zEZFgpreyYl26qVZjX/bRSC2Cz+5ryu1PDb5W7vMyaIxP2kre/621psAszfMkqTKK\nUGzPcKbF2BDxUeIr0phqo1Dr75pm9Gx3FGxwx0UGLkua6dsjLcp+zwBmAqSoijEaFqyhGU1AirTb\nObjTWLufX7X04zFYam4CPAh1zgJYRRe2bRLbeOlxK2j/3hMLcW5Vzm1bF7GCjM0/4gNUtrVMYCvc\noGAmxbh5vzabKlC9NS9N4HWBwN4EDERjP9dBjOtRI5Nh5tADZhaf8n+LiDTBtze/Z6oIEWSOkDLr\nnPugbB7S6sCzWq8xZm+d8jUksjpYxDivB1SloXx8uGwttdTyz75cq2Z3ha6wNg2UDAAWeAEAizFW\n65EpxIBGH3RVpbBt0CPD1jqZoSYY6ik02zooaiFQg5xpx5kHo8So2PjqQkEsCVThnXC7Uo++Opsl\nWJ+dvjaPn6Q+rrAIdf9BAIaZTEEp3154frdvX3hLQETkc3sK110bUE0PVsgLaPLwDxKffmHqL97R\n/c/OqzXnIiKdnh6/QJukzBEoYmq02/TD8YXxtZnJZFqNTTTpM4uIRBvTRivAhCckOdTnkke4vimS\nyfbw/JHeY5HND9z3bZN+e63PqoDPv9oPKuMSEQlg2bBFM332tWnYWQJTcHnLQdfLvXYVEXFg4MlG\n/rmS2aZ9os+VIBm2cK4cz9BFyz+zJqC7IRpIONazG5+/zJIirJISqmw1+fNd9lqz11LLTZH6x15L\nLTdErj1Al3aqtFTJDlBpoDFaEflmelhFoP89C9UGpGlqEV80YZogZmS1lohvuTVsq5lFmuLcrHU9\np6YtK8loNn/LBNjYfXW/NcfxeuLIpPm6qAln6m5hWDGfrNVsv9NS3P1ZrPcRm1wkXQ12byVG/sxU\nts2WaosyQHc+9rYxe7tNQdnMeoBi6a+RtatNvB3IGANTvde8ZBAP/xuCxtIzwe6sR1gbnHaEkgBS\nOWc7wLGbJhEhqLkzpESDviHV7OuzujxFQw6k7nabvtYgaBJlyLyUfjQmhtr7HCb2KYJweBydU//M\n2udVwkdi9UU8QaRbYZ+2YTmFMOW13kddB8z39Y55h1esyMN5ez64vNrR68Erk+y2IugWL/gAH7sa\nN/AyZx2kAC2ycdtrqEit2Wup5YbIJ0IlbSmguTRFCKhxtQ9MqomQ6cFAV/vNNkoiIhm0PAEnu4Yh\nhj3PPwWCR2Lav7/pK9o2V72TWIN3X3Uvld+xTdEAoJr3LpQx0TY+oEZ+PNLV2fY1J50zmzGwfZEV\nWgbtskMsQCFG67L2//hSNXo+9hN6Fuh3bL7BtGMw91pmnes4GmNWohEkY9KdTInSCLAahEAdVCYy\nvWaDckx/lVYArTBznvxCxx0xhWeq5tjDoUyp4vivDzz7WfAQHWKXrDbD+GztBb7bqg0wRiE79jIF\nZrelSKNlh2D16aO60lBJL/dhPTnd1h7nW9fMyko8goS2f3oZOwijLZaty29OpPJdCjy/ZdxJux9Q\nA1G5ci211PKHXj6R9k+EN4qIhOBM64FbjCmiPdNc4f1cteQQ7YLIz5YZbddr6/FsHGgbCPI49hPf\nDRTEYnsBPEGnv9aGz942TReZspvG8M1Ya236vM/GqgHe6GvTi6sq454s1Xdnf/KTuffHY1SyEeI7\njXQf20aKEjBVlW7nXAgpPoEVkJlSw6Kb4ZzQ6F34s13L/FNtyBF6Kr4SCstUWYRGm9ZnpEbPN1zc\ndOB3Cnb1mWVIdTnDPBShcUTcwAkwH2eXfq54XT6i9jnqwc10sAEEv0O2VFa7BgAErZu2tysEo13O\nQ3VbYmDETCcy9JJ2YSkYw43jaGGM7e72Ty9EG6pgneL+/D5NWFEJ5pXU1hXroS11+6daaqnlEyiE\nESeSGQ0SbhTgkq2k2/CYx24XPvdIfe6nDdWMrOcW8SysKTRyw/j1tBa6cCQ/11FgxrnxmVl3/i74\nxM7WAE+YFbxktcVnC9rHAlZyaNIpuPTu9E0YGzKN1TJ5NtW4QL+93tqHGn6AbbblcgOtndjYUAxj\nb7LQ+7hoqApgrXhranw7zFFzAl8Z2RHb7KJ9Xi2ksbyB1Fgs3CD0lMAVEQ+8KYtUwC6UZl7VMy6T\noQCmsOyyPA9q5YOOzvWdHT+fj/b0GXWfEVqL8xo8VnMGoAp83RUi91brltqRdfrmVzG/u61BN/dh\nrINQbxbZmD6ZpUZmsdBq388Da9V7zzBWgHLYBFJEpDUBDBoW13rEzIEZ1HNq2UVqzV5LLTdG6h97\nLbXcELn21JvLqnXPTLl9z65WosUjHdJ/cOcflfv8w8kPiIjIj/TfFhGRN9HxNSm2g1bsNd41pU8k\nf/zJ/u+JiMgE+3ym4e2eRaHm4VlfKaBJT/X5tsdif73n+3OJ+H5fmSkXYwqQKbTzpcdQj9FLjSSK\nc+D5K00mGOxDvnEBWqTh0IBJ2FwBlEThzKzZezpuuhirgKkaM26AYGL2zMM0kMJJRKRzDNOesHlD\n4kh8eTICPTPaw9uOoqVrcKjP4dXb6oI96/p6BLovcR9uhake7MONYz0/Mf/7bZ/fewj8fH6KfuiM\nuJo68OVeFfe+uAsAzJ53JRszUqPhPmz1HVzOvIP929vEl01U4sVrvY94gjGPzXMhth0Vh6mpPecc\nM7CXdfULW58eoJ0wM7ll7bvFldWgmlpqqUXkE0i9SVFlVomfqeZ7vaVaM0aQ5te6Xyj3eYT+5f1Q\nqZxfv9R98yvyDCm0/Yvdi/I7AmR+s6Hf/UhLtfV7JiAUYIn8tcnnRUTka2d6jW8M7pT7fOXrnxYR\nEceacdTZt478NFKDLl9RS6FvYLvsMc7AVHG+Db2MQPTIVToDGGY8t3kcBHJAE90dmzQQau7Hd0Fz\nPQZAw2YAN+q/HQJakWG95v4GOOrvkQAZahKeb72trQjqiYG7tYFG1uWThccChy6RXr2FdmBMe366\nf1ru886+1pGfv6jn7r8FAs62H0djCoLGaZVNJzBU0gxyZe0rVCOHtNG2jKAlEd+iah8NQk4agEEb\naK2DJRDHJLf022hRLFG11wCB6Hq0DbcNYbA2wFCTdUxQ9ENUd63Za6nlhsi1p96yViG2uSO5xajR\nm/Dhu6H3uQdRNTVFjb4yqbOnl8PKtqdTzwTCwpF3wS7y3sGbIiJykXiABhsq/uP3tcXT7EhX58dN\nz0LSeRe+FJr6EWbaPvP30wLbySWYSAzLswD3U67AfYJAAhPDKFfwaj152vGr/PJWVDlP74lJveGW\nEoBBmAay/lxjDC27Zo33NhiF4ZCyfrprtPbGW1Pu2/XjYG37Cm2bnpyrY28LlKKgqkmtpbaG//si\n+tQPgEo5aPjU2xcOFO78VaRdLx1SkU1zXmjSkloczUF3dr3vz8aUS8Ce04lpPjIC8AdNLkY7Gv9h\nTEREZAEAVwsxh01uPRGRgg8LMRnLME7uALZyiodID5qQFOMhZBXy1pTfp5k832+vNXsttdwQuX5Q\nTWAim+JbObVRwNLGitk3+EwysjD6zpJQFriIiNwe6IpPXjeyuYiIPATcdg+lqaNQndO1QVbcRaXB\nLmC6wZ2i8r+IyHs5GGXYWHLGPrp+zeTKahsVUAhLpV9Py8ACIwpEhmOUWxICagETjIaX/GxDw2iS\nEnKJMW4UiYhY8AdOHWwXsnTOUD6KCHdmsMUEgaygZeIdgmpMNJ6QaGhZMvou115rdkY65yxsik0M\nJYVmfzjTeM1hR313W1hEuTtUGPQSMRELoxb4yjlVKTT0bO7Pw/0JP6blo7vrcRFiH5NL/d/t+veL\nGRSeJydLrsn2BLj/gqXblWKdKuyXz8NaU9G6Ctel5SemdXqwrjV7LbXUIvWPvZZaboxcf9Wbk9K0\nEhEJEKwbIH1xAHONpIwiIt2mbrtINfpEoMWw4feh+T7Ad7Y3GoMpC5Q8PY3VrGcKT8STSB5PkDYB\npnw2M/3AT6r15zS/Gsb8ZVyR9ccVvLKjKQas/jHSMQZ/DyxPmbpi8IsVUfa6BYJxNN3t8dyH5rth\nq/a16muytwDrf2mBJmiWEZKNxwI8iC+nrsB4zNvEWFtOMkiY2I2mD2yR+Yc02WtjxkdNVubpdydL\nfS6t0B8/bCJoB6DNEQA7ll+AKbI1vSGY2h3Ti361hmmOtGDFEsaN5DDJGVzudv3xMWoK2JiEve8C\nU8Xnz6PfJYbUMm+xswfcgdMrSteI24cXRFPfmu22/v0qqTV7LbXcELn2Lq7Rwkli6r8zrIpMf7Ai\n7KvzB+U+bNRwjrwS69kJPxURuZzq31zJDWJSUnQLZQBoBO1/vPDQzVtdDfDt9DUgN8UqzU6jIiY1\nRe61DuCiloKZtdG97XRWkFZhmayfvgrySDhkAI1uGUnYtaoEg9i2P6563c3mCLoRH4wjAXDSMK2E\noqlqrlKDJFbfYa4X+t2CqUCj/ZNhlSGnhBY7P59vNzStOX2iz6HSPgoadAq4MKsYT0269OV75yLi\nGX9owV2atl7krisr+tiLPjFAqKQaNAssCjuoaktSYs/PuubLqzWqbUFGWHhM5iGzn1tv17+LVOHH\nfP4MxobbhZKStV1dz15LLbVct2YPoPFsUzz4Zj0UPjDFcqd5We5DtpghgBUPu5qOsQUoXNXPwLQa\nmFWV6Y7zEwW6vI7V+vTEA2+mu6pBxqcAZrDZ4NpfY/C0qpkJLrEw08YMoJpdbJv7pZYFFlHZhH5b\nIzSWRWUTQS02vRYPoa1QpBEvtp3lZF/nzKF2PTTw0NL/ZGskbFuYevZ4oNqWaUILty2bC+I7wlMt\n2CbBGAukB11Xb74wzRvLFBlBMAu/jRqUz6OBxhbJxPvjv7r4rG5DHGCNwqDCWAg7YOyZbHAd3h9N\nyn2WqR53CdjupdHUt3eQ8gMwi9p6v+eDIIQA891lLCK44vm+eXGo924sNb6XBe65jOFULL7qucp0\nrfk6HhWVuMmm1Jq9llpuiNQ/9lpquSHykc1451woIq+JyOOiKP60c+4VEfklEdkXkd8SkZ8uiiJ+\n3jmkEAnXTtLYrDHszIp02jzRoI3FSd9qa/CMKDea75YOqof03AIUVrdRLSUict5VW3QFDDOrq3oj\nb38zQMguqHmyES0Rj4ojmSLNchMzKoNMBeyrxKDKaJalQ9A859sVaa1x1Vwj2s1WNBEf7QZqqsc9\nz8PExhnsdspgaGKuwf5nNLEFY2yd+YvQfC+JFa9A8JU93zpVHL+ISAOUV6U5D+QakWgiIqsOI5Ws\n9LPgfAQop9gfZjx7vtl9SBPWQ2OJnY5Pyb48PBMRkfOOPiSma++2vRn/xgTkoOgKa/sEnsItzFY6\njt6OvjOPz0Z+GHh3ott6r68OT0TEk5eKeJM+RiXbK72zctv/135JREQetfTdX81BkW1SugyUMohH\n1KKtM7G99q6S70Sz/wUR+Yb5/6+IyC8URfEZEbkQkZ/9Ds5VSy21XLN8JM3unLsvIv+6iPzXIvIX\nnXNORP6EiPw72OUXReS/EpG//mHnysOi7MUuIpKig+esBzJ+rJI/sP+k3IdY+LKzak+Dd2+eHZb7\nMOByKrqC77Z8AIXElATurNLt22ZzCeKr2bbImcolFtmRBIdpMqvRaGyU22zGiqmdJVMtCOQYjZZ0\nqyu4rc0ur8Fad45t4q0QWhZJJ8JYof2MBsh6OuDmOSvj9Pur6tkZcLTYegYopy9W7zk09eyspJOh\nWhgDVIvNJ14jjkb63WWIjqQrD2BirT2DbSHGb9NZewiS8XmS3PPZzKdUz0Ecyu66TZhKs8wH+mag\nBmd9vYGbS3ZZ5RxYzvH/mT+eAKZ3nDYR+dyOVuPZqj7WY4xBBLo20cx5DEw/LF5ilWwZAAzX0tIj\nu46tmAzi7w42/q+KyH8hHly0LyLjoiiYLnxfRO5ddaBz7kvOudecc69l8+32x7XUUsv1yIdqdufc\nnxaR46Iofss59+Pf6QWKoviyiHxZRKRz90HRmLtKnW4DRPiXaNrIuuO3d3xDRTY1fHVXV+dvnGgL\nIMIcRUTmXV0dqZnZgEHEr/wjNHYsUyxTD4xIe+itzXFDkxSmRjufVbV92QZpGxXpa7vNtnxXrYfD\nW2qZsCFjtrQoDqzy7apGJd+bbgMcE9DL7os+TUn/lQwv7x3pPBbGQjk8UA14saPzmqLpY97085l2\nmGYE08vEWB9QnGzKSYsltKkzWBQx4gMLVJk1Lw0bzUyfZ4b2VY3lthVDDU/evcXUP9duqxoiOgPf\n31UpL1KWs+Gmbb31Qn+ytT/lCUBVBZ5jCw1I1wd+n2QODj2kdEM89LvNcbnPYVSlFF+bHNleRy2U\n4LYefzzDu2/DPfjRlJWT/Y36dhEJA/dctpqPYsb/mIj8m865f01E2iIyFJG/JiI7zrkI2v2+iDx+\nzjlqqaWWT1g+9MdeFMVfEpG/JCICzf6fF0Xx7zrn/hcR+bdEI/I/IyK//FEu6PKq70DXibxu5Ex7\ndLFT7rMYqwZgu6XLC2hkA554X3R/NkmgHybifTFG3Bcr+F1P/T7THoohSPDPNsZm5Wyd4Ts4L62J\njofsMiIe5koWU8veQj62W71Z5fPR2N/rZcKCZUS4GTnvmKYX0OgHAHyMWsahhrw80Gjvkwv1kQPD\nr8bMRQ/Mr7QGHobemkqgpRkDsA0gGGuIDtUPbSHyPw+9rxxBgwcA07AgxjKrUFtnANVkLf9mEFpc\nNp0E1Nm26X6hjzp2gGL4fthnv7ktgqnQCX2knDEh+vPjpXeWOe4CgKPkKmjsBkaV8YDcqNlx1q1s\ne7r2sQuyLDFuVKAwxgK6mAniZ1nMZKyhIJHnNor4OHn2nxMN1r0l6sP/zY9xrlpqqeUPWL4juGxR\nFL8hIr+Bv98WkR/97g+pllpq+YOQa69nz6NqeoBprAJBDuIQWP8sIiWVEIn9+L81c9KWmlnRqe7z\nKDcRFJr7cBUcsORNA0JgX618BFMqBCmjoSiiuU5TnZ+mfZknitxwB0RE5JnaYE921WxjPzf2W9cD\n8cnjEejKxe9jsd8iIrttn2Z80FGCxhdbWhH2+FBdBKaeREQe9HUfBq1o0k4NZdRqAHJNAJBSU2ve\nbev+DJDRDZibICADeznqEhp93ddWdjGdJgAFJYl/niHnHaeMF0ST+H1IS/ZCF2ArjKMz9CY6wVls\n+sGuuruGg4tVlaQpa5gXdHqgz+ydsbo4rL2f27kCCGeEiknOq21iMgHF9yXQSm+Ofdr48kQDtd37\n+szKevier41bO70eO/CW5roJ9DVmQQVPvyk1XLaWWm6IXKtm197sPq0k4v92WMW4cL90yzd5OG7r\nyvd9hwpW+Mr0ZT2fAZy8fFcDUk87qjXvDT1c9gJ170tURSXQmmnHtECAlmE6Ky+1p1+dydriV9Xt\ne2T9ObelIxNYQ+ugF0eakhmvdVxWo+bgm2YQkU0fLCiGASFql/enPsC311QtT61Czbbb9oiZPsg4\n2YN+HKMtldEKZYdaaGibpizvlRBQVq+ZlFcBZEpzoNd68UCf51sv+nv94TtPRcSnSY97/XLbRQNV\nd7D4WoDLZsd+n2+e3xIRkQ5abe3gHmPTy573eK+n2p9w2WdrX/FIrct5tN2BqcnPztmEHWlXw0LT\nAWsNW3+9O/P045TzWOfv2Vyv++TpbrmtcabXeyZ6XOMU1zePvINGICs0kkhv6TUrAbqacLKWWmoR\n+SQaOyYigcEjpoNqoQRTPaGBGroN3/LWgaZcLoy2ITz2qWDlNIUK6bmmUgLyslH5RmbppHLi0FCc\nYXt9EybKOEPJGmKwHQ7pvZJK2qy0EVJUp0sFs7Dtka3Lp/+ek96YjRwGZj6QhiPtNuHDIiL3W6pB\nFxgsNdvA0G7PwMVH35SFRquhfx0m0Pb0cRcrn87KocHaaIrwqZFaVV+Z+OeRA6hDeOt4ifMZHr9v\n7Co4ijTiFSGoiWkogpV6fh5WYJsZAiw1RlyiZRpEkifhfK1je/dStWe/6eeDMYcLFKCsFtttuRhT\nEr4zl6agh3EF0F/TQrLPlbyJZdzJAKlK2m8WNEHTkwlJRCSbV0FOglgIU5MiSAn/AaXeaqmlln+G\n5Fo1uyvUpygsrxcLRrBSUfseXXqABiPBJdcY/Khhz4NJyA7LRooz8ZoogX9VgFys2N3WJPTB2mAd\nXWJcSWzDm9VSTGr9wLb5oYXwQC2NXTPGl3c02vrqQEsg2fDg2dLf6zsXGvWlzx73dSC7d432RtHP\np9Dk8I8OvlVum4J/7U5D4wLBre3I8Pe21Vf+veV9EREZAJP7ZOmtIfrj2RX4S2rrBJrrEky+xdhr\nxCbaFS9D3bYGW6+pHi2lgfNZa2410Oe4cvoc2Txx1fKAlyFKWSNYKCyAsZbC7b7Gbl7u69w/dGg6\n0fYxHYJpJms9907Pxzdofc1YTkumHQOkiWBhNcGYQ/irLcFmpmCdsKzZH09SpphlwLAUK81D8EvN\nN5pPVtpIuec3d6w1ey213BCpf+y11HJD5HpTb6FWb9nUW7DH4nCAMBAY2zOEfgy4kF0kB3LgyFBB\nM+XUCK+wE2F2s2Osg9kY2BrvzX5d6IfemJk+biVRP5lZEDQxmJjVHoItcAtskIbjHaG5Ac3Hphnz\nDnrLsdEBr2HPQ7w3G2G8F3sA0aMV0j5gz2Ggbpz44NntxqTyXRcRx8naA29YR8A01nrh7cUArgUD\ndCWrkPF4ymAmCEUZVH1mAER7XZ2Hq7D9WRc1ATj3CCb7pG2Cski1EVyTowuqTZ1dwjR/EqiLwqYT\nDFyK+OdQjt2Y6DnmoQAQK+T7YDvOTvQaKQJsDr3n2qahRYL0YNYGQMz0gSMoi/h7PnNnvM0yRY3c\ndIE0rv0tWXLTq6TW7LXUckPkejW7YEG09d9YmFg5lQa6/tgaY2rt7+0qqIbBo8ggCO60dP9RSwNc\ny9hrohVhh6y7RiOJ0Gh2BlNYiUWus8S0Yw24qmJVLrtsmoAjudcISomusDSmAJFMY9U2VsuwAQYt\njALQSdvUYAZmk2iAXvaBz/293NY02ADMKPebGpiyfc0/1TzWa6ESi5qe3GkiInOomwmadkwGXuvT\n+mArJrZhssAfWj+0nmYIdDmTcpog1VVyyDX9fSS4/xjcb+QkyE2DkeM5aMNR292JVBWmxgrKoMlX\nmOMUEazYmGP5c6Jaabqxjdx41hhApWYGjTzeASuOefYc/1UBuk36Z85jmWYTw8/Hdw0pQGtVPi/t\nJlJr9lpquTFyvak3wYpoe0pDy1KTcdMq9Zp5FqiWeXelvulprCs6gRIiIkkPqTtoaGpWEZEIKZEE\n6R/6YUHgV176afGa+2z7oWSTZbqDPnylxRPq35fwuU0/RZniABZKzAB3DYPtFA2LXdxKx7oyDC3k\nP1nubDPgZhgwtdXvL5U5tWVUEbc9Xis89DjWWMK7cw/z9I0TVLMTMKL7Q6PugsUG82i1FZlOY6RU\nl4WOvzBNPWOwwCSoYycXm4gH8eRTHcdZA0EIYxmML/X5ky34sKfFLcUVPZCYQqSGPc097DaFeiRI\nKzNFPyxe4jdX8Q4SRMPa98sFQFzmuRIsxT710vcOedqtchiUDUMto9MUsRvc/xrvp20D5fIr+474\ncX7wplpqqeUPk9Q/9lpquSFyvWZ8JhLNXAXzS3OZ3VfZUZP4cREfJKL5SXyzTbG8Hmqv9adTNUmX\nppIsIdaZ3TJhdq3nJp3UgZnLOAhNsJWpsUaFUYbUE/vBWXqssv8Z0iiFpRYCQuoR0lgOaKyiZWxC\novNAC9V9goDS0NTugxjyNadczqmJ0pys1DztAYb1cKKmesegyg47au4+mWkdwSXIINdzgwln4waM\nPzQpSM7RM5jLOy9owM4G6NgcYmdPr8XA5/nco/R6aOjRgcu1MEHVblu3rUEX9uBAnZfjlje/SarJ\nQCHTk5YqfJ1UX3E+qW7DBwPbCOwdjPR8Nrg7YVddYP2Z+mo92uYgIO33Eqk4G7gNyaXALrumPx85\nC8r3iXTklh+ATWjb1ZoBG4x02fMJJ2vNXkstN0SuN/UG7G68Z/pW47NMVQE8bVNv705Uk78ZK7vH\nGL3Yc1u3jKDK9EwtgrDjA1IE0TT3VVsQwzyfGpz1QLVTFFaxx6dmBU4QkGLDg5IE0QIsCr3+Puia\nT4983XQhVYugdUvP0zGUyLz+yZFqQLasstVebqj70/qwaSRWlz1e4fhn6GlvtMTRLU2V0dIIjzQI\n1TagDPaXZ6AtNA0kGOtbiga03kg1CBhdGm0H7cQurC1g3W16iPwC+UYbJxFviXQHVYYZa6Ekmc07\niUzXeh9l7bl4ppw5mHJSBAWPG8aaIrMMLUAT5XKsdmvTZMNYjZpk1SMDa3kT87qwhJF6HirxyDRK\n6T5jYE7ng1j5xDI6g13loQAAIABJREFU8XVmBBu1DrYqs3lZDRxuSq3Za6nlhsj11rM71OgajrAO\nqsxYqcTUmW3yQAYQ+nb7I/UDLQ8YU3chVuCBqTabL6v1yXcHqnXPTQnWIWidz9FoIGb6xabezgFT\nTRFP4FJptGYLoBr6wW5mfDPcdvOCviX9cb8PmxCEbYCMRkjBGSrp4RCgFszHF4ZPy21sGPjtmaYp\nf2eiTQOdiQvsDtWi6LdUa76HtkWpSWuFfT33it/ZZpz4MwA89HMPFOz0RuybArmwagWtkdIkW4+I\nb4hIC8m2dmIFHOHP3zNUINDDcDs9+AJYaJieO+h6frn30W6qB+vpDI05LCS2YNoXTSRtGytCUFcv\n6Tha57pv0/SVaKJFFn3sENagZY1JBtXmlzZFxrZbwFiV9OqpaQraOq+m2tj2yXIyZC2pOehqqaWW\nT4CpJlw7ief+stMC7CBt8MMBHnk59Jp5Te44WAEExfB7EZEUx9Gnmgemnh3N+RwimI1burxSm4t4\nzfFGrv7nFPXw1vdn0J2RVdYxV6LY5GXDd82Z5QgDw8s5fDRkHpK+jx2kDQAy8H/B6KuJ6k/RCDGC\nhfNw6bXd04XGCGihRGfwi5t+zk+RKZiAOy04pq/qb6OARmdLJst1Rm0Sj3RM40Mdj7PwYxQCJYiG\nM8tCLSqiTT5F/Py1+x4hwsj8Cs/43fl+5b5EvNa3cGMRkcRCYfGuEJqbTADbtTwF2H2jHka/oxFK\nqDS5Cs2+axTgkDmXMFc7Z2yfVWpmWzSE+eR3LCKy0XhaCRyP4ytj9gnXUjPV1FJLLfWPvZZaboxc\ne5OIIhDP+Ci+hjejKZNuNyUgpp3SQpMCfoqIZCSshAm1v+NN9EUHFE80KWHaVXqCgfLqdkeDdySw\ntLXW7D93b1ejMyFsq+nQn+eopWb0IUzsk8TTPLcfs5+c/s8+BVnLmPq4JZp7NOVsZVSMlOFFW6/7\nf8+9G1DW42M+2sBUJx6jJDlqqpNUjw8IzTbkhTRJy9SbwWCTUtuh2myJeQ0M7j1DymvvQOeqMdJt\nE9NHjUCbHoKzlouAz+EdAKj2W9vtvvuRDuqPDN8XEZGvXLwsIiK7Lf9eLLtIXcJGJojLul7MZuUI\n+BU9P44A7kewxPvFDr6FCYzhlsraCTyrdGDAY6iUbJ2iLsO4AUzjsUMrzfjc1LwXaEhScijw92Ky\nj+1zV2Pja6mllk+AcDJIPORQv8QH0gy9Xc1DfPbwqNzla5lCYRnBoNaPLHshgRkTXZ1PnAezhFEV\nins80/TLxTO/zzNQTxPOSA2QzwyE8z2drnf2FJKbd0FzbDq9to/03k7v6HI/eM+AJ44QIFzoNab3\nkXIygJVNLZF5o8HfKqGTANo0TcrqhT20QoL18miONkMmRdM7sLV4IvFA74tpUBGRPuCqYwS2lms/\nDwQlcWhMZY7H3nxwgPtezDSgttvXa9qU1xf2jyvffX7gU4inSRX224MWfzjzzRW62Pb78zsiIrKC\ndWbHygYSvMYUgcO8bVhkyMKDtGDv0M9PvIN3DcNOkHadGmMzvs2omX6E7Fxr23QRfR2Ci8AAmNrK\nGyoNfLfY1XHYzr0JfqoRhlZqdHOJeFgTTtZSSy3yCaTegkSqiX/U9XZQFMFab1vcQUYZ1kbvABTy\n4tC0iOqqtj0ag+et79XlZ/fUSvi9E02rff5A//+qgVse9NUnvN1VLXWJ4hvL/fam0+O5RHIFz8wy\nT46xsrmBSTWlKJSw/dxFqv44WJ3L7wjFTDzdnqRdVmfoB+vCRXxNeFnMccVyTl+5rCdH2tJq3SWA\nSISwZit/DQefmPGB04Vqu97AxzdmkyqQ6QRglsT4ym841cgEvMxSb8aQ+468Bn/89psi4mGzIiIv\ndZWFZ47jmD617w75Ctl+iZDawNBWky+QPHvzsWflYcqNqbresX42L/0znCPAwQIvWg3OFMIUG4w3\nychff3EX93oL3w10fvsm/TxH0RJayZepvNVtcx8Lt9kqviK1Zq+llhsi197YMe1sNClE1JjgCXLR\nNS0zJwEZ8KOptdiiSMQ3b6SWmoWG2QWrOhsvsLlfzxSgsExy1FCL4FZLNfw6N1rzJWhNRJ+biAVc\ntn2EeYFCmPa+nicwKjkG9LWFQodwvQ3QoG+XdaBBxoy6+iU7A7MLo7ebLZxFrmBrMXNO66lkZAGA\nJjeaqCTq4Xe2PXZafR4RtGRktCX9zVGv2sb4xPkiFTLosvWzbci4QLukp4il/GZTYb9PLn2chdmU\nS7DaDBBnsE0s2aq6j5JWtoyyLcGDsNqCzKrAAJYnS1zXe4Blm8wFobMZQFPLDql1/XnIJRitWEiz\nDZhh85QQWY3ANsrEfGYOlgZaWucmc1BMghouW0sttdQ/9lpquTFy7aAaV1Qxv9FIzSRWspFZZJF6\nM6uFgFAX1XKkHCbwQsTXcefAVzdNWo546vVMzb1vX2pFGCugRETOA92HAaEXBxr8eTz3oBiaqQP2\nHQMjynLuXYbWMYA7l3ru3Tc9GuUSbkD3WI9vTdQUs33i1yP9rjlD/zN2jN0xZh/MfjKc2MAazdQF\n7pU15nnkrzFZqGnM6rsGzM+0ZwJbLZj6SO/ZdGk8BVEmApRs4sEeZyIi8yVqC2CKHp2r+Z2aVKaD\n2fwMgb4jixdH3zj2/vt6guDoxBCREtMODPoMgKqnOx5Uw8BYG3XxpHu26JPSncG+4cBUZSJwvGb6\nFsxFy1uGwWjFwKv+3z4CUMwwMhUhQTn6f3NqyTmrx62Q1Lxs+XuNJqSg1v8bwIw5415V3OMrpNbs\ntdRyQ+QjaXbn3I6I/A0R+T7RENK/JyLfFJH/WUReFpF3ReSniqK4+IBT6HkK7W0emOAGq5BSMMWQ\nfcQ2gCC9M2vey9ZEJsVCphpq9IbR7J/bPcI+eruf23mG+/Ir4RTBnldH2iiB1Mtjw1XGABLbDX37\nXC2EokL4j/TLPbQr+pQP3q13AexA+icFTJYgGxFf+UQILeODpnuThHG1ljkzVYRZD6k/fBeh6i41\nx6doLrG+RS5s/QgsX95Q758aMTapt0az2kX3Xk/54RrmmV3u6n1/eqiIEfZJP+l74A2f9S6aTlgL\n5Umi4JkcrygZaywciMHDfID5wHNotkylIiyaJpmQAJyxAUyOYwarI1v592rGIDAsGzLMBAZIFdIQ\nKKqfgamLp1rNywYQflOO4wmvjW7pfFQorS8w/4T28nHYSyTuu1L19tdE5B8WRfFZEfkjIvINEfl5\nEfm1oiheFZFfw/+11FLLP6XyoZrdOTcSkT8mIn9ORKQoilhEYufcT4rIj2O3XxSR3xCRn3vuyXIt\nqGif+DVmhRV2MoMGxP/P5j5lxaaLbBxQNvtreNBBGys3GU66De93vdJR7fJVp7DbGMsi2xbZv++3\n1Thhym2Z3ir3Ydsl+vWTC9VSjSO/TJPMPyFXmVmcExSOzO+SoQZNFi6NRmUfbtY0EzizY4pMoJnp\nr3UMvHMXqa7VAMU/8DHzlj+eTLo5tD9ZcIqO0VbNar/7pYHktjC3ZLrhfNoc4iKpFi+x5txy/NEK\n20GqbJ6YOE1PLQFTf6Ni/fo1ccPkZcPcWzAL0oNz+MFtFE81Iz9W8trNkR5rdJKt45Oiej+2SEXY\nEANMM2UKzViwKXgJyA5rWWjyhqts62GMK9PvktcjNJuWWgVS2w8q79umfBTN/oqInIjI/+Cc+x3n\n3N9wzvVE5HZRFAQzPxOR21cd7Jz7knPuNefca9lyu3KpllpquR75KD/2SER+SET+elEUPygic9kw\n2YuiKOQDvIWiKL5cFMUXi6L4YtjpXbVLLbXUcg3yUQJ074vI+0VRfAX//x3RH/uRc+5uURRPnXN3\nReT4w05URCLr3Sr97SYVUInFNpj0AsGR2EXYR7+fm/Qc0UbsGd4yCDzio9lV85WOBuFO137x+UxP\nv2O30xnKzw7bvi5+gl5zFyS6ZBDM9sVmzCunuWVM45JyWD+jaLtWnA1ZWWpP842mux4I5B1M3WHX\nYKgxtjYq2Jb7CAIa6mQ2ZxijyL0BLHdignAkf9wHeeOx6Y320khdnS6aK9xtK4TsFjmQReR+W4N2\nI7Ap8vkcrbx7RsJI1qoHRl/cHulzeAw+AFbh2a6qeWMDbw5se2QCdAwA93bV1eEVbGqWrl8H2H7r\nalxeqL0cnuNZL7efGXHyruy/xtpzfz8pTPpkwAibP56/AbpuCyAL8xMf3G0jTVpW1vH6JiVaRMXH\nQ9AVRfFMRB45574XX/2EiHxdRH5FRH4G3/2MiPzyh52rllpq+eTko4Jq/iMR+VvOuaaIvC0if150\nofjbzrmfFZH3ROSnPtKZXFGujvo/8MQk6MdKlRoq6LLKDNopBwjjMvErHztwnrLW3XRxpZVwNlVN\n9veefL+IiCTGemA67wzabgiEBNsp6Xl0rGw7dBnptsL5qMhmPfF6z3g3ZbAN1kuL2Hg7H/ggYw22\n5Wem6qzUBKrJjm3qD/MXXug2atSk71XJBOxAIcgoE1Aot84NqAYWxZtHmnoKDEHjGw9U89Ba+Fqg\ngBdi00VEYqQ5X9lRausnM41eEccu4gE3rFkfr32tAy0zWnokoLQMRqxcS9lhFlNNbS7igTvE4Zf3\nZy1HHLeY4F5tAwnMJ1t+NS6r4BgR3/ZpU6taKmnGMAmKMSUXkqDGIR3oAX1YXtOhv9dshipEDLt1\nAevBpgDXrnLNTflIP/aiKH5XRL54xaaf+CjH11JLLZ+8XHs9u8uqif+EbYbgN2YB2vXYxg4J+cOw\nyoKE36ZqWFtNQIQFRkxA1Uw/NL+i6Pd0qVqafig56azPTs45MqQEgIumXT+NBLoQ8MF0iogHYqQA\nVqTUFlNTuw+fPYWSW+1izIce3FMg5RSiIuv+vu9YwJ7vnfu4D2jEYXsriSWTHZ0XgpWOz0yrKlgL\nEVJwmanL/947Gp6529HrEoB0HvsYyNFSffNbLZ2/x9DsiWm02EJa750L5ZlbGmpwwqczaNb2Dmi7\nQ78PWXlmACnNANFdG85CWgZdwl7XtBD8/SSwDBpHsHRGtrcTrC+kMNnQwZTVl9q2zEDSvbaNIJAd\nDS5pafl3kG53AcBP2VDDtB5rjqvvbAQgFiHPIvLctJtIDZetpZYbI9er2QMl0CeIQMTAS9niGBrF\nMpoI/Edq5hJDYRsBIvo8h49mmUEI2qCW+2O33xIRkbfnB+U+9Btvo9h8BvK3g4bX7NOuasL7TY1G\nU+t/87aHGDCyS3ba35YXy205ijh4zyGKgNwj76v6Ihf9ZLuhxDDh0kJIBjpnj0zr6QiwyjH56c6Q\nyVhsF9Kwi9US2qtrijNofZFlNjSP4/XFAxERef+OausXhjpnZ6aBw+VC5+rxhe5DbZub50IIbp6w\n1ZZXhWcFYiVQeyx+moifK2p0xlIYwT8JvIUxPFTt/4MHj0VEZAcqllkCEZHXp9q26rfael87Jrsx\nYIusI21SsQKDUWAMpZI3kFwEgDovfEhpq/mjNS4z+Ow7B/o+3err57cz/37G57hv+uznOLZKCPQ8\ntGyt2Wup5aZI/WOvpZYbIp9AgK4KqiFYJADpX8B+WafGbEXahzjvcK5m35OeDyixVzvBLOHY31rZ\nmRXBoS5s5NTkyUhVfJmq7ZVj24UpN2Maawq7jUE8G/C7372sfMdqLRGROYKGtiOriO/wKaKgI90J\n94Ua7axjcipt8hjhf5N649ym6EdHUA8BHyLeReA1iLvfBKmI2P5jhtYKgUGmNAmOqdxTXH21cpjs\nwcykKXlOBDpJUSYikgJA5TBn7z5USuxgYp4rOAhyzMcEXWXDx95+PmlooPAf7OrEupDvmak1gBvU\n/5Z+Xhz4dOv4ASiwL9RebtEdMvZy3KPvpR8O6UGbBiM2vnAkrPTbyq6+cDNZR1Dk1t3FJchazevb\nuJ2TrfSflVqz11LLDZFr78+eN6qND8gKwtTITkeDI89Cw50M9dJCcwJqjY5p/7Rid0yyLLf80juZ\naXAjBpzyH528KiJV8sLjnl6PxJXUWqbLj7y0p4E51rNPAAI5n29r/zZSePNjHyyKQBLIdBJbXoUm\nZcLgW4OtoZoAHfW8mmCakkSJ6ePu1vEResl3n+r/7VN/I/GgWjmVg0UlNMAZkigyBZj4qSopvQXH\nkycgNoAXBk9T9ion2MfWeOPvDBqNGlpEpGDqtAsrCNaLBSCxBVKIGvMW0pNxYNphIWV3564+O09V\nbtiBejrGCcrWCqP1abMwqLq6redrXBhwD95hUlInTq2AxtCkSzEfMQPID/2PgAV1bPZBsJftV0/A\nTWOMFCDoyG1dvTip2z/VUkstnwAH3aawlRBBH3Os8snCFLkA2CHQ6CVDiWUbgU/IlJv1l0pGGmiH\n/baqTavZ2VSQmvnBnuY2qMVFRP6Ng6+KiMgfbb8nIiK/udaUzZuHd8t9Xmxq7XyOdfS/W3stc9RV\nzdHpoSEGVu5Z16eTyDBTAFzE1JeYGu0UtMauUaUnFvGtpJjiCaEBorU5vrwctGXCT9nah2ENCwMm\nzJYasQVoMhs0iogsm9UWSA6+duCVXQkUWWtWS+K+UUtIBzZZ7AP49GLfxweW4Nlj/fkff1FTqr8u\nnyn3uT/UNNZdxFJutTU9d7o2fjnYaCZjmCqG+Sd/X7+jr87YRfeZn/NLcAfQ6GC6tDj152HdDYtd\nIpMKTcBrwPeUUOPMpClZI89YARuOWMnDOvVWSy21yCeh2Z0HIYiI7KPh350ewCyJrtbnjz2ra44V\nrzivIghiEyFunqomjA/A4mJ8OxZPBIhQs7hlvfa3T7+TrX3ZgCAxjQv+j8n3iIjImyttW/TNmYJp\njhY+vvCFHdXeDxca/T3+/cNyW9m47wDXhWYOL4wGKKO90Ihcq40f6TZYRJPbXqNmbT0XfWNmMmYv\nGE2CMALBHAn8Qbv0s91RyZ3Wt5UfiAPAR2VZMaG6Ih6OWgBmWzAabawQxm7KOIOZhwwtrtIGGkRm\niLybiD1hwzHm6slSLTVb8lsyAoPViKw6NhPD9lUBOf3M/NIoZAyFPSPXe346SlZX3FrJQmOi6Zu+\ndGqsGLLQEpK8hBOf2+6RtLDwOtLisz571vqYJa611FLLHw6pf+y11HJD5HrN+FwkmroSJywi8uSp\nmrvJLZhrEzWpOo/80NgjuwR2IFhhCfIZMAkJ3jBW5ypXvwF4BnlrrgG15ok30R/dgU0J8z89hIlo\nwCHfeqzkk100omAPOpv6ev+O3k8KoEb32I+xg/TXaoYAG6znyFDzsXJqvQ/TDvcctjwQh/jyglVR\nxkR0MLEbl5wPfJ/7nVYIiG1FcyyZI9JgQVmSZUxSgILIHnMLnW8boZ8H0kM/OtH5yMZES/lrsHIs\nY8DVkjiSsBOmcAnKmdsKQVc57o2n+lzZxEJEpNHXl+14qq4bg7qWRpzMMNmM8+nvlQSi7Lnnm+CZ\n+8Accc7oITgzweVhPI05AdN6m1Tp4dK/O3yurXN8TvRaq70qoKpOvdVSSy3Xq9ldodosNe2veyNV\nPUx9uSuWpsaMaQ/9n9Valj2lcwJNlJD/yx5P7aQf7MRpU00xVs4QcE6HBgrZ0kwRmVCg0eML1INP\nTCUXAB1saNr0HaokJVUzzhOPkBYzHHaLewB9DDEfoH1+cDAu9yHby6qr2nI5NhFPgbWBAFDe2mY0\nSXZAHQ2NIqBQLlOc4htBkBVob+jpqkkh/RO3vykiIrswTSamEwWrBhn4fAxAUxJ5zZyQenmA9lEH\nfrLIXfBgR+87wgMl442Ib99FoMz3v/BEx2G6+95HA4unCw3esTtsaDrOEsh13gasOvZjXKIbcNbi\newBLy4CDqNH5rATdgnPDZVBE1TZatlFKAeBQH23F+Ft4e8cQtMKKzZDCzBq0avwuYeyem3urNXst\ntdwQ+URANdafnp9iNQXfVnqmWqppa0U2AQnQ9JaZI0ix4mLFDDLryOKDPiLSW9HcFKAAykorgs0a\n2++ZdBJSKsag0PNaAM+GZZFapbvht3GfIDG+HbUtP2BZbDZdEBHJ4Qg6w8rj4wBVoIy1Yli/zms4\n+MEmUyRJUbVC5is/D+NLfWa/7jQV+anB2dbYWCRU3hl9/ys40phKXJt7XIG15p1cc1w9aL3J3Gvt\nErCDXuePjdandNAM9BaagHZQxNMyL2GKF2k21Hscx95Cye/oud/d13FQ6y9nfj66uxqfICvtJS0D\n+wqChcZ1CTbyerY/0uPZXozAMlvMxVQyW01FK1gKJjWcNZ+j1qXW7LXUcmPkWjV7Eah2tNDLshUR\nVij6MmX0U6TUiCwP5PHWQmjMwWIDHzWMDTCiWlEq8Yj7mEtsaJwmimyydmtrH2pCrrZWa5bnxOVb\nF34cJe8YP0rgz/a90o9m2WdiikwI/GFxRWCitiwAyiMWjuB7YxiQcZWQVJa/OsOqSiZfNhfstPxk\nBYieE3D0CG2tI+MHM6K8ANCmnCtjFvHvpKGv4cqAWcgzuID2phWzNgxGDhH+DHM1aanWvzX07ELk\nok/ZBBSqcW3oXWmFnKOPgG0qGiFlQJAQMzou8mMlaIvbbMlxOVaYcyWnn3l3D/oa8zjs6LhXGM9F\ne9cfj3csD6vP9XlssptSa/ZaarkhUv/Ya6nlhsj1pt5yxX6nHROQQgCKtMwFtsWGzjcv8ch5eR4R\nkdA0m1iP1JRKwMzCmm0rpbldbHyKSO9J1UWYTNV8N52ESqxFSFrhBQEsfh+SabK6qznzdlZpxhNv\nDWuzbVwWgjji/bxyzZZpesGGCTSn46HfxkDQOgK+eqNtkIiIgOiSbC3kAOj0vKk+6lZJOg+6PvVG\nCu8ueq6TWvvY1AgEZQUXySRx6+aNC3BKBjVzU88QgF7bIUi1GKC106V3Zzj/nPNVT8c1bXlT/3fO\n7lVun66GTfGukB6ky0GzWsSb9GyIUVKWm1qFdkvnk0CiNYKLmcHxs9adICFLdU4+BLpFY3AqRJMr\n3DP8FvieW5c4b1SJLDel1uy11HJD5HoDdE5XogqZPeGQXAWhrQw+o9JoQcSvcs4EQpb7CGQN/bXK\n6zLrgwWfYBJL1M9rcJ+D26quT3OfznEIrjAdxrY/wdoARcAnx/rjIN22UDZbAC0ODGvKLdXSETRB\nC9DcUctXcvFvaqf3xJdgvXpLG1S+daI0xKscFV0GWkyNPhyoal0grbbX89qbVYiXoE7eaXnq5cm6\n+t2dtu776uCk3GeeqmX0VkPH8RaBJmPT2KNVDdpZbRemVRVFjW6DobSQaOEVMMPaxgriWfY6em+X\nGDs5DUREmgjCzdJqiygRkcfgPFihaQm1dafveQ5e2lUWnPtdBfDQ4rHNSUeYK87n8czX048A6uE4\nAqR4nzQMxyLeeVqD/Sd6jdWez+0WDVfDZWuppZZPAC4bxlWtzfQP2UaSeHv9KdMMUNcJap0z06yw\nAAyzxIKY09CSYN3xJuBExLCvYsE/H0MjGjbUgPXv2JUNKhumkKVAaoSQXtsAsAHl2JyhIGYX4J6V\nSROyEAfWQwRf8WLlwSRsisDU23rh82rPZuo3Ly+x4rfAHGsUJeGlU3DzUVs9Tb0Vc4ommBEoVrKB\nn9AZ4LpPQjDvIJ110PQpr2MwwRCeSsuNsFERkby5ob0H/oHkiOE0oK2Zeott8xCmDBf6jO7vKnCG\nVomIT2NRhrCKGGcQEenjb7aefrT0Ka+SS4+xJVh3hwNrGegYyXhDdthO5O+HqUgyHY/PvWafd6oW\nBVmXKr+EjXRe1sR8Gis5b9T17LXUUovUP/Zaarkxcu3Y+MJVa25Jmjjqq3m1hJk0F2/rs1KI1UFN\n1CjHxnwtwipBozVvGCzbJNpvzPxA2Pqr7J31dTXJQtPTi4G1hBkmBogMmSODhnQVUlO41EBRF2mh\nuNTaQCHTUMGemoajng6Mvc5EfFpr1NRtb5pA0EFXzcuTTFFtm5hsEZH9HTW3SSf1+NRTgFFK3D3+\nt9TLxIc/XlSPm/e8ic3qtKNzDTIVSyDxnvhXLlxKRdKx8e9wYXb55dx3J9X6bRHv5j0ulCbs9L6h\n74YbQmx9I9ju5HuMjrMkpbTCxg1lIBk8BUzXifjnQXdmih50tp6B12UqszBmeYr5zBHobZzoPk1z\nr2C5Ln87rKcIjQvYnLgttKiVWrPXUssNkWtPveXNKhkkmwFMpqpJUwTBIlPxw0q2bIGOpAj2kGRS\nRKQx5Qq3XedL2l2H7BWZYUKfzZIG+l3Hg2rgw3bJZMseG3TblBI/v1HZJuLTerQWmhPWRhv8fA+Y\n8qHef4LxWE00AL01Uz1N0yz8x/e0xvwIqZ0O0lCxqY7iuVgXnyIg1DatqqgRmY77ob1H5TamrZ7O\nVWuTqSYwJlsP6adBX9X3BbRW2jUVemn1WaVdW/9dZa9pAGCSm1gWKZjLzqgbjC8iPq1FrW2JJim0\nQuZNfUA2qEcwU2mEwBKlBSXiU4+7DZ2ryVCDo7HB37eh9Rm0e/PcVNZhbhysnwxpNks3zTQv33My\nD1lAFms+PkhqzV5LLTdErp+pJvUsLiLi2V/Qv5uN+6xG5IqfI40Uoa1OPvHpqKQH7Y+Vv7KAbzQ6\n2OynLSISwyLIcEouyqZHRKmBo40e23as1Nb0OU17d0//m1bP17w0/HKAfiZDsNCM0ErItAJidVSI\n4vCjpYepfm2mPcbHSB1OkboLTUUZW1qxQYdcke4krJRWwzI3df2oIKPPz317powwbVZTTbyGjaXE\nO9RS+n9mWlyx+rFMiRJz5XE/5bMmN2HZoik2sRw8JN4H02KJiUGcXepcEXgzNmnO6RzMQ4Dv5n0w\n+Fiff63zTyDR2UrPZ60pAqE4H8HU/PR427xHgIwqMSXcN9O2QQyAl6X/zqRmqqmlllo+omZ3zv2n\nIvLvi64br4vInxeRuyLySyKyLyK/JSI/XRRF/IEnEV3V035RWd2jEaCFhMuSRMWuWIx6w2dPJ2if\na7Qu3VaOwGqygWlUAAAYCElEQVRtutjUBM3p9vLHMa13qpo5GG9Hf6m9S3/euEqM6tMyqACICn6H\nenrEDNK2aVZBvwvql+y2U+ed1VUPVpCr1pWLiLyPCHmeVtfxxICDWIu9A3863tPPfsdP6LCtg6N/\nftv0GGY9fRSoBrvX1tjBeWKi4ND2dwc4TutR5NRAQAmGIdjJtrLOQrY95uRWuQBETPtiRLa7D6G1\nz72ls0I0P3kAzUxwjmENJuCHgCTLHcBtjhQAsDxP5h4U89aRQoJfOlTY7MNnCl9ud/3PgS2dVmgb\nXuya7AqbU6CGn+CtSgNUPJr1EPfa0jFa3obmZfHxovHOuXsi8h+LyBeLovg+UUbDPysif0VEfqEo\nis+IyIWI/OyHnauWWmr55OSjmvGRiHScc5Foo96nIvInROTvYPsvisif+e4Pr5ZaavluyYea8UVR\nPHbO/Tci8lA0A/G/i5rt46IoaCG/L6Wh9hzJtUIpMkGWWQfRMqZNTnX9aRgK5gQWUxnIgRlsUxPN\nS6RhYtIymeNZ476DoE9JBmnALEUVvMF4lE2zleekSbeu7iti+pfhuMRcowGCSwdyzBKzfwW1EINO\nrKMOTBdXBn7OYwThVt7eo2nPTrcMULFZgl4PlVsNtYPjtp7v1V1ftRYFVXtwZB7aSazmLoNeGSZk\n1PAomT0UDAwjdQfeCLWBw2Ltg2erhk5cE/j3e3uTchubOtwb6XfsvffszOP3A1Aus+NvgHRtYYgX\ng7115TwMkBUmwJYj+Hmnry/dycK7I6O72s33SVOvy3ndN6k33hPHTOqp+zv+flihmCC4+TjxrkIP\n7tM5etW1LnRsrbHpE7/Uv9cDujcg6TQ1C0UoHw8b75zbFZGfFJFXROQFEemJyJ/8sOPM8V9yzr3m\nnHstW8w//IBaaqnlD0Q+SoDuXxaRd4qiOBERcc79XRH5MRHZcc5F0O73ReTxVQcXRfFlEfmyiEjn\n7oMiSKqBhwJwThIsJkOAJyIDJ+yxRhzH9PSYeMevVWx3VJBo0QQu/v/2riZGsuusnlv/Vf3fPT3t\nycyEsXF+cCKBrQg5wgtEQISAYJNFIhZesEQiICQUi0WExAYJAVkgJETEAiGCMBGxjAQC4wUrQ4IH\nFDwZe2Ycz4znp6dnuruqu+u/Lot7zrvfq+5JxoxdVZO+n9SqfvVevXfffT/3u993vnM8qYYVLMtG\naPMWVG21AmwDRtN6BhGqgFB/IU/6Z+miJQpR22aKpmzYSugJdNbkmXB/MdYTA5Rs4zJVbh9fvJdt\ncq4RqJvP1YIWfN9EM59ZvAoAeHH0DADgFuGqC3MRQbTbouZ4MZzAImWcVitx9JYiqgAmN2uxI5pM\nMV28FxRqd+ZDqsqCe1pU4xXTyx6r31QXDgAjjm6+cjiddUDt9R2SSFY4ipcr0dVSCjGjwmY1pE0/\nVau2AD7Cjq2HUSFQRrXu1jQiSytdVXi2vv9uNXgCTabp+pusfluJj9c9ss9sLOxxv7ZNoY3+ZLh5\ndjok8DyIo/9BkQw5AmbR08lJdpXdQ1e9XQXwrHOu4ZxzAD4D4A0ArwL4PLd5HsA3H2BfyZIlm5I9\nyJz9NefciwD+CyGL9TrCSP2PAL7unPt9fve1H3g0H0ZDC0GV3NKQoJgSOdiK5oUskUNPUE2R9NN9\nw1PX3sifisQegDjXH2T8doeLVMQZlzHN6KVqcnjdlfxrUzJWNt2hcxPs1nooIvHPMfWMLcvrELDC\nAkSytrJNi4UwumxUY4Dj49WbAIAPL4Q0kKCsS5U4sl+th3rtTyyFbTWvPlWJElNna6FzrrRDWmnD\nEO3t9MOJL7C45OzcNn+zfaitfZ6c5vk2viBGGY3an1i+la1T6u6JheC9rLNW/t5qvGg3DsI8+gIL\nYIqcew/MfLhC2K9gqk2OiF0zshfq+YzxSi2O2mqHOOTEINQZxN+vz4W2SUijVznMc6f9qEimYyS7\nrrOgy9N7KO3zs23uPcabRE2uXVtGJjfA9wXVPFCe3Xv/FQBfGfv6CoCffJDfJ0uWbPo22RJXwmUH\nK4ZdtuqzdUCEUJZM+aOE//Q6q1HAoWdHXYkqSMBgN755VTY6XJFAANk/q/H3AmYI4OHHxBbsNlEG\nmYc0I3tWRitIrIkdNO4yek0QTenA57YFIuxXWQAVYtzrRnROnQwxT9Zuh23MDu4OQwBABR8aUX60\nESPtYpT5xaX/BgCUebCiGRYaZA56qxpkqodmMniPLtHr/TMAYqnr3a5xlWgq2pGw4tDIHm0z6n1q\nLUStxW4DxJLaeQY6npt/EwBwuXcy20bQ10aN83COrDbSrtFVzDSC+JaNIMZji5SG4uj/9HIs+hmy\nH9UexRU+1IiRdkXYr+0wrkHgzp1WPB8xzlYYQ3AdI8dcI7NyxubDSLvxXCM4i9mVg8NDeLHrE1w2\nWbJk6WFPluzY2GSr3hA8cQt4ybxDBt8K2+OYaMBJM531z/tcV6jaYnFRmyjIYWrmGa8ZlfI+jjfL\nctulGCt2GymEAhHoIqJJpX4s44rSc1mNunEpM01tnr+q4IYGeKPXrwKMm5vBZW8vH67keqcegmfn\nd85k66RbdnEruLs6w6vNSKLYJT5bwTNVz+2ZnKhIKUWiaOvAr7WCu7r1dsCAbzNoVizF66HUUoNp\nPWHDeyYwNpQiKhldDgYxciu8v9hfahQ7u2lyoXdEaqnA2Pbh1NkO74dLxdBXrYOwTcdse6MQphgN\nuti94bls3Wo1pCNvtUKAscE02eXmiWwb6an3+/nI68DUJwxYs65rV9oz60DNeGm+S8PP4poyhprw\nqXvJWrGHRCWdLFmyaTDVlPPKk0W+4QbF/CvJak37cd1pQUENj5cCHgW+HfUJmLdhNYwgCn5Zxhyp\nZGbAHR3TtPXwiKx6bAvbDZ+9edVjm3OSKMIgfyzLqiOSlIzpRrJBcZMskDQSK4/xHsSWslAzJYGI\nqTgAaPZrud81CK6pF/MAFCBWuLX6cSQcDPMjmAAnVu5ImutivNHoV1mIwB2N9oKezhvyAP2vdOBC\nIURFV0sRhbleDYHGtyvUTpenZ+6LpeVwPIFp9HndjMKLDdJLEz5sGXdO8BiLteBh1dj3NggoL0RB\nvy6BYhUDAOqryo77zqfM8vBvMfBYhMyQaeNBX9V/eeg3EILLSf4pWbJkEx7Zi0BvyaO/buR5CFVU\ngcFAMkVmPl2keMCIb+zVlfC2rZg5oiCLKhjpbEW2kUx/nLXzPYpAOlNcon0fpa0tK4yN/mLQKfTj\nbxQfqOwfLshR4U3GlKPa6qXYDoFyRusULjgRUjzPbVyJ+2Ea6FQ5jHqlwtls3fndMH/vctQUm6kV\nTjhN7rpPzgWE88lSWPe9npmHjhVGP9nYzP4/2whewmuVcwCi3JGFm6rIRnaX4oWj0eHxRQCVumF3\nFdx2rUjAig8dc6W9nm3zxnYA0wha65r03Iw3tVsK90V/3BvpxeVWO7RbHkupGM+9wFF2XOTBjv7L\nBOFkjLx7oe+7jeiyScC0OEfPoHTYc5XnJxvUzYKm8/IGeV/VduN+qs0RxuqXcpZG9mTJjomlhz1Z\nsmNiUxGJyLnKPdUgM/VGF2xUib5YgxVb+3t5d2vvILqNqmkWsWFl2wLOuY1TyZDPHRMAHCmtHTHp\ncq0KJogn7XS5So6BqaOEJAiqyhBx1kp08aURZ1F6cs+GrA672Qpu60utiE6Tu9iofTQcw9JEc53S\nPmtL4SCWHlnpue4oj7vf6i8c2uZmJwSmbH375d3g7m/thtTXIq/P0AQq5doKuaYKu4pB+yno92Fi\n6z86dztbp2mEauVv9EPq8GwtVv9dqoZ2DFbC+d/V8U2wt07c+zK10xVw7HTiuRd5023MGxIFmkgk\nD8ZqFKTFDkQ9O6Xu2ovh01bcjUgRLu2+zlLsz5Jce1bGCcVpo7JKG0tLUNRo9v4q9lwK0CVLlmzS\noJohULvn4EyZV1bd9VhYFl20fTvvt5gqaoW3674AL2bfJeKLBzylviXM57/VkyENIzDMwBILKp23\nMMb0Yqq0RDMNgWqUirPpOYFqiIkvb8bz6LD+XoNbhmExb/AMu+KVWmFQz3hDQ7Z7R6SHlulGQA62\nf4f9WDVBp7sMMpUZyWosBdekb67LTt8yZQIjA6o5Pb+b+1xhHbxNzymNt1ENQbcuXZaS4REXBbPW\nHRjKn11Gp25S90gj/bxxo1QtpyCkGGJsENCPDXWSYSqYdiggp8Bl23hBddJjq0ZhZy/0y24z9o/A\nRH1WsqkWo2VJPxVg02I5Hn/A+1oCt0VWvdlAo56TcbPOmXcPX8+eLFmyHwKb7JzdBVaXoUE1qo58\nHDZbX4z113XWTTeL4W0qSSErNniwn1d+UO07ADim8wT+UG3yymIEeIjiV3PeEmuj7du5SxadYlP4\nW2cPGb5i+mSHMkclIzGleXxGUy2mGzuIKq1HwYRRLXyWTZqwL3pjMplU7xhPaSw92CXU+FonXmoB\nXu4QzFI+Qs9KgB2N0N3R4VtFoBx5CHZev1rhvvmdPndNPmmf8Njdbvhu2XDYnb8bKA3nK13uOxzj\n3Fycs795O8Qzuq28vrkVo++7fOos44Lbi16EPMVr+yu58wJirGOvkz+GZZoRC1CLv+vvHGaRUSxp\nOI6aAjIvbrDA/pNzZoA/GeSbsaQB5+p2ZB+VkUb2ZMmSTQEu60tjED+2QLDZIdd1q0bulnMivZXb\nXUN1Qxtt8zuNgOYNJ3GJfj3sc0Q448AIAAokcXIpgDgksXS+F0lz9Tbv1ZlB4KjvDUx0WOMIoii+\nrXERlFeS0ft6Ox+uXtDvC+2wv+6+OWdmDlSs012NXkyxnYde6vijA3Opebzru2E+/O+1jwEArrTW\nsk3ElKr685IBMFX5/4DrOkuMpfRjG6+2AoRVo+Qer5llbxEX3nAv/P6dZVOsc48egK4nvZEb61Fk\norfZ4DkXcudssyODRjjv248pzUHGl9uxP3r0kC4QnFOux/uiSu6EfUqNOXp8NhZQFIcC5Zx1zdx+\nvC8EvBodwVakXYlZWcuV1uFtpLBVJItN0dwW/TmHI3QrYxvuvypZsmQ/TJYe9mTJjolNRcXVphQy\nTW5533RfLWXwUJRA/BxQx1qBNsBgjQXOMbRDA6Y5FtdC0Eguu7THAWCbgIYOMeUZ/nwl4rVFM7TE\nwJ7cNwX3gOimysnrmXWdsUBSj4HD+npsRxYg1JSBul/LCzF41WPqrc2UpPfmXElzrd5TP5RMKnPQ\nDL5fcy+c8yXWxUtTHYhVamvsI+t+63r0SW8sAQeRWwKxNl19LWELG/wqZDTNTC8OrG/LT23Ovsph\n3MeDUdqfwZj3l6gwSz07AWiaJUOhxc3nV8K5dg3gRtTXqrAs0A0/ae6LU6w7EE9Aq6Q6AJMuVf1E\nBpiJ6+oroW3tWrgehX2dYxyLNdWr7HKqwttBtOYA0J/3OCKOmlka2ZMlOyY2+Xr2CtA7ZSqi+MYr\nzJF0jwqtJTNq1xlI0yi3vmpEz2mboxC4kUfQNyqdenPvEW6rCqS9uZgDHN4No9M2BShulsiI0jKU\nwwz0dQr5IIvFbUj0otDg2LplpZnYHJ5zhcIWQ6M6moEnBATibvYWYluVsiuQXcdZwE2dASSOQKOe\niBJjG6V2WmI67sYaYZrm1S/WlH0pjDbiDka1MVYfHutqYzXbxtP7cuwPT6YWW82okVjttyNhgSN5\ngRBlQZU7zSj/tPguwU26Dvy0nqOEFvrNcH+o2G1u21Qq8rwVaCx14rreMj1FLqtm/1Y7BhMHG/S0\n5AWwz50J3PqxQCNMpWTmSdArVcC3u2aAP+08F4KqKrurJlDYdrlzH7c0sidLdkxs4qAauMi+AgBe\nXGPUstZLdckUGogPrbQcRomNBuvZi3Fev1YP863Ld8L808755UecXg9wyB3WLy8aNpf9ufBWbpMj\nTamV3p6Zj7PZGpnHgQ7hfDhK0XuABfcM8+wiwpdI3goAPNlWCmTeGc2H89CcEwCqrBVXPf+BSUVK\nW7y9S0+gN5YKREx36vhHymGRGUVsPAL3AEBpMVwr0SJ3WIRUqRlpJqasxC/QLeaFEACgwmveL0lU\nwRxfnIQnyUHA67G+GufKt05wdOVoWWnQAzRpypUTIY4w6uWv68GB8djKEoBgStHEDpbJrFNmyk0x\nh4qBH//YSijgubMQYjrv7IR2tfYigGiOwBvFcLqbEUnl6VXW7rKIi6GP8l68L9oneO/wMogJyVKu\nN58YpTl7smTJpjBnH9Q9CiU7qaKgI6GBQwJGbIRbJa37B+ENKOjiYj1iUZscrTuCQZoRREUHkh5S\nqexeM755M2CMoq51TYzNCYjMpmbmnciDhDR6l5oCvsR18gDkGQhoUTYhiMI2swEcWR2zA/s+jgSC\nd2aAk66By85zRFVRCLexTLydAuMIHO0FLRas2J5TNqIbuK5Gx/Y+GWK2KNpYN8IcghJ3dYzDnH4i\n4NX9YKWrHWMeBY6oitQPDUS6xgIeyT3VqodLS7W9PIwePZ2CiR3IoSiNSTwBQJ0FMLp3xO1Xtmw2\nHG7XqiHbM6JU071qzG4I9rvTCffc1kq8ZoLb7jRC7KanaLwthMlg0KqooecZQxgYzQ9zBWTjlkb2\nZMmOiaWHPVmyY2KTBdWMgnjD4LpxnwUemWd11E5wYbYPYmpDVUwSTugh/H6zHH0YpTbqu8QXW+wF\nPXIF4bQqh0lnO0TeP6qErnG2hxSgU405q5SUkgNiYE17zh1DwCHuR1pzRUN7rWNkKrSqpDIBtqw2\nuiRXP64byN3Wd5qdmDRQdStPvljQOnsIxdocwSzzBpQjnTJ9MF1YMCSO4yq4hVbBLoZtpNUud9UM\nPXJbR8VqbtWdpinzUnu5zyZTidX5GHhVOq9I97aYubnRDR9yGiDiSm/qCNpzAtUQQFRk+tZsc3Vu\nldtw6sJj2IBjQfryBDkNze8PBHDntE4CJ/Ye1rRuvObBkpW6yihVvSVLlgxw3t9/Qv++H8y5OwD2\nAWxN7KDvj53Ao9dm4NFsd2rzw9mPeO/Xj1ox0YcdAJxz3/Lef2qiB31IexTbDDya7U5t/uAsufHJ\nkh0TSw97smTHxKbxsP/5FI75sPYothl4NNud2vwB2cTn7MmSJZuOJTc+WbJjYulhT5bsmNjEHnbn\n3Gedcxedc5ecc1+e1HHfqznnzjrnXnXOveGc+1/n3Jf4/apz7l+cc2/xc+UH7WvS5pwrOuded869\nzOXHnXOvsc//1jl3mJZ3iuacW3bOveic+65z7oJz7tOPSD//Fu+N7zjn/sY5V5v1vgYm9LA754oA\n/hTALwB4CsAXnXNPTeLY/w8bAPht7/1TAJ4F8Ots65cBvOK9/wiAV7g8a/YlABfM8h8A+GPv/ZMA\ntgH82lRadX/7KoB/8t5/HMCPI7R9pvvZOXcawG8A+JT3/pMI6OsvYPb7GvDef+B/AD4N4J/N8gsA\nXpjEsd+Htn8TwM8BuAjgFL87BeDitNs21s4zCA/HzwB4GQElvQWgdNQ1mPYfgCUAb4NBYvP9rPfz\naQDXAKwi1Ja8DODnZ7mv9TcpN14dJLvO72banHPnADwN4DUAG977m1x1C8DGlJp1P/sTAL+DWFay\nBmDHe6+Sllnr88cB3AHwl5x6/IVzbg4z3s/e+3cB/CGAqwBuAtgF8G3Mdl8DSAG6+5pzbh7A3wP4\nTe99067z4fU9MzlL59wvAdj03n972m15D1YC8AyAP/PeP41QM5Fz2WetnwGAMYRfQXhZfQjAHIDP\nTrVRD2iTetjfBXDWLJ/hdzNpzrkywoP+1977b/Dr2865U1x/CsDmtNp3hP0UgF92zn0PwNcRXPmv\nAlh2LivSnbU+vw7guvf+NS6/iPDwz3I/A8DPAnjbe3/He98H8A2E/p/lvgYwuYf9PwF8hBHLCkJA\n46UJHfs9mXPOAfgagAve+z8yq14C8Dz/fx5hLj8T5r1/wXt/xnt/DqFv/817/6sAXgXweW42a22+\nBeCac+5j/OozAN7ADPcz7SqAZ51zDd4ravfM9nVmEwxsfA7AmwAuA/jdaQcrvk87n0NwHf8HwHn+\nfQ5hDvwKgLcA/CuA1Wm39T7t/2kAL/P/JwD8B4BLAP4OQHXa7Rtr608A+Bb7+h8ArDwK/Qzg9wB8\nF8B3APwVgOqs97X3PsFlkyU7LpYCdMmSHRNLD3uyZMfE0sOeLNkxsfSwJ0t2TCw97MmSHRNLD3uy\nZMfE0sOeLNkxsf8DZWkCdHQcW2wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "generator = build_generator(SEED_SIZE, IMAGE_CHANNELS)\n",
    "\n",
    "noise = tf.random.normal([1, SEED_SIZE])\n",
    "generated_image = generator(noise, training=False)\n",
    "\n",
    "plt.imshow(generated_image[0, :, :, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nlxxEHDIOqjW"
   },
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "id": "LOnTxIXnyeEQ",
    "outputId": "ed8e14da-88d9-4d02-9338-07bb71eadb7e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([[0.50032634]], shape=(1, 1), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "image_shape = (GENERATE_SQUARE,GENERATE_SQUARE,IMAGE_CHANNELS)\n",
    "\n",
    "discriminator = build_discriminator(image_shape)\n",
    "decision = discriminator(generated_image)\n",
    "print (decision)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-ChOo3D1OsVc"
   },
   "source": [
    "Loss functions must be developed that allow the generator and discriminator to be trained in an adversarial way.  Because these two neural networks are being trained independently they must be trained in two separate passes.  This requires two separate loss functions and also two separate updates to the gradients.  When the discriminator's gradients are applied to decrease the discriminator's loss it is important that only the discriminator's weights are update.  It is not fair, nor will it produce good results, to adversarially damage the weights of the generator to help the discriminator.  A simple backpropagation would do this.  It would simultaneously affect the weights of both generator and discriminator to lower whatever loss it was assigned to lower.\n",
    "\n",
    "Figure 7.TDIS shows how the discriminator is trained.\n",
    "\n",
    "**Figure 7.TDIS: Training the Discriminator**\n",
    "![Training the Discriminator](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/gan_fig_2.png \"Training the Discriminator\")\n",
    "\n",
    "Here a training set is generated with an equal number of real and fake images.  The real images are randomly sampled (chosen) from the training data.  An equal number of random images are generated from random seeds.  For the discriminator training set, the $x$ contains the input images and the $y$ contains a value of 1 for real images and 0 for generated ones.\n",
    "\n",
    "Likewise, the Figure 7.TGEN shows how the generator is trained.\n",
    "\n",
    "**Figure 7.TGEN: Training the Generator**\n",
    "![Training the Generator](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/gan_fig_3.png \"Training the Generator\")\n",
    "\n",
    "For the generator training set, the $x$ contains the random seeds to generate images and the $y$ always contains the value of 1, because the optimal is for the generator to have generated such good images that the discriminiator was fooled into assigning them a probability near 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gBaP98zAySJV"
   },
   "outputs": [],
   "source": [
    "# This method returns a helper function to compute cross entropy loss\n",
    "cross_entropy = tf.keras.losses.BinaryCrossentropy()\n",
    "\n",
    "def discriminator_loss(real_output, fake_output):\n",
    "    real_loss = cross_entropy(tf.ones_like(real_output), real_output)\n",
    "    fake_loss = cross_entropy(tf.zeros_like(fake_output), fake_output)\n",
    "    total_loss = real_loss + fake_loss\n",
    "    return total_loss\n",
    "\n",
    "def generator_loss(fake_output):\n",
    "    return cross_entropy(tf.ones_like(fake_output), fake_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gIlZvHGAxbWf"
   },
   "source": [
    "Both the generator and discriminator use Adam and the same learning rate and momentum.  This does not need to be the case.  If you use a **GENERATE_RES** greater than 3 you may need to tune these learning rates, as well as other training and hyperparameters.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "79UDhOCa0R4h"
   },
   "outputs": [],
   "source": [
    "generator_optimizer = tf.keras.optimizers.Adam(1.5e-4,0.5)\n",
    "discriminator_optimizer = tf.keras.optimizers.Adam(1.5e-4,0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "frCpCNn8yRcM"
   },
   "source": [
    "The following function is where most of the training takes place for both the discriminator and the generator.  This function was based on the GAN provided by the [TensorFlow Keras exmples](https://www.tensorflow.org/tutorials/generative/dcgan) documentation.  The first thing you should notice about this function is that it is annotated with the **tf.function** annotation.  This causes the function to be precompiled and improves performance.\n",
    "\n",
    "This function trans differently than the code we previously saw for training.  This code makes use of **GradientTape** to allow the discriminator and generator to be trained together, yet separately.  \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uzyh-LqU0j5d"
   },
   "outputs": [],
   "source": [
    "# Notice the use of `tf.function`\n",
    "# This annotation causes the function to be \"compiled\".\n",
    "@tf.function\n",
    "def train_step(images):\n",
    "  seed = tf.random.normal([BATCH_SIZE, SEED_SIZE])\n",
    "\n",
    "  with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:\n",
    "    generated_images = generator(seed, training=True)\n",
    "\n",
    "    real_output = discriminator(images, training=True)\n",
    "    fake_output = discriminator(generated_images, training=True)\n",
    "\n",
    "    gen_loss = generator_loss(fake_output)\n",
    "    disc_loss = discriminator_loss(real_output, fake_output)\n",
    "    \n",
    "\n",
    "    gradients_of_generator = gen_tape.gradient(\\\n",
    "        gen_loss, generator.trainable_variables)\n",
    "    gradients_of_discriminator = disc_tape.gradient(\\\n",
    "        disc_loss, discriminator.trainable_variables)\n",
    "\n",
    "    generator_optimizer.apply_gradients(zip(\n",
    "        gradients_of_generator, generator.trainable_variables))\n",
    "    discriminator_optimizer.apply_gradients(zip(\n",
    "        gradients_of_discriminator, \n",
    "        discriminator.trainable_variables))\n",
    "  return gen_loss,disc_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NjrRgDR10lSF"
   },
   "outputs": [],
   "source": [
    "def train(dataset, epochs):\n",
    "  fixed_seed = np.random.normal(0, 1, (PREVIEW_ROWS * PREVIEW_COLS, \n",
    "                                       SEED_SIZE))\n",
    "  start = time.time()\n",
    "\n",
    "  for epoch in range(epochs):\n",
    "    epoch_start = time.time()\n",
    "\n",
    "    gen_loss_list = []\n",
    "    disc_loss_list = []\n",
    "\n",
    "    for image_batch in dataset:\n",
    "      t = train_step(image_batch)\n",
    "      gen_loss_list.append(t[0])\n",
    "      disc_loss_list.append(t[1])\n",
    "\n",
    "    g_loss = sum(gen_loss_list) / len(gen_loss_list)\n",
    "    d_loss = sum(disc_loss_list) / len(disc_loss_list)\n",
    "\n",
    "    epoch_elapsed = time.time()-epoch_start\n",
    "    print (f'Epoch {epoch+1}, gen loss={g_loss},disc loss={d_loss},'\\\n",
    "           f' {hms_string(epoch_elapsed)}')\n",
    "    save_images(epoch,fixed_seed)\n",
    "\n",
    "  elapsed = time.time()-start\n",
    "  print (f'Training time: {hms_string(elapsed)}')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 884
    },
    "id": "vWmEHprD0t1V",
    "outputId": "7eaaf566-6cc0-4878-e26d-4362c58e0084"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1, gen loss=0.6737478375434875,disc loss=1.2876468896865845, 0:01:33.86\n",
      "Epoch 2, gen loss=0.6754337549209595,disc loss=1.2804691791534424, 0:01:26.96\n",
      "Epoch 3, gen loss=0.6792119741439819,disc loss=1.2002451419830322, 0:01:27.23\n",
      "Epoch 4, gen loss=0.6704637408256531,disc loss=1.1011184453964233, 0:01:26.74\n",
      "Epoch 5, gen loss=0.6726831197738647,disc loss=1.0789912939071655, 0:01:26.87\n",
      "Epoch 6, gen loss=0.6760282516479492,disc loss=1.086405873298645, 0:01:26.64\n",
      "Epoch 7, gen loss=0.6668657660484314,disc loss=1.0990564823150635, 0:01:26.75\n",
      "Epoch 8, gen loss=0.6794514060020447,disc loss=1.0572694540023804, 0:01:26.72\n",
      "Epoch 9, gen loss=0.667030930519104,disc loss=1.099564552307129, 0:01:26.83\n",
      "Epoch 10, gen loss=0.6715224385261536,disc loss=1.0793499946594238, 0:01:26.66\n",
      "Epoch 11, gen loss=0.6730715036392212,disc loss=1.078121304512024, 0:01:26.76\n",
      "Epoch 12, gen loss=0.6647288203239441,disc loss=1.1045489311218262, 0:01:26.72\n",
      "Epoch 13, gen loss=0.6678280234336853,disc loss=1.0965650081634521, 0:01:26.85\n",
      "Epoch 14, gen loss=0.6659991145133972,disc loss=1.0995577573776245, 0:01:26.63\n",
      "Epoch 15, gen loss=0.6716247797012329,disc loss=1.0771788358688354, 0:01:26.71\n",
      "Epoch 16, gen loss=0.6729282140731812,disc loss=1.070194125175476, 0:01:26.70\n",
      "Epoch 17, gen loss=0.5819647312164307,disc loss=1.2202180624008179, 0:01:26.77\n",
      "Epoch 18, gen loss=0.6757639646530151,disc loss=1.065081238746643, 0:01:26.61\n",
      "Epoch 19, gen loss=0.67451012134552,disc loss=1.0673273801803589, 0:01:26.69\n",
      "Epoch 20, gen loss=0.6766645908355713,disc loss=1.0598846673965454, 0:01:26.77\n",
      "Epoch 21, gen loss=0.6780093908309937,disc loss=1.0609248876571655, 0:01:26.69\n",
      "Epoch 22, gen loss=0.6759778261184692,disc loss=1.0606046915054321, 0:01:26.82\n",
      "Epoch 23, gen loss=0.6769225001335144,disc loss=1.054925799369812, 0:01:26.65\n",
      "Epoch 24, gen loss=0.6781579852104187,disc loss=1.0544047355651855, 0:01:26.73\n",
      "Epoch 25, gen loss=0.676042377948761,disc loss=1.0546783208847046, 0:01:26.75\n",
      "Epoch 26, gen loss=0.677901029586792,disc loss=1.052791953086853, 0:01:26.80\n",
      "Epoch 27, gen loss=0.681058406829834,disc loss=1.0429731607437134, 0:01:26.64\n",
      "Epoch 28, gen loss=0.6791036128997803,disc loss=1.0511810779571533, 0:01:26.73\n",
      "Epoch 29, gen loss=0.6798704266548157,disc loss=1.0473483800888062, 0:01:26.71\n",
      "Epoch 30, gen loss=0.6804633736610413,disc loss=1.0467922687530518, 0:01:26.79\n",
      "Epoch 31, gen loss=0.6794509291648865,disc loss=1.0476592779159546, 0:01:26.77\n",
      "Epoch 32, gen loss=0.6806340217590332,disc loss=1.0451370477676392, 0:01:26.61\n",
      "Epoch 33, gen loss=0.6815114617347717,disc loss=1.0450003147125244, 0:01:26.71\n",
      "Epoch 34, gen loss=0.6811343431472778,disc loss=1.0472784042358398, 0:01:26.67\n",
      "Epoch 35, gen loss=0.6814053058624268,disc loss=1.044533371925354, 0:01:26.79\n",
      "Epoch 36, gen loss=0.6829107403755188,disc loss=1.0392040014266968, 0:01:26.59\n",
      "Epoch 37, gen loss=0.68181312084198,disc loss=1.0412304401397705, 0:01:26.70\n",
      "Epoch 38, gen loss=0.6876026391983032,disc loss=1.023917555809021, 0:01:26.56\n",
      "Epoch 39, gen loss=0.6833422780036926,disc loss=1.0362210273742676, 0:01:26.75\n",
      "Epoch 40, gen loss=0.6848627328872681,disc loss=1.0325196981430054, 0:01:26.57\n",
      "Epoch 41, gen loss=0.6868091821670532,disc loss=1.0275564193725586, 0:01:26.60\n",
      "Epoch 42, gen loss=0.6845845580101013,disc loss=1.037258505821228, 0:01:26.65\n",
      "Epoch 43, gen loss=0.6817746758460999,disc loss=1.0453448295593262, 0:01:26.80\n",
      "Epoch 44, gen loss=0.6872225403785706,disc loss=1.0249252319335938, 0:01:26.50\n",
      "Epoch 45, gen loss=0.6867306232452393,disc loss=1.0273733139038086, 0:01:26.66\n",
      "Epoch 46, gen loss=0.6831635236740112,disc loss=1.041325330734253, 0:01:26.63\n",
      "Epoch 47, gen loss=0.6846590042114258,disc loss=1.033777117729187, 0:01:26.76\n",
      "Epoch 48, gen loss=0.683456301689148,disc loss=1.0370010137557983, 0:01:26.63\n",
      "Epoch 49, gen loss=0.6844978332519531,disc loss=1.034816861152649, 0:01:26.71\n",
      "Epoch 50, gen loss=0.6809002757072449,disc loss=1.0432080030441284, 0:01:26.67\n",
      "Training time: 1:12:52.22\n"
     ]
    }
   ],
   "source": [
    "train(train_dataset, EPOCHS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Mvir4efcLlwi"
   },
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "15Hia_feD9sm"
   },
   "outputs": [],
   "source": [
    "generator.save(os.path.join(DATA_PATH,\"face_generator.h5\"))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "name": "t81_558_class_07_2_Keras_gan.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.7 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}