t81_558_class_03_2_keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bLEEW13uCtiJ"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_2_keras.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5oX6K_-JCtiL"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 3: Introduction to TensorFlow**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FzlMdsudCtiL"
   },
   "source": [
    "# Module 3 Material\n",
    "\n",
    "* Part 3.1: Deep Learning and Neural Network Introduction [[Video]](https://www.youtube.com/watch?v=zYnI4iWRmpc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_1_neural_net.ipynb)\n",
    "* **Part 3.2: Introduction to Tensorflow and Keras** [[Video]](https://www.youtube.com/watch?v=PsE73jk55cE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_2_keras.ipynb)\n",
    "* Part 3.3: Saving and Loading a Keras Neural Network [[Video]](https://www.youtube.com/watch?v=-9QfbGM1qGw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_3_save_load.ipynb)\n",
    "* Part 3.4: Early Stopping in Keras to Prevent Overfitting [[Video]](https://www.youtube.com/watch?v=m1LNunuI2fk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_4_early_stop.ipynb)\n",
    "* Part 3.5: Extracting Weights and Manual Calculation [[Video]](https://www.youtube.com/watch?v=7PWgx16kH8s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_5_weights.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CpVJpj2DCtiM"
   },
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wU1AMzx8CtiM",
    "outputId": "9783a6d5-c6b8-433a-b832-1f83ea7d8007"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pOY9MHVrCtiM"
   },
   "source": [
    "# Part 3.2: Introduction to Tensorflow and Keras\n",
    "\n",
    "TensorFlow is an open-source software library for machine learning in various kinds of perceptual and language understanding tasks. It is currently used for both research and production by different teams in many commercial Google products, such as speech recognition, Gmail, Google Photos, and search, many of which had previously used its predecessor DistBelief. TensorFlow was originally developed by the Google Brain team for Google's research and production purposes and later released under the Apache 2.0 open source license on November 9, 2015.\n",
    "\n",
    "* [TensorFlow Homepage](https://www.tensorflow.org/)\n",
    "* [TensorFlow GitHib](https://github.com/tensorflow/tensorflow)\n",
    "* [TensorFlow Google Groups Support](https://groups.google.com/forum/#!forum/tensorflow)\n",
    "* [TensorFlow Google Groups Developer Discussion](https://groups.google.com/a/tensorflow.org/forum/#!forum/discuss)\n",
    "* [TensorFlow FAQ](https://www.tensorflow.org/resources/faq)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1Zg-zNx0CtiN"
   },
   "source": [
    "## Why TensorFlow\n",
    "\n",
    "* Supported by Google\n",
    "* Works well on Windows, Linux, and Mac\n",
    "* Excellent GPU support\n",
    "* Python is an easy to learn programming language\n",
    "* Python is extremely popular in the data science community\n",
    "\n",
    "## Deep Learning Tools\n",
    "TensorFlow is not the only game in town. The biggest competitor to TensorFlow/Keras is PyTorch. Listed below are some of the deep learning toolkits actively being supported:\n",
    "\n",
    "* **[TensorFlow](https://www.tensorflow.org/)** - Google's deep learning API.  The focus of this class, along with Keras.\n",
    "* **[Keras](https://keras.io/)** - Also by Google, higher level framework that allows the use of TensorFlow, MXNet and Theano interchangeably.\n",
    "* **[PyTorch](https://pytorch.org/)** - PyTorch is an open source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing. It is primarily developed by Facebook's AI Research lab. \n",
    "\n",
    "Other deep learning tools:\n",
    "\n",
    "* **[MXNet](https://mxnet.incubator.apache.org/)** Apache foundation's deep learning API. Can be used through Keras.\n",
    "* **[Torch](http://torch.ch/)** is used by Google DeepMind, the Facebook AI Research Group, IBM, Yandex and the Idiap Research Institute.  It has been used for some of the most advanced deep learning projects in the world.  However, it requires the [LUA](https://en.wikipedia.org/wiki/Lua_(programming_language))** programming language.  It is very advanced, but it is not mainstream.  I have not worked with Torch (yet!).\n",
    "* **[PaddlePaddle](https://github.com/baidu/Paddle)** - [Baidu](http://www.baidu.com/)'s deep learning API.\n",
    "* **[Deeplearning4J](http://deeplearning4j.org/)** - Java based. Supports all major platforms. GPU support in Java!\n",
    "* **[Computational Network Toolkit (CNTK)](https://github.com/Microsoft/CNTK)** - Microsoft.  Support for Windows/Linux, command line only.  Bindings for predictions for C#/Python. GPU support.\n",
    "* **[H2O](http://www.h2o.ai/)** - Java based.  Supports all major platforms.  Limited support for computer vision. No GPU support.\n",
    "\n",
    "In my opinion, the two primary Python libraries for deep learning are PyTorch and Keras. Generally, PyTorch requires more lines of code to perform the deep learning applications presented in this course.  This trait of PyTorch gives Keras an easier learning curve than PyTorch.  However, if you are creating entirely new neural network structures, in a research setting, PyTorch can make for easier access to some of the low-level internals of deep learning.\n",
    "\n",
    "## Using TensorFlow Directly\n",
    "\n",
    "Most of the time in the course, we will communicate with TensorFlow using Keras [[Cite:franccois2017deep]](https://www.manning.com/books/deep-learning-with-python), which allows you to specify the number of hidden layers and create the neural network.  TensorFlow is a low-level mathematics API, similar to [Numpy](http://www.numpy.org/).  However, unlike Numpy, TensorFlow is built for deep learning. TensorFlow compiles these compute graphs into highly efficient C++/[CUDA](https://en.wikipedia.org/wiki/CUDA) code.\n",
    "\n",
    "### TensorFlow Linear Algebra Examples\n",
    "\n",
    "TensorFlow is a library for linear algebra.  Keras is a higher-level abstraction for neural networks that you build upon TensorFlow.  In this section, I will demonstrate some basic linear algebra that employs TensorFlow directly and does not make use of Keras.  First, we will see how to multiply a row and column matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bkh9pHqPCtiO",
    "outputId": "c2b05474-0644-4c5c-c7fc-fb94e1e6a69d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([[12.]], shape=(1, 1), dtype=float32)\n",
      "12.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "# Create a Constant op that produces a 1x2 matrix.  The op is\n",
    "# added as a node to the default graph.\n",
    "#\n",
    "# The value returned by the constructor represents the output\n",
    "# of the Constant op.\n",
    "matrix1 = tf.constant([[3., 3.]])\n",
    "\n",
    "# Create another Constant that produces a 2x1 matrix.\n",
    "matrix2 = tf.constant([[2.],[2.]])\n",
    "\n",
    "# Create a Matmul op that takes 'matrix1' and 'matrix2' as inputs.\n",
    "# The returned value, 'product', represents the result of the matrix\n",
    "# multiplication.\n",
    "product = tf.matmul(matrix1, matrix2)\n",
    "\n",
    "print(product)\n",
    "print(float(product))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pMrXqwskCtiP"
   },
   "source": [
    "This example multiplied two TensorFlow constant tensors.  Next, we will see how to subtract a constant from a variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "qKyFCaoTCtiP",
    "outputId": "3c75a8cf-8d1c-48df-fb71-f603053cf616"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([-2. -1.], shape=(2,), dtype=float32)\n",
      "[-2. -1.]\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "x = tf.Variable([1.0, 2.0])\n",
    "a = tf.constant([3.0, 3.0])\n",
    "\n",
    "# Add an op to subtract 'a' from 'x'.  Run it and print the result\n",
    "sub = tf.subtract(x, a)\n",
    "print(sub)\n",
    "print(sub.numpy())\n",
    "# ==> [-2. -1.]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "epQ9LrM6CtiP"
   },
   "source": [
    "Of course, variables are only useful if their values can be changed.  The program can accomplish this change in value by calling the assign function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vtnW5aU-CtiP",
    "outputId": "77d3ae14-dfac-4877-dd7f-d30ae9c9751d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2,) dtype=float32, numpy=array([4., 6.], dtype=float32)>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.assign([4.0, 6.0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zHwJP8MjCtiQ"
   },
   "source": [
    "The program can now perform the subtraction with this new value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0e26Fe-GCtiQ",
    "outputId": "979cf4e5-4d6d-45e2-c7a7-c12e6b169baf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([1. 3.], shape=(2,), dtype=float32)\n",
      "[1. 3.]\n"
     ]
    }
   ],
   "source": [
    "sub = tf.subtract(x, a)\n",
    "print(sub)\n",
    "print(sub.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Rq61YFBYCtiQ"
   },
   "source": [
    "In the next section, we will see a TensorFlow example that has nothing to do with neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iMVIS9pBCtiQ"
   },
   "source": [
    "### TensorFlow Mandelbrot Set Example\n",
    "\n",
    "Next, we examine another example where we use TensorFlow directly.  To demonstrate that TensorFlow is mathematical and does not only provide neural networks, we will also first use it for a non-machine learning rendering task. The code presented here is capable of rendering a [Mandelbrot set](https://en.wikipedia.org/wiki/Mandelbrot_set). Note, I based this example on a Mandelbrot example that I found [here]( https://chromium.googlesource.com/external/github.com/tensorflow/tensorflow/+/r0.10/tensorflow/g3doc/tutorials/mandelbrot/index.md). I've updated the code slightly to comply with current versions of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 537
    },
    "id": "v4ll1OLwCtiQ",
    "outputId": "e0f9479e-7bd7-4c4f-f8dc-b658a72ee747"
   },
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAIIAlgDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDyuiiivROUKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiilAJOAKAEoxnpUixf3j+VSABegp2Jc0RrET14p4RV7U6ighybCiiimSFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAIQCMEVG0X938qlopDTaK5BBwRSVYIDdRUTRkcjkUrGikmMooooKCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACjGelPWMnrwKlVQo4osS5JEaxH+LipAABgClopmbbYUUUUxBRRRQAUUUUAFFFFABRRRQAUUoBJwKeEA6800rickhgBPQU4R+pqSiqUUZubG7Fpdo9BTgpNLsPtVqD7EOfmM2j0FIUWnkEdaSk0NSZGYz25phBHWp6CAetS4lKb6kFFPZO4/KmVDVjRNPYKKKKBhRRRQAUUUUAMaMNyODURBU4NWKQgMMGlYpSsV6KcyFfpTaRqncKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKkWLu35UCbSGKpbpUyoF9zTsY6UU7Gbk2FFFFMkKKKKACiiigAooooAKKKKACiigDJwKACnKmeT0pypjk9afVKPczlPsAAHSiiirMwp6r3NIoyafWkI31IlLoRzS+SudhYdzkYH1qQFWGQRilYBgQRkGsq4jFvMiRsyRyHaQOle/h8LRrU7Ws1+JjFcztfU01Kuu5CGHqKQqD7UseAgA6CnEV5+Kw6hNqOwRmRFSKSpaQqDXA6fY2U+5HTWUN9aeQRSVm10ZafVEJBBwaSpiARg1GylfpWbjY2jK42iiikUFFFFABRRRQAEAjBqF028jpU1FIaditRT3TbyOlMpGydwooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAoooxnpQAU5ULfT1p6x45b8qkp2Ic+wiqF6UtFFMzCiiigAooooAKKKKACiiigAooooAKKKeqZ5NCVxNpDVUsalVQopelFaJWMpSbCiiimSFAGTRT1GOe9VGN2JuwoGBSjrRSiumEbsxkxay9Rj3Oj5fjrjoPc9q1Kq3FtHMdzKCcY5r2sFs1e1+vYinNQmmylY3jea/muyqBwsi7SPqK1gciufmh2XKxoM91QnAzWnb3QeJS3ykjoa66mHdRckneS38/M2r01pOGzL1RySLGVBB+ZtowKpy6kkblCCeM5zgVCss11dqUUIYgThj+HI7VzRwMOZxm/lfUzjSna70RqEVGRj6VIm/wAtfMxvwN23pn2oIyMV4demlJoqEiKgjIwaCMHFFchsRMmOR0ptT1Gydx+VQ49jSM+jGUUUVJoFFFFABRRRQAVC6beR0qaggEYNIaditRTnXafam0jbcKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiipFi7t+VAm0hqoW+lTKoXpS0UzNybCiiimSFFFFABRRRQAUUUUAFFFFABRRRQAUoBPSnKnc/lUgGBgVSiQ522GqgHuadRRV2M27hRRRQIKKAM9KkC4+tVGLYm7CKvc06iitkraIybuAFOoorrpU3sjKUgqNuhpxaq886xjlgK9jC0XfQhJt6FK+jDgnJB5H15qiJN4LOWCk4YL379T+P6VLc3DSKAFPzjK85yM4/pTJkRLdHAwH6JnJVu+fwx2Fb4qtBSgoPXun0vb+ux6tGLjG0iW3tfPbc6qF/ugYrWtreOFQEXGKoWVwpAXOSOuB0rUQ9K0xUfZw5YbHDiJzbs9h9Np1Ia+brx1IgxjDIplS0xhjntXDOPU3i+g2iiisyxrJu571ERg4NT0jKGFS43LjK25DRSlSp5pKg1CiiigAooooARgGGDUBBBwasUyRdwyOopMqLsQ0UUUjUKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigApyoW6dPWnLF3b8qlp2Ic+w1UC9OvrTqKKZmFFFFABRRRQAUUUUAFFFFABRRRQAUU4IT9KeEA+tNRbJckhioT7CpAoXpS0VaVjNybCiiimSFFFKFJppXASnBfWnBQKWtFDuQ5dgAx0ooorRLsQFOFAGKK6KdPUzlICcVVurjyYiwP/ANao9QEzKPLfag+9gc1ktKSxDozliBmRto7f/X7969ulRVOHPZvTp/XQ1o0Oe0mzQjviw2yYDgc81UvJ9zZVuen4f5xTZbeRpEKeWF2hQynIJHXn1ppWG3hDXeVxwVUc5zkZ+o3fgKcsfThRXNG769Leep1wowjLmQkk6WkEU7xgyZzgALnnOf1I4B/SqsGpmWaRJtxSRQirnOPTJP8Ah19KbeX9tcWyQIsiqhJHHfHXGee/0B71WsJIobgSysRtzgBc54P5H096+Uq4xvEQUZrl0v8Arfbodah7rbWpr/LFcqke7KNg5OR79h05rZjlBUc1kCOG7lM6XICOGI45BHbA/OmC6LfIHIB43Hj/AD+dfW4OtRq0HzO1tXf9O/4nFXouo1Y3TcRqyqzcscCpqw4sNe7Zc5HTcAMkHqBjgcf55rbByK5cVSi6anBbnFUp+zaQlHWnEU2vFnGzKTuRkY+lJUvWmFcdK55QtqjVS7jaKKKgoCAetRMu0+1S0UmrlRlYgopzLjkdKbWbVjVO4UUUUDCiiigCGRcHI6GmVYIyMGoCNpIpM1i76CUUUUigooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiipFi7t+VAm0hqoW6dPWpVQL06+tOopmbk2FFFFMkKKKKACiiigAooooAKKKKACinBCfpTwgH1pqLZLkkMCE/SnhAPc06irUUjNybCiiimSFFFFABQBnpShSfpUgGOlXGDZLlYaFx1p1FLitox7GbfcSilxS4rRU31J5kJiloorWMLbEN3CiikJrrpU38yGyN+RWNqG0Y5GeuK2JDgVj+ar3ZSRRIHwFH93J56jrj0r1vaeyoNtXvp/XY6sJF81+xWv8Az5IYp4nWOMkFmGRg9CemcA46Z7deAMm5uZbuQSSnLAYq7qFxGqNbwjYeFkGAQ3f7319u3WsyvhsyqL2rhFvpdX0uuny/O569KOl/uCiiivNNixZ3RtJ/MC5yMH1x/L862LW7h1B9ph8sJ0bOQOn5cKeua5+tzTN0+nsglCsHIJLc7cDP4YHTPY+lenltV+0VOT01e39evyMKySXN1LFtFJPIZCwJPGTzjGOQfWtyJdkYUdAMVk2IaOUoG3KADnHXPpWsp6V9pVhBUUqa0/y02/yPHxUm52Y+kIpaK8WpT6GMWNopSKSuWUWjVO4hUGmFSKkorNwTKUmiKipCoNMKkVk4NGikmJUTLjkdKlo61DVy4uxBRSsu00lZmydwooooAKZIuRn0p9FA07MrUU512tim1JsFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABTlQt06etOWPu35VLTsQ59hqoF6dfWnUUUzMKKKKACiiigAooooAKKKKACinBCfapAoFNRbJc0iMIT7U8IB9adRVpJGbk2FFFFMkKKKKACiilAJoASnhfWlC4pa1jC25m5dgoxRTq6KcOZmcpWDFFZl3eTQTPGxXBGVK8Ef5/wA+lC6hlFUDMpAwp4yT0r1qeDi7rm1Q3RqNKS6mizqgyzAD1JpQQayJbl7rEYVgQRu25OORnBFSafNJsIkYEDgd/wBe9axwsHLkjq7Xv0CVCUYczevY1KTIqPcKa8gUZqo4V3sc9myUtTS1Zn26RkZwvyrwSeBnOMZ/WhL7LFXI3DuMgH8DXRSo03LlT1/yN/q00rsuyNxj86yLzbuDEcZ6VYkuSUyFyuM5B7c9fyP1qhNN5hrtXs3CUeZdVudFCnKLuRaoPOt4ZIgSsa7XIPT0Pr3+lZNbkT+cotpfmj2kKOmCTnnHUZArMvLOS0cblYIxO0sACcYz396/P81oP2rqLXv5dF8n/wAA9Sm+X3WVqKKK8g1CtmwQRWnmECObja7rj5c54HftyR/KqVnp01yUdlZLdicy44GP84q88wZEiVAsacAde3rXt5PheaspTVl/Wv8Al5+RhUfMrR6bmjYZWNQSSBzjPStNWGKwoLkIBnir8VydgcqQh/i7dcc+lfdVoQaUU0r7f8A8mtSk22aIalyDVI3SquefwqvDqGPnfcY2cjd2XivPq4WMfidr6GMaM5JtI1qKjjlWRdwOR/KpK86ph3FkXtowxSYpaKx9g29h8wmBRioLvYYTuAwOearabdtKFhIUbE/P6V0Sy9ez5lvr+BpFScXJdC4y9xTalIqNhg14tSFmaxlcaRkYqEjBxU9Mde4rGSNoO2hHRRRUGoUUUUAMkXK59KhqzUDrtbFJmkH0G0UUUiwooooAKKKKACiiigAooooAKKKekeeT0oBuw1VLHiplQL9fWnAADAop2MnK4UUUUyQooooAKKKKACiiigAopQpPSpAgHuaaTZLkkMCE/SnqoX606irSSM3JsKKKKZIUUUUAFFFFABRQAT0p4UDrVKLYm0hoUn6VIBjpRRWsYpGbdwoopQKtK7sS3YBS0UV20Y66GUmUb941hPmdDxVC3VyUMgUwryjE/dPUDI/D/PFX7uEyZIbjaVx25rFuF2TfOF652gYH0r169OpOj7qVl167f16ndhOVx5b6ly7ibCiBAgTDOfulTz6/09B7VXgacHZGvzDHB46/5/Wm6vLcx7HjdfLxkP8ALlu2c9ScdcVVfV3zPsXO9Qqs3Xjufr/XvXjxzNYSUoOVr2e1/u/4NkdUYOUFpf8ArqbEd8CSPmAJO3PcUy6ufkIBGfT1qrLMlzAs4neV+h3YGPwH4f5xTHCR2vnzM43Z24GcYI7d85r24ZjRdHnlv5drb/dqYrDpNSsWPPjtbAzKZFyTwOC46Dntzjv68VHuW9tmmgEgEZw24/KBjrnA9PwGKrX8ll5KxKzPjLIyjBII43ZHr+n14zY55YlZY3KhiCceoOQfY18rWzOdGr7r038767/h8ux0xp395b+ZrSXIgkiaJvMDhd4yFTJyRkjHHse2emBTJ9QgjRI7di0LcuhHIwfoOT9T/KsiiuL+1Kyvy9bfpf77a3+Rr7FXu/6/4boaR1iTyyscMatnhiAcLjpjH4/XtTJNXupgRL5cgOcBl4HQ9O/TvmqFFcv1uvr72+ny+4fsodhzOrKAI1UjuM8/mackgQf6pCw6M2T+mcfpUdFYqck7/wCX+Rduhbi1K6hi8pHATB+XaO4xn/PpUzaruLf6NF8zZ5HIX0B/rWdRW6xddO/MyeSPY24tQtbrMM2YlyBGxGSPqeOOg5p1vdrdvIAobZHhVbpgc9RyBwPb6dDhU+OWSI5jkZD6qcV1wzaun735f0vnvt2IdGPQ29kuws7iMxuoyWA69859qk85bxjtZvKhUjzNuQxOM/Qc/hx61jS3089uIZZGcBs5J6/X1q3pUcfk3DtKPM2jYnmbecnn8MD/ACa9CWaSxVeOl1b/AD0/4bV9DN0+VXf9bG3p8mxPL3KyrjDKODxWhvrBhMtuPmBDDHyYOef6+3t9M2vtoKjB69hX1cI0q0bxle255tag3O66mgbmIMV8wZHUZ6UomVhlWBHrmsZDvdjNGBCcyEsSCcDtz1xk/TNE8rpM0sAYwuxCtnIY5yT/AJ9K5fa0IVXTnpbrdW7j+qX0TLOoS5XYSQh6kf4VJpZKq4cAMG5A7e3/AOuqEq3PmAFMlfmyBkcdalkBt5h5TuA+Cq9W98k9Dn61c6tGVblg73VtO9/+H62NXS/dchucGmsMrVawhmTfJM+5nx+GKtHrXiY6jGnPlickdHZO5FRQRg0V5R0ELDBxSVJIOM1HWbVmbxd0FFFFIYUyRcjPpT6DyMUhp2ZWopSMEikpGwUUUUAFFFFABRRRQAUUoBY4FTIgUe9AnKwiR45PWn0UVRk3cKKKKBBRRRQAUUUUAFFFKqlqAbsJ1p6x+v5U8KF6UtWo9zJz7B0oooqiAopQpPal2e9NRbFdDaKfsHrRtFVyMXMhlFSbR6UtP2YucjCk04IO9OoqlBITkwoooqyQoopQKpRb2E3YAKWiiuiELaIzbuFITxQTio3kC9TivRw9Bt6EbjZOh+lZTCN7mVZHVOmN/Tr/AJ/Wr7zKR1rJvJFZuvNetLD81Fweh14ZNSKWryTMUBSRLcnKDJ2njsPWsyt+KA6hbtAYxkYPnEZPHQZ7elY9zHBGxEMxfJyBjgL259fw/wAK+BzLDyjUdXpt8/JdvS/metTkl7nUsafepaRSeapkRuAm7oeOf0/lUE95JM8pG1FkOWVRwen+A/yar0VxvFVORU1ol+O/X57bF8kb3CiiiucsKKKKACiiigAooooAKKKKACiiigApVYo6sMZByMjNJRQBet74CSR7lpGLdPLCjnuc9u3Tr3q2imVwbY5XG9SWGQB6+nSsarVldNBJsZyIZPlcHkAeuPUV62BzGpSfs3Kye79XrcylBJXQt3eyyyyJnEZblAQRnGCf/r0ltf3FqzGOQ/MMc81Vorg+s1ebn5ncvkja1jcspZktmeZ1Jb54yArHIB6n8enUfznht7gx5TDc7treuOv1qK2gZrBYc7PLPzBv4m5OeCcDHfvitexQhG5JG4gZxkduxr7PLORUH7S7ku/yf5+X3nnYipyK8S1CpWMbic+9OPWnU09a4sXPmlc8+G9xjjmm09xxTK8yaszpjsIRkYqGp6icYY1nI1g+g2iiioNAooooAilHINR1O4yh9uagqWaxegUUUUFBRRRQAUoBY4FABY4FTqoUYFBMpWBVCjApaKKoyCiiigAooooAKKKKACigAnpUqoF5700rickhqp3P5VJRRVpWMW2woopQM/SqSuIACaeFApabNJ5MRkKlgOTjriuqjQdSXLHcylMdRSqwZQQetLitp4acNGZ86G0UuKMVn7Nj5kJRS4FLij2TDmQ2inUU/ZeYuYTFGKWirUIoXMwooorRRb0RNwpCcClprGuujS1sS2ULnUBFKYwhJAyeO3tVVnkuHy7IqEbSM5Iwc9Mj0H50/UARMj4GF7nt71BLI8VnOYCwkQHIVSAgB6evTOT7CuvFTdKLjsmtNN/Lfy8n2PQowjyqUVqQK0/lk7DtxngY49cenHWkMssVk08aK2xsPlVPB/X+f+FX+1i14JWjHlY27eCQMY6496df3qqXgiiEZ2gHD525wSOnXt1x145rx8RnMa1Fx5vh8tb200/z+7Y7fZtPbcgvLwTKiRqqqEAbA4J4J689c96pUUV8xUqzqu82dMYpKyCiiisxhRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAOjCFwJCQp7jt6Vr/ZbCAb4/MuCSdpY4VTn26n9Kxqu2NysbeW6ocjCu+SF5z0APv2713YKrShNKpG+u/X07fgZzi3qbMCi4dXdVyBj1J+tbEQCgADgViwxXED4bAUZ+btxn8unf1rWt5N8YJBB7g9Qa+9cqM6VqO3X/gnjYpSve+hZpDQDmlrw69N6nPFjSMioqlqNvvGvNqLqdMGJUcg6GpKa/3axexrHciooorM2CiiigAquwwxFWKilHzA+tJlwepHRRRSNAooooAnRdo96dRRVGG4UUUUAFFFFABRRRQAUqqWNKq7j7VKAB0pqNyJSsIqhRS0UVoZBRRRQAoGTUgGBikUYHvS1tCNjOTuKOtJIiuhVgCD1BpRTJpPLiLbS2Ow616WCg3NW3MJ6vQyLjfaz7IpWCsMMCc7Rnr+ta8R+Qc9qw53a5uAfLbaDkrg5I55/wA+tasEqmMYIxivcnD2kZRXRr8jevF8kW9y3RUYf3oEqsSAwOOtedLCPsct2SUE4pu6qt1cGFNwGeQDjrVUsJzMaTk7InjuY5JHRTyhwalrAjbOoMwcHnkqTj8+/wCVbSOCOtaywylDnitDStS9m1ZktFN3UbvauZ4fyMbjqKbu9qCSatUmFxSajY8YpScVTvJmjQbcZyM5OK7KFK2pUIuTsiG8mQAqepHSs63w0xTa53jHyPtI5zU6zx+SzTc5yJDtzgcgYP8AnqOtVDcWv2aXyGmLKck4Xp0HUepA/I9sVzZhmEFD2TjbS+tk7d7eh61GnyJopXtvb20zJFMZcccdjxnP6/561KdJI0rl3OWNNr4WtKEptwVl0/rU7oppahRRRWRQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAalkBJbySMS7sdoDnGMcnHPPGO3cfjq2d3HHGFLAY9awNOkEd6gIJWTMZwOfm4qydRhiZhbpJjGBITgnpnI6Y6/pX1OV5rToUHGpv3f3+bOStR9o+VnSi6QpkMMetRWt6z3HluDtc/uztxnuay1zdEtDIrRu2WCjPl59R7Zq7aSrHeODCVYsEGFxgY4/QZr23WpV0o07aq79ey/z8jhlh1CL6s1jUb9RUhpj9K+drqzZjTYykb7p+lLRXKbEFFFFZHQFFFFABTJBlPpT6QjIIoGtGV6KKKk2CiiigCzRRRVGAUUUUAFFFFABTlTPJ6Uqp3P5VJVKPczlPogoooqzMKKKKACnKvc0KueTT60hHqyJS6IKKKUCt4x5mZt2FpGGRS0HpXoYfR3RjIyNRtmdwyqCo64OCaqCSSI7Q4UHOCewz36fXpW44zWXeRqUPY4zXvU4OonKMmm15fLc66Fa6UJK6IjdSxxAt3xggHGPXJoguWglJlG3cAaZaTMJz8v7tVJKqTwPbn/635Upnsrtpg8gikV+WbaN3pjPP+cn286WYSpVFGr0uui6d9v+Bc6nTjqraM0Uu1fjoR1yKqXsm9ODgg5B71TkDwPnejBv4k+7nuP8+opDHMyBjjDDIy4yfw/A13RxmH5Fz6NrVfmRHDcsuaJND++aZo4SHA3ZB4B/oO/4VZhuZIwhk6MOo5HUjH6VVeRNPgS48tiz9MnOPYj0OOv19KbaTi8sfKWPfOg/vHPUc49BnA988da8qlmUYVeS+nZ9v62XkzSpTUltoaL33zBE27iCck4HAzUlrdGVTuPIOMjHP5ZrKgEscx3KuzgPuwVIz1+nTmp3/wBFcSRj903Tacgfj3PBr0IYqnVrOH2ej8zGeGio2W5sB/xpS1Zkd78yqwKluRkY49atCbI+9XQqMZax1Rxyoyi9SZmwMmqF467CG/Gpnk9Dk1QllQTHzMEY4JGRnP6961f7mDna9jWhTvIYk7PEY58+RsI3FfujGOP1/wDr1hShFkZUO5Rxn1961725u7dVBaOSFl5HKhvwzk+vHrz04xa/P82rqpU219OnS2uzPXoxsrhRRRXkmwUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFACo7RuroSGU5BHY1YFjcOu+KJ2j3YBIAPqMjPHBH51WrVsTFPaiLzER05OVI4J6579QOcdPxrrwVGNeqqU3ZMzqS5VdFyGERmJbSMYAUuxADMccnnJH6itWztnjbzZWG8jGFAAx26Co7FUEQIAGQDitAdK+1q4eGEhyw3tY8avXcnygaa33acaa33a8Grq2RDoR0UUVym5C33j9aSlbhjSVkzdbBRRRQMKKKKAK7DDEUlPlGH+tMqTZbBRRRQMs0UUVRgFFFOVCfpRa4m7DQCelSqmOT1pQABxS1ajYzlO+wUUUVRAUUAE9KeE9aai2JtIYBnpTwuOtOAxRWsYJbkOVwoopQK1jFyIbsAFLRRXVCk+hk5BSMe1BOKguJTHGWAyRXo4eg20kTq3Yc7AVk3jAttLFRjGew/D8/89XvfjcQRgj3z+oqnNI0gDlSIycbuv+TXqr2dOldysn1X6HbQoyUtUPjJNmyW5USNwVPVsg9/w6f41z5JYknqeTXR2kq2QeSSfMSnDKoJHPGTxxWJezRTTOY4wMtkvk89f/rdeeM96+Iza0knzbdO/n+mu/Q9Kk/eaS/rsaFtfQJZRi5+dsHBB3N1Ixjt1B544PrVSXVbmQLtbYy45XA6Yx/n6elUaK4J4+tKKinayS030t136GipRTuyzPf3NwhSWXcpOcbQBn/P9fWmRXUsCYibyznJZeGPTjPpxUNFc/t6vNz8zv3vqVyRta2heTVJkikTJDOSxZTjJOcn689sdKsWFxbvbG2ncq5ffvI9jxn9fxxWTRW0MZVi43d0tPk3fp+ZLpx6aHTTiG4VntnBkg+VxjB6kc9sdfoP0Z5rwv5coAYDkdcVjWd/NZMTEcA9QO/Bxn16/wD6q0zKt5arct8smNp245xj8+v+R0+oyrN1ZQfq1287v+m2csqDWj2Jnn+XqPoKjt/3s/Rml6oq8ZP17VW3VLFIscMshLIRx5gG7bn24/D3r2sfioewai9/6/roKNPlTKOpz3Ul0UuWyydAOmDyD+tUqdJI0sjSOcsxLE+pNNr82rVHUm5vqdsI8sUgooorMoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAq/YTiK3nVpxGpKgAk8568AdOOfoOvFUKtWlotyGZ5hEqsASVJ6g+n0/WujCSqRrJ0leXQidram1b3D/ACxw5bcNyFgFyPxrWsJHltEeT73esyGLMawwRyMg48wvyw9DwDj29q17aLyIFj3FiO5r7arVqzo/vlZ2XTr1PHxPJ03JD1pj9KcetNftXhVXuZw6DKKKK5jYif75ptOk+9Taye5vHYKKKKBhRRRQBFKOhqOppfu/jUNSzWOwUUUUFFmlClulOVM8npUnStFE5ZTtsNVAPc06iirsZt3CinBSaUKB71Sg2S5JDQCacFA606itFBIhybCiijFWlfYkKMUuKWtY0+5Dl2DFFFITgV10qV2Ztlae9WCULIMKRw3v6VKkodQwPBGao6lEXUN1VeqjvVeC5MI2O2R1U56ivbhh4OytutP1NlRUqalHc1i3pVeZgVI7YqB7olgqYZm4AzjNVTeB1zXTSoqMrdQhRluQSMY5Nh2lC2TuUHr15xmn3l59ijzDFmKQ4BI+Xp7r7tjn147VGixzOoyxkJHyEYDc9M544qDUr7dbxwJDLEMdHJ6eoPcHkV87nNWEXJx06bddOnn8vzPThG9kynf3Ru7pmGNgOF4xkdP6VVoor5CrUlVm5yer/r8DsjFRVkFFFFZjCiiigAooooAKntrya0LGIgbhg8f16jt+VQUVUJyhLmg7MTSaszdnhZ4Y7pFISQDg9jj+VV7svFpwQyMBI2THg4Ppz07fpUmiXSZNtOWKn7gLHA/DPr6D1o8Q4S4hiX7qpnrx1r2p4mpPDOotrWfk3pb9fmc6l76pNefyRjUUUV4Z0hRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABV+waLyXUyFZC2QoB5wOO/uaoVu6YvlmFVSHzNufMxknJPQjHrg/SvTyilUqYleztda6/IyrS5YXNbTpN4dGwHQ4IBzgfX860Kr2lstuhA6sdx+tWD0r6XG1XJ+89TwpuMptx2G0x+tPpjferxamxrHcbRRRWJoRyfe/CmU+T734Uys3ubx2CiiikMKKKKAGyfcNQVYb7p+lV6TNIbBRRRSLNCinBc08LjoK6owbPOckhgT1p4AHSlxRiuiNF9EZOYlLgDqaWqt7IFt2PUrhgPoc13YbB+0kkyOZt2RawKKghuEkAwwPGetTZFaVMFKDtYlya3FopCwFQpco8jxg/MhwacMHOS0FdvYnopu8e1IX960jhX2FdjicU0tVS8uTDCWQjIIqFb0MmScEdcGvQo4T7y1Sk1zE1w4wc1lxFWeVC21uNn1z1Hv/nBonmaWXajctgDnvT4LfZumnT5IzkPuADc+/b06VOPrQjT9h9q6/q/3/cehRp+zjdjI7UoyyPJ8ik7iuVIx6ZHrTRbwu0gjmBKjhW+Ug8cH+X1rLnvXe4aSFpEBHUsc9c59uf8APWq3mPvL723k5LZ5z6181Wzm07JuVtL3tp5ad9dfQ7I0pPVux0EMEtuJZWUBoxlTjPJ447HrWDcSCa4eQdGOe/8AUn+daEOo79PktnC8RnJc53HtjPQjjv249Dl1wZjXVWMHGV76v8rNd1+uhVNSu3JBRRRXlmwUUUUAFFFFABRRRQAUUUUAOjdo3DKSCPQkfyq3qQfzYndifMjDjPoc4/lVKppgRHASPvR5Bz1+Zh/SumnWkqE6V9HZ/c/+CS17yZDRRRXMUFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUVJFE00gUcDPLdhVRi5NRitQJ7CAyXCuYiyLk5x8uccAk8YzXSaXaiOCNipXAIC5Pr/AJ/IVV0+wXamVIjHIVjnJP8An07CtxBgV9pgsAsFR5qi99/gePjMTze5EdSGlpD1rkryvc44ISo2+8akqM9TXDU2N4biUUUVkaEcn3vwplPk+9+FMrN7m8dgooopDCiiigBG+6fpVerNVqTNIBRRRSLL2+87W9v/AN/z/wDEUnmX2f8Aj3gx/wBdj/8AE1KH9aeDXq0q0Uv6/wAjypRfb+vvK5kvv+feD/v8f/iaYZb/AHgeRDjH/PU//E1czS5rsp4mC/pf5Ge3Rf18yiZL/wD54Q/9/T/8TWTeyXReTzI4w2zgeYfQ+3NdJVO+gV4HIUbjwCfXtXbSnGtF01Jpv0/yX5mlGqoTV4owtOlvS7tsDOSdxc7T/KtQTajzmCLrx+8P+FXILRIiWA+Y9T61Y2irp8tCmqTm5W66f5MdbERnK6ijGE2pFW3QR9T1bH9KrRvcCbdGimU53jeePT+H+proSgNV47NY5pJO7nP0raNWDt77VvTX/wAlCNeKT91GS8+obk3RqDngBzyfypZJ9R8klolHuGPH6VueWPQUhQdxTjVi38b/AA/+RF9Yjp7iObeS6dlEygIcZ+Y9PypsoIn2lI1TJztkJ7nvj+lbN9bF4SEXJyKgWwUR/Moz/KqnhnW2qyWndfdt8zqp4mCje3yM+2G4kssZcfcDSEZP/fPNQ3sl8086+XlAuXAXcq5XJ5I45yfr9KszRCCYEAHaQQD3/wA4qeKVbtXhnEe1jhARluuTyOf5ZryMbg6kW488rX+L5PR6barbudHPf30v68jmqKmuLcwysisJAoyWXt9ahr46pTlSk4TVmjsTTV0FFaFvpzNZvcsoZfLYjGflIz16enr/AIHPq6tCdJRc1bmV16CjNSbt0CiiisSgooooAKKKKACiiigAooooAKM0VLKcxwcKMIenf5m61SV02IioooqRhRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUATQLbtnzpHU84AXjp6/wD1q1rNEG0AAKeRjo3Xn371h1Nb3DQOOTsJBZQeuP68mvWynMKeDrc04Jrv1RlVpucbJnaQ42jFWB0rH0++D7ULh89GXoT/ADrXU5FfX4lxqRVSDumeBVhKErMdTT1p1IeteFV2HESoz1NSVGeprkqbG8BKKKKyLI5PvfhTKfJ978KZWb3N47BRRRSGFFFFABVarNVqTNIBRRRSLNCgHFRq/Y/nUlbJnC1bceH9adUVAJFaqb6kOPYlzQwVxhhkZzzTA/rTgwNb067i7xZm4D6KbRWjrN7kcg6im0uaarCcBaKTNLW8arJcRCAaYV/OpKQjiu2jXZDVijcRgg8VlxIgklkcZKAMo/H/AD/ga0dRl8tQo4LfxdhVOCEzfPyqqfl5/X6/59K9SrF16SpJ76+i8/U78O+WHM9hiXMbuN4KFiSZGPK8k9gD1/maYDaRyyyKpdz825uNze2PxP6Vca08t1eNV3KcgHpmqpsyq4PWud5XGUrKWnfRu+35G8a0HqOSdrqCaExjDAbFUZ5HbHfgVz80flTOmc7TjOR/QkVtxbIZUfe3mgjAIwoOe5z0xVPUrOSNEnaaOXdwWUBeO3ux6+/TNfM5vhJRTS15W/W3V/f6/Lr005RUtNL/AJmbRRRXzp0hRRRQAUUUUAFFFFABRRRQAoG4gDGTxycVavyPMiQIF2RBDgY3YzzVrRLZJZ2lkcAJ2zyfXv8Ah+PtT/EKqLuJ0I2smcD69f8APpXoQpQjg5Sk/ebVvRP/AD/IwdS9VRsY9FFFeebhRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAF3T5m85YCVKMSQrLnLY49xkgV0unXYliRWdWcjPBHIrja6DTfM2QyIhEe3hQ3A5wep7kV9JkeKnJvDSa5d9b/AIHFjKUZQudHSGoredJ0yhBwcH2NSnpXZiqfK2eRG6dmJUbfeNSVG33q82psbw3EooorE0I5PvfhTKfJ978KZWb3N47BRRRSGFFFFACNwpPtVerDfdP0qvSZpAKKKKRZZpyuV+lNoqjBq5MGDdKWoKer9j+dWpdzNw7ElFAORkUVRmKGIp4YGo6KpTaE4pktFRhiKcGB9q0U0yHFodS5pKK0Ta2IaHUU2lBrop1dSJRKdzZG5lG9sRqOAOpNTRwiNAozwMZqeivUjjG1YlylZR6IiK1XmQBT6VcIqvcRs8TBeprtw1e8rNig9TDkjaWTKIxTPLAcY+tPv7a5mhEVs5MSdRz6DPO3/e7/AIVNJYMWYsepzgdP1qs4liQR7tqZ529T1/xPtWGPwVXFJz022T1S338/I9WnUi7KL2Mm8tWtLh4yG25+UkdR/jVeuhigh1FnidXDsRhsgnA7ZI449P16VjXdt9nkYK6sueBuBI9Pr9enIr4vG4J0m5w+H8n2/wAvI7IVLvlluV6KKK881CiiigAooooAKkiglm3eWhYKMsew+pp0FrLcMFjQknOODzwT/SteKL7DaqrEx3BBbK9cEjHP4f57d+DwM68lzJpem/p/VjOc7aLcdJO0NpFZoQdgG4gY59OOO9VL5zNYpnfuRuem3/PWnEFjkkk+9SoPMs5oWkfYfmKJyTjknnj/APVX0WKyx06N2ulvw/S3n8zJSilp3/4cxKKVlZGKsCrA4IIwQaSvjjpCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACr+nrGqmfzHWZW2rtIGMj8/WqFXLG6hgV0mWQq7KfkxnADev1FdWClThXjKr8JE78uh02moV8x24LtnaOn+etaNYsNw0cayxTxvAxwpbOfoemOhrWgmWeFZEPBr7HE+yqR5qOy/X7zw60JKXM+o6mP1p5601+1eHUWjHDcZRRRWBqRyfe/CmU6T71NrN7m8dgooopDCiiigBsn3DUFTS/c/GoaTNYbBRRRSKLNFFFUYBRRRQAoJHSpFYN9aiopp2JcUyeimK+eD1p9WncyasFFFFMQoJFPDA1HRVRk0JxTJaKardjTq2TT2M2rCg0tNpw5rohO+jM5IKawp1Ield1CbuZNELpn+lZN4oD5Ynb1wOp/p3/AM99hzWXeMoQ554r3aF5wavY3w7akV42e3smkhLMSc43fKDg9f04+lc8xZnJcksTkk9c1v2iP57oGCqylWYMCOf0P0+tJ/Z1nA0z3JLlnwq4IK/XBHXn6j07fI5lhHWmlB7N97Pz9ejt+B60ZqDd92VLXTY7qySRmEZ5G5eSTnuPoOg9RVOWyuIFUvGwz2we/T+v5GtSdzK4VY2QJxtJyQfT/P8AWlF1KqqNo3KMBsnP8/8AOBWksl9pCLUem662trbbX+rjU5LW/wAjGe3miz5kMiY67lIx/nI/OkEUhjEgUlCduR68f4it64RtQtUhSVS4OSB1Y8449Mnk/wCFR6fC1laG5V4xKwIALc9Rxjp7/Q15ay29Xljfl76J37W/4b8CvatLXcyhYz+U0jLtC5yCDnjOfp071bsLGB7f7VOzeXu2bcd8H8+349qu20k0k5RE4dtzBcjHqc/160+WNZGEMKqEBySvIJ5Oc9e9ejQyde2jBL3lvfVeT+XZ9e5EqrWj/AeUi05JBblt85BIzjGGPGByO4/ycRMj3EpldQGPXb0NTraOzo0jZKjGenHarYhAHINfT4HBww0feWupxTrJbO77ma8GB0xTbY+VOW3BXH3VYfK31/Q1ovEMfL+VUWhzccMVIwVwcEnPb3/wroxqjOhK/QqlU5tGY+oCX7a7TQ+Uzc7cY/H8aq1saj9pujHH5JCgAgyOMjPPXOOR69wfSsevzXG0vZ1Xa9n5deq6Xt3/AAR6NJ3igooorkNAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACrdnBFNDMZFY7SuCrYI68dMc/0+tVK19PAgtPMczEScBAwCkZ5/wDQRnpXbl9FVsRGDV0Z1XaOhbt45FIaLLqE2YcY3DGOcGtfTw4s08zO73qGxC+Uo4JwKvjpX2+LpU6MOSC6WPFr1nJ8rQHrTH6U801vu185V6ih0I6KKK5jYif75ptOf7xptZPc3WwUUUUDCiiigCOXoBUVPl+/+FMqWbR2CiiigZZoooqjAKKKKACiiigAp6v2NMooTsJq5PRUSvjg9KlrRO5i1YKKKKYgp6tng0yinF2YmrktApAcilroT6oya6DqQ8CgUyZXaIhG2sehxnFelhLSmkzGS6Gff3TRMFGAG43HtVBt0pztZlGRwfvc/hjj2/CnTiSO5CysGLHAYgHj6EVqW8CrGMDtXuJX5uZtRWmjt6v/AC+Z280aME0rszAlwYQE3IFGAAR069R3/wA9qbHA91MfMGCAMnue9bvlr6fpSJbpGWKrgtyayi6FNppf8Ht936mf1t2ehRjsQmSRnPsBVe8iEcZOK2dvtVS8t2kQbMBgwOcciuiliU/dj8jOnWbmnJmXAGtnm+eMuF27e+e4/Q9Papo4XnSNGG1VHKgdwTj9DUcaqt8VC7Ru6Fs/l61tJGAK5qUKcUqsr81/TX0OnEVnDbqZr2JU7kUEYIKHgN6frzUtnamNSWXGTwMdB+ZrQ2+1Lt+lN4mKk5LdnI68nHlZGEx0GKCpqTaaTGKhV7vcxuyFlDfWqF5EpUk1qEZ+tUb2JnQbQDk8g+ldlGqrG1GXvIz4/PeF0meQW+wlj/s9c/z+vNYMoQSN5Zyh5Ht7fhXQiEzWxSSSQKAd46hAAenP9fXrVBrC3W2laKZnYnbgxE7R19Pbr6Z+lfH5th3Od4JJat73T7f11/H2KU7Xv+BlUUrKyMVYEEdiKSvnWmnZnUFFFFIAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAACTgDJNW1vniTy40jKBshmjGSMAc+nQdKTT0L30RDFQh3lsdAOf6VK+m/MTDOrp2yPmPrx+f5V2YejXa56O9/63Ik43szWido/La2/eRuAS2D8pI6HHAPPua1LW6MvySIySYztKnp61hwrHZPtjhOVbEjH5icdcccVoWkMhu3PmbSrDICgcY4HHsa+woe2qQUK3a/8Awf6+Xc83EU4OLl+Jqmmt92nGmP0rxq2jZyQ6DKKKK5Dchb7x+tJRRWR0BRRRQAUUUE4BNAEDnLmm0UVJuFFFFAFhWDDNLVdWKnIqcEMMimZSjYWiiimSFFFFABRRRQAU5W2n2ptFANXJgQRkUtQgkdKlVg1WncxlGwtFFFUSKDg1J1qKnK2PpVwlbRkyVyQdaGOF5pKZcRmeFowQN3BJGcCvRwcoqaUnZGElcx7x1nuP3QLqMF8cjH+c/ma2IVHljHTFOSJEUAKOlP6V6dfF07NQ3fX00CpPnSilogoozRmvPdZt7kcoUhGaWinGtJMXKVorNIppJRyX9e1WQMUUVrLEyluN3k7thRRRWXtWKwUUUVcaqe4mhpFMdcipaYwruoVGmIyry3B3HOCetZ8G1ZSWeNUUZO/ODWjfs3mLGBkN1GcE+w96qzKYrSd4ciRhzwMfe4wfTqfwrfH1E6TjGN5eS23t/SPUw7airvcyb97WWdntywHX5uh6cD0x/T86dXhpVx9rEDYx1MgBIxjPHr3+tP1CwaIvMhiKgAsI84B4Bx2xk9OvPpXwdajXm5VZxt3O+M4q0UzOooorjNQooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooq3ZWhuHLYDKvJUOAx+nB9fxq6cHOXKhN2V2WNP2CCXG3zk/vYXAPBA5ye3GP586lnaB13hmG4Y4OKijfz5MG3QKc8BR745/Lp+Va1rF5cQBxnvgYr73LsM8LQcKkd+631/q55mKrWWm5H9iTy9uMj3qOztJFut5yqR8LwBuHTB79u9aQFLRiMWmrNarY4Y1Z2a7iGo36in1Gxyxr56rK5pBCUh+6fpS01/u1zvY2W5FRRRWZuFFFFABTJDhPrT6ilPIFIqKuyOiiikahRRRQAUqsVORSUUAWAQwyKWq6sVORU6sGGRTMpRsLRRRTJCiiigAooooAKOlFFAEivng0+oKcrkdeRVKXczlDsS0UgIIyKWrMxQ2PpTwQelR0VcZtEuKZLRUYY07f7Vp7RMjlY6ik3CjcPWndBZi0UZzRTEFGaKKd2gFzRmkoqlUkieVDqKbSg1pGaZLi0LSEZFLRXTSqcr1IaMy7sZJZvMV+2AD/DVdC9u+xkKgDJZD0BOOBkD0/GtojNQSwJIMMufavT/d1k1LRvrr/VjeniWkoy2MENOQdw+YDbll5A/yfrSGKd7ExwxFnc43bDwvQgH/AD1/LXa2RVwoAHtWXcxiNiQcE8HBratl9OpR5aej/S1vl+XkdlPEc7sZd3YvbbD1UoGOOcdM8jjrkfhVSt+ArBbNPPHHJB02sAxz/Tmse5FtuP2cvwcYI4I9c9fwx/hXw2NwkKLvGXy6/Ly9bHdCbbs/vIKKKK841CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooqe0tzcThcMUHzSFeoXuaunCVSShFXbE2krsijCFx5hIXqcdfpWx/oUihLW4Ubc4SUbc89ie3PANYtAODkda6MLi3h5XST1+f3kyjfW509p+6ZVbGSMjB7f0/GtiPkcVz8MqtaieVRIZOhHG3qNp6ntweta9jJuQgEkA8HGM/TnpX3NLFSxdG9tuvf8Ar17nj4qlb3i7SGlpD1rxsQ2mzmgIeBUVPc8UyuCo9bHTBaBTJD0FPqJzlqxlsaQWo2iiioNgooooAKrscsTUznCmoKTNILqFFFFIsKKKKACiiigApQSDkGkooAnVw31p1VqkSTs3507mco9iWiiimQFFFFABRRRQAUUUUAKCQcinq4PXio6KadhOKZPRUSsV+lSBgatO5k4tC0UUUyQooooAKXJHekooAcHNODA1HRVKbRLiiWiowxFPDA1qppkuLQtFFFUSKDS02lBrWE+jIkuotIRS0V3UZ6mTRBIOD9KymWN7mXzlZ1GMKCe5x/OtC6mEeVwSdpb8BWLcszzFWBU5wc+vT6V7E2vq8lzWf9fodmEg2ypqxlWVY2ZCq8AqOTgDqcc/0rNrc1gzGGOJIWNvgFGwePf6ketUG0yceaU+ZYwGBxgkHp+PP/66+HxuHq1KzcE2rL5abLul5f8AD+pSmuRXEsrL7asiqyoyfNubPT0qGa2mgdkdD8vUjkdu/wCI/Otby47a0EEbSbycsHUqefbPsP0pHEUto0bMUckliFyXyR9M+vNdn9kylQU2rNLddX2+92uL2rvdbf1qYlFat9ZWscSyxyMiHKqWXJfA+g6nj9fWssIzKzBSQvLEDp9a8WtQlSlyyNozUlcSiiisSgooooAKKKKACiiigAooooAKVVZ2CqCWJwAByTTzC6wiVlwhOAT3/wA/1q9pi2rRT+aqecigpvYgHk5/HpXRTw8pTUJaX11JlKyuQW9kZWkWVmhKYJ3Rkjn19KvRuLUGOIB49uDnOH68n061PGXmbeFxJn7wAxnOc/4+vHpzYFntAK5DKcg19fgMjUIt1NH08+39dTkniEnaRz9zbSQzuuzgc/KCQB1qOG3lnLCJC20ZOB0rfVfMLQzlDEcgksBhsfe9zg/16ikl8yGdobZvlQkgr976E9e1eVLI/wB+6cXe3T8d+nTu+5oq+livYrJNZFTAyrH93arEO2OvXrwfb29bdveyqgWNNzY/DgVHJLOJcrGqljwoUHJ9fr6VJKyPIBD8yqQdyjKOSM5OOh/P8K9zCQrYRfV5a3Xrptey+7btv0wmozV2jbifegJ/lilPWqtjdGdWRoyjp1Bq0etcWYQ5KljzVFxk0xjnnFNpSck0lePJ3Z0LRAeBmoOtSSHAx61HWcmbQWlwoooqSwooooAilPQfjUdKxyxNJUmyVkFFFFAwooooAKKKKACiiigAooooAcrlfcelTKwYcVXoBIORRclxuWaKjWQHg8GpKozasFFFFAgooooAKKKKACiiigB4cjrzTwQelQ0dKakS4Jk9FRiT1p4IPQ1aaZk4tC0UUUxBRRRQAUUUUAODetPqKlBIrSM7bkuPYkopAQaWtTMcKKaKdXVRnqjKSKN/EskR3MVx3BxWdbhZCEeKPy+gY4G5uwz/AJ/pVq9hnmlc7WWNRj72d3vgf5/lSLYyeXG+cumCqtwB07fhXtVKXtqfLZade+my/wCDsdlGap01eW/4EV7I0eHAWTf95iR8p5HGOnT1PSq8UtxuaQDeWAB3dCP8/wCTVh4XtcSNtC7txAAIXkdM1JpyMyZMe0dRjpj2qMPhqcZyhVWujT/rqauoo0uaOq2GfZXkbMshfbkLu9M1HdW2EyBz9eK2AmBTZIg6ketejCtCK5I6I41iHzXZiS2kU2nldw3Bjtk2YAxzyfyHXvTY410+2eJWSR3YMRtPA7EHAIP8u3eriW1xCriPjdzkcHOfUc4xSLaMxJk3AZ4Vmzj+n+RXkxypTq3n5631tr+d/uO11463d0Zk1nGZYYoUUMdu5XBGSOMZODnPp79MZpk+mIPL+zuzr/G56DJwOMD+Z/StVoHjQrHhVYFWx/F1x/P/ABqnIjxoU3NsJyVzxms/7AjJS5kvL5W+69tfXuaQr3as/wCv62KLaTdiNnEe/a2MLknpkH6Y/Gozp12CQYHUjPDcE/T1/CtKEMpEkjFY1UldxwGx2GeOv5VR1DUpbs7PMJjXODjbkccEZ6cV85isHQw9+Zvys9381t/wx0KU27KxTaNlUElcH0YH+VOSF5B8gBJ6LuGT9B1NR0V5acb7fj/wDXUmW0uHXcsLlcE5xxwM/wBKsHSbobjtUhW2k54+v06/lUljqr26pbyn/R8ndtX5uff2qeSFlIckMrYIZTkcjIr18HgaGKmowb+emvy6ef6mTnJXvoNh0mOPdLdSExKRtKDh/Uc4PoeO2aksrR7WSVlbJK/LzgdxyT9R09e3QyxwNKo3EkDgc9KupbM6Kj4IUbc4ySM5xnsOK+iWQ06TTjb73pp+Pp/wTkniLXuyhJOZoXW5UururAHOABnPf6U6KBLJ3UE+VIMrgDcpGMgn/OPzrSNoMZHDDkEdRVeOzllxHI7bUY9euMdj6Vricvp+0U6SXb779unl5kQxEGnfRD7CMMpYKVTjaCcnp/jmtDZxjiiCBYYwi9BU2K2qYnlsk9jz6k+eTaKZsYWLEr945Izxmnrbog+Ufj3qzRiso413sJzm92Y1/FtAJyqeoGTmpdMBkEjuBuLc49e/+RVu7RTAwbGMdziqumQSJiVs7XXI+bpXTOpde1v0aOhT5qDXU0QiqSwAyeppGOBTiajY5OK+axFZzd2RTiNoopjtgY9a4m7HQld2GMctmkoorI3WgUUUUAFMkbC49afUDtuakyoq7G0UUUjUKKKKACiiigAooooAKKKKACiiigAooooAKeshXg8imUUA1csBgwyKWq4JByDUqyA8Hg07mbjbYfRRRTICiiigAooooAKKKKACjpRRQA8SetPDA9KhopqTIcEyeiow5HXmnhgelWmmQ4tC0UUUyQooooAAcU8NmmUVUZNCauS0oNRhvWn1tGfVGTj3HdaMD0ptLmuqGIa6mbgNkiSRdrDIyDSqiqMAUuaWuhYqTVrkuL2DA9KTApaKqNcnlGFaaV9KlpCO9dlKu+4tirIgx0+orKvAAQuQMnGa2pBkGsYQGS7OSMR4Zg3IIzz9PpXfPEclByfXT0udmF1d30Kepu1vaRQq2VlXJzk4GQePxGelY9aep27Y89XZ0ODtXlUHTrnjnOOKzK+AzOTliHfayt6d/n9/fU9elblCiiivPNQrbsz51gokzIykBIwNpx0yvr0649c9BWJW1pflQWTTvGGYybd2TnHH+ev8q9HK5zhXvHt/l/SMa3wmhYkOox1rTVRisyxYtIAoYRgAKD29a1lHSvva1ZuCk1Y8XEaTsKFpQoFLRXk1K7WhgkFFBOKbmuGdXuWojqTIpKKy9qy+UHVHGHUMM5wRQMKoVQAAMADtQSB1phYnpSniJcvK2NQFZuwplFFcjdzZKwhOBk1ETk5pXbceOlNrKTubxjYKKKKRQUUUUANdtq+5qCnO25vYU2pZrFWQUUUUFBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAPWQrweRUoYMMiq9KCQcg0XJcbliimLIDweDT6ozasFFFFAgooooAKKKKACiiigAooooAeHI6808MDUNFNSZLgmT0VGHI6808MD0q00zNxaFooopkhSgkUlFAEgYGlqKnBiOtaxn3Icew+ikBBpa0IFzS02itI1GtyXHsOooorrpT1sZNEb8CsXUFBx1J+taGoTvCoARtp6sO1ZLK8jHc6tgjIf5fT/H1r241I06Lcu3a/odeFpu6mRag32a2SEwZRsCRgefUjOPUcdeh68isiWKSFtsilWxnBroJZzFLHiIbQA64bJ5x3/Dp7UyVIr+MG5by5DzvA64OB9Byc/QGvlsdgZ1kqvNpsu3muu7v5fej0ac3Hdepz9FaF1pqWsSym4yjZ24UZPTpz6n9D1qCygjuZvLdmBIOMD2JJP09O9eI8NUVRU2tXtquuxupprmGW1u91MI0xnqfp9Oprbs7WWzdhcTh0YY2A5JGFGPyPb0HpRElnZSmLy33KrAyDgkn8+MdKZ5LcFizRryAfT8+OnrX0OW5VNx9rHddddfJHPUqc2j0RPZzmIldhwPmOPQ//rrbiYMgYdCMisaIB78tLknrhlIIyeh49+v/ANattQAoA6V7dao3QTm9f+B17s8zFKPMrC0hNKabXiVJ/eZRQUUUwv6VztpbmqVxxIFNLE9KbRWTm2WopBRRRUFBUbvngUO+eBTKiUjSMerCiiipNAooooAKZI21eOpp54Gars25iaTKirsSiiikahRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAU9ZCvB5FMooBq5YDBhkUtVwSDkGpVkB4PBp3M3G2w+iiimQFFFFABRRRQAUUUUAFFFFABRRRQA9Xx15p4IIyKhoBI6VSkQ4J7E9FMV+x/On1SdzNprcKKKKYgp4f1plFNSa2E0mS0VGCRTwQa2jJMhxsLTqbRWsJ8pm1cVlDDBFU7y1EkR2rz7DmrtFerh8VKGlzNNxd0Y8Vi333UBiOg7VVu4BGcKOTzgV0BWqtxbrJzzn1FetRrRlDkil5XN4Yl895GPcWz3VnFDnY2eA3Qc7ecd857HjPpVG0sZBcszPt8obwwzhu/Xr3H/wBatCaB4NrrtDJ0Kjrz1pbiTfAqeZlgBuYNw/PT14xnn3rwMTlajWhOej9fOy+7fT0Z6UKl17uzGsFe6V0bdvOcYPTvzgVsRwrtGRmqNlbDhjn1wa1kFe9Lmw9Pkvc87E1E2kuhAbKJ2RiuNhyMVaopDXjYrEyn8XQwV5biGiimMe1eVKVtWbJX0Bmz9KbRRWDdzVKwUUUhIAyaQC1Ez54HShmLfSm1DkaxjbVhRRRUlhRRRQAUUU122r70AtRkjZ+UfjUdFFSbJWQUUUUDCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAHrIV4PIqUEEZFV6UEg5FFyXG5YopiyBuDwafVGbVgooooEFFFFABRRRQAUUUUAFFFFABTlYr9KbRQDVyZWDUtQVIr9jVqXcylDsPoooqiAo6UUUAPVs/WnVFUinI961hK+jM5RtqOBpabSiumnLoZSXUWo26VJUEsqpkEjPXFexhG5NWMra6GffSKoIJ5OeKoBMLskJUZ3BsZA7f5+lT3Ewe6G0huMbW6Hn/ACfwq9BaZi/e/Mx+9716VZOcuVuyj163/wCAehGaowTfUq2tyYm2SjHvnitaGRZFDKwIPcVny6WHclSoXH3cf4U2FZbO6CFQ5cHG3j36dqid6qcZWfn/AMAxqRp1FeD17GvTaELGNS67WIGVznB9KOlfNV371jKCEY4FR0pOTmkrhlK7OiKsFFFMZ8cDrUt2KSuOZgtREknmk60Vm3c1jGwUUUUigooooAKKKKAEJwMmoGbcc0533HjpTKlmsY2CiiigoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAp6yFeDyKZRQDVywCCMilquCQcg1KsgPB4NO5m42H0UUUyAooooAKKKKACiiigAooooAKKKKAHKxX6VICCMioaUEg5FNSsTKNyaimqwb606tDJqwUoODSUUCJQcigdaYp7U+uiEr6mMlbQdWVqTRh0ViNx7FcnH8/5Vqk4FZN7IklxGAcqh3OR2Fe/gablCS6CofxLkNla+bIyTBmUDIJXYD9AOnb8q2woApsYAUYpxNZYmuqcVShsiak3Vldi1FJCJGjbO0o2en6U/NISBXAsZOGsQUBxNRs2fpSFiaSuCpUcmbRjYKKCcDJqJn3cDpWTdjVRuKz9hTKKKzbuapWCiiigYUUUUAFFFFABUUj54HSh3zwOlR0maRj1YUUUUiwooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAHK5X3HpUysG6VXoouS4plmioll/vfnUgIIyDTM2mhaKKKYgooooAKKKKACiiigAooooAKer9j+dMooTsJpMnoqEMV6VIHB+taKVzJxaHU9Wz9aZRVxlYhq4s6PKm1XUA9QRyfx7U5IIo02KihfQCkDmjf7V6CzCagoLSxk6b2HIqxoFXOB6mgsBTCxNJXJVryqS5nuUoWHFiabRRWLbe5olYKQsFHNNZ+w/OoycnJqHIuML7isxY0lFFQahRRRQAUUUUAFFFBIAyaAConfPA6Ujvu4HSmUrmkY9WFFFFIsKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAClBK9DSUUASrKP4uKkqtSqxXoadyHDsWKKjWUHrxUmc9KCGmgooopiCiiigAooooAKKKKACiiigBwcjvThIO/FR0U02iXFMm3Ke9LUFFPmJ9mT9KQso71DRRzD9mSGT0FMLE9aSik22UopBRRRSGFFFFABRRRQAUUZx1qJpey/nSGk2PZwvXr6VEzFj7elNopGiikFFFFBQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFKCV6GiigCRZf7w/KnghuhoopkSirXFooopmYUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUxpAOnNFFJlxSZEzFjzSUUUjQKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA//9k=\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Import libraries for simulation\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "# Imports for visualization\n",
    "import PIL.Image\n",
    "from io import BytesIO\n",
    "from IPython.display import Image, display\n",
    "\n",
    "def DisplayFractal(a, fmt='jpeg'):\n",
    "  \"\"\"Display an array of iteration counts as a\n",
    "     colorful picture of a fractal.\"\"\"\n",
    "  a_cyclic = (6.28*a/20.0).reshape(list(a.shape)+[1])\n",
    "  img = np.concatenate([10+20*np.cos(a_cyclic),\n",
    "                        30+50*np.sin(a_cyclic),\n",
    "                        155-80*np.cos(a_cyclic)], 2)\n",
    "  img[a==a.max()] = 0\n",
    "  a = img\n",
    "  a = np.uint8(np.clip(a, 0, 255))\n",
    "  f = BytesIO()\n",
    "  PIL.Image.fromarray(a).save(f, fmt)\n",
    "  display(Image(data=f.getvalue()))\n",
    "\n",
    "# Use NumPy to create a 2D array of complex numbers\n",
    "\n",
    "Y, X = np.mgrid[-1.3:1.3:0.005, -2:1:0.005]\n",
    "Z = X+1j*Y\n",
    "\n",
    "xs = tf.constant(Z.astype(np.complex64))\n",
    "zs = tf.Variable(xs)\n",
    "ns = tf.Variable(tf.zeros_like(xs, tf.float32))\n",
    "\n",
    "\n",
    "\n",
    "# Operation to update the zs and the iteration count.\n",
    "#\n",
    "# Note: We keep computing zs after they diverge! This\n",
    "#       is very wasteful! There are better, if a little\n",
    "#       less simple, ways to do this.\n",
    "#\n",
    "for i in range(200):\n",
    "    # Compute the new values of z: z^2 + x\n",
    "    zs_ = zs*zs + xs\n",
    "\n",
    "    # Have we diverged with this new value?\n",
    "    not_diverged = tf.abs(zs_) < 4\n",
    "\n",
    "    zs.assign(zs_),\n",
    "    ns.assign_add(tf.cast(not_diverged, tf.float32))\n",
    "    \n",
    "DisplayFractal(ns.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AtvjPq7HCtiQ"
   },
   "source": [
    "Mandlebrot render is both simple and infinitely complex at the same time. This view shows the entire Mandlebrot universe at the same time, as it is completely zoomed out. However, if you zoom in on any non-black portion of the plot, you will find infinite hidden complexity. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZeWypKgGCtiR"
   },
   "source": [
    "### Introduction to Keras\n",
    "\n",
    "[Keras](https://keras.io/) is a layer on top of Tensorflow that makes it much easier to create neural networks.  Rather than define the graphs, as you see above, you set the individual layers of the network with a much more high-level API.  Unless you are performing research into entirely new structures of deep neural networks, it is unlikely that you need to program TensorFlow directly.  \n",
    "\n",
    "**For this class, we will use usually use TensorFlow through Keras, rather than direct TensorFlow**\n",
    "\n",
    "### Simple TensorFlow Regression: MPG\n",
    "\n",
    "This example shows how to encode the MPG dataset for regression.  This dataset is slightly more complicated than Iris, because:\n",
    "\n",
    "* Input has both numeric and categorical\n",
    "* Input has missing values\n",
    "\n",
    "This example uses functions defined above in this notepad, the \"helpful functions\". These functions allow you to build the feature vector for a neural network. Consider the following:\n",
    "\n",
    "* Predictors/Inputs \n",
    "    * Fill any missing inputs with the median for that column.  Use **missing_median**.\n",
    "    * Encode textual/categorical values with **encode_text_dummy**.\n",
    "    * Encode numeric values with **encode_numeric_zscore**.\n",
    "* Output\n",
    "    * Discard rows with missing outputs.\n",
    "    * Encode textual/categorical values with **encode_text_index**.\n",
    "    * Do not encode output numeric values.\n",
    "* Produce final feature vectors (x) and expected output (y) with **to_xy**.\n",
    "\n",
    "To encode categorical values that are part of the feature vector, use the functions from above if the categorical value is the target (as was the case with Iris, use the same technique as Iris). The iris technique allows you to decode back to Iris text strings from the predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bmb3cFrUCtiR",
    "outputId": "6aee69f2-aa06-411d-a7be-4362e4e35f7b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "13/13 - 1s - loss: 1456707.3750\n",
      "Epoch 2/100\n",
      "13/13 - 0s - loss: 672845.3750\n",
      "Epoch 3/100\n",
      "13/13 - 0s - loss: 233912.3594\n",
      "Epoch 4/100\n",
      "13/13 - 0s - loss: 55254.3047\n",
      "Epoch 5/100\n",
      "13/13 - 0s - loss: 6793.9985\n",
      "Epoch 6/100\n",
      "13/13 - 0s - loss: 973.5875\n",
      "Epoch 7/100\n",
      "13/13 - 0s - loss: 1006.3837\n",
      "Epoch 8/100\n",
      "13/13 - 0s - loss: 819.4775\n",
      "Epoch 9/100\n",
      "13/13 - 0s - loss: 617.4404\n",
      "Epoch 10/100\n",
      "13/13 - 0s - loss: 573.3674\n",
      "Epoch 11/100\n",
      "13/13 - 0s - loss: 563.9692\n",
      "Epoch 12/100\n",
      "13/13 - 0s - loss: 555.4481\n",
      "Epoch 13/100\n",
      "13/13 - 0s - loss: 548.3217\n",
      "Epoch 14/100\n",
      "13/13 - 0s - loss: 542.3599\n",
      "Epoch 15/100\n",
      "13/13 - 0s - loss: 535.7351\n",
      "Epoch 16/100\n",
      "13/13 - 0s - loss: 531.8860\n",
      "Epoch 17/100\n",
      "13/13 - 0s - loss: 526.1758\n",
      "Epoch 18/100\n",
      "13/13 - 0s - loss: 520.2935\n",
      "Epoch 19/100\n",
      "13/13 - 0s - loss: 515.1531\n",
      "Epoch 20/100\n",
      "13/13 - 0s - loss: 510.9998\n",
      "Epoch 21/100\n",
      "13/13 - 0s - loss: 506.2865\n",
      "Epoch 22/100\n",
      "13/13 - 0s - loss: 500.7303\n",
      "Epoch 23/100\n",
      "13/13 - 0s - loss: 495.5735\n",
      "Epoch 24/100\n",
      "13/13 - 0s - loss: 491.1421\n",
      "Epoch 25/100\n",
      "13/13 - 0s - loss: 486.0746\n",
      "Epoch 26/100\n",
      "13/13 - 0s - loss: 480.8650\n",
      "Epoch 27/100\n",
      "13/13 - 0s - loss: 478.1591\n",
      "Epoch 28/100\n",
      "13/13 - 0s - loss: 472.8083\n",
      "Epoch 29/100\n",
      "13/13 - 0s - loss: 466.3623\n",
      "Epoch 30/100\n",
      "13/13 - 0s - loss: 460.8217\n",
      "Epoch 31/100\n",
      "13/13 - 0s - loss: 455.6329\n",
      "Epoch 32/100\n",
      "13/13 - 0s - loss: 450.3487\n",
      "Epoch 33/100\n",
      "13/13 - 0s - loss: 445.4745\n",
      "Epoch 34/100\n",
      "13/13 - 0s - loss: 440.2110\n",
      "Epoch 35/100\n",
      "13/13 - 0s - loss: 435.0027\n",
      "Epoch 36/100\n",
      "13/13 - 0s - loss: 429.8761\n",
      "Epoch 37/100\n",
      "13/13 - 0s - loss: 424.3570\n",
      "Epoch 38/100\n",
      "13/13 - 0s - loss: 418.7068\n",
      "Epoch 39/100\n",
      "13/13 - 0s - loss: 413.6172\n",
      "Epoch 40/100\n",
      "13/13 - 0s - loss: 408.2991\n",
      "Epoch 41/100\n",
      "13/13 - 0s - loss: 403.3253\n",
      "Epoch 42/100\n",
      "13/13 - 0s - loss: 397.9519\n",
      "Epoch 43/100\n",
      "13/13 - 0s - loss: 392.1912\n",
      "Epoch 44/100\n",
      "13/13 - 0s - loss: 386.9039\n",
      "Epoch 45/100\n",
      "13/13 - 0s - loss: 382.2827\n",
      "Epoch 46/100\n",
      "13/13 - 0s - loss: 376.5594\n",
      "Epoch 47/100\n",
      "13/13 - 0s - loss: 371.0152\n",
      "Epoch 48/100\n",
      "13/13 - 0s - loss: 367.0930\n",
      "Epoch 49/100\n",
      "13/13 - 0s - loss: 360.7953\n",
      "Epoch 50/100\n",
      "13/13 - 0s - loss: 355.2081\n",
      "Epoch 51/100\n",
      "13/13 - 0s - loss: 351.5594\n",
      "Epoch 52/100\n",
      "13/13 - 0s - loss: 343.8980\n",
      "Epoch 53/100\n",
      "13/13 - 0s - loss: 341.4225\n",
      "Epoch 54/100\n",
      "13/13 - 0s - loss: 334.4555\n",
      "Epoch 55/100\n",
      "13/13 - 0s - loss: 328.9076\n",
      "Epoch 56/100\n",
      "13/13 - 0s - loss: 323.1662\n",
      "Epoch 57/100\n",
      "13/13 - 0s - loss: 318.5339\n",
      "Epoch 58/100\n",
      "13/13 - 0s - loss: 313.3682\n",
      "Epoch 59/100\n",
      "13/13 - 0s - loss: 308.1586\n",
      "Epoch 60/100\n",
      "13/13 - 0s - loss: 302.7293\n",
      "Epoch 61/100\n",
      "13/13 - 0s - loss: 297.7411\n",
      "Epoch 62/100\n",
      "13/13 - 0s - loss: 293.1888\n",
      "Epoch 63/100\n",
      "13/13 - 0s - loss: 289.2863\n",
      "Epoch 64/100\n",
      "13/13 - 0s - loss: 282.9069\n",
      "Epoch 65/100\n",
      "13/13 - 0s - loss: 277.5980\n",
      "Epoch 66/100\n",
      "13/13 - 0s - loss: 274.5194\n",
      "Epoch 67/100\n",
      "13/13 - 0s - loss: 269.3029\n",
      "Epoch 68/100\n",
      "13/13 - 0s - loss: 264.0596\n",
      "Epoch 69/100\n",
      "13/13 - 0s - loss: 258.5668\n",
      "Epoch 70/100\n",
      "13/13 - 0s - loss: 254.6998\n",
      "Epoch 71/100\n",
      "13/13 - 0s - loss: 249.3832\n",
      "Epoch 72/100\n",
      "13/13 - 0s - loss: 246.2544\n",
      "Epoch 73/100\n",
      "13/13 - 0s - loss: 240.0666\n",
      "Epoch 74/100\n",
      "13/13 - 0s - loss: 235.0115\n",
      "Epoch 75/100\n",
      "13/13 - 0s - loss: 230.9614\n",
      "Epoch 76/100\n",
      "13/13 - 0s - loss: 226.7763\n",
      "Epoch 77/100\n",
      "13/13 - 0s - loss: 222.6867\n",
      "Epoch 78/100\n",
      "13/13 - 0s - loss: 217.8021\n",
      "Epoch 79/100\n",
      "13/13 - 0s - loss: 213.8861\n",
      "Epoch 80/100\n",
      "13/13 - 0s - loss: 209.1690\n",
      "Epoch 81/100\n",
      "13/13 - 0s - loss: 205.4605\n",
      "Epoch 82/100\n",
      "13/13 - 0s - loss: 200.9314\n",
      "Epoch 83/100\n",
      "13/13 - 0s - loss: 196.6704\n",
      "Epoch 84/100\n",
      "13/13 - 0s - loss: 192.4684\n",
      "Epoch 85/100\n",
      "13/13 - 0s - loss: 188.9522\n",
      "Epoch 86/100\n",
      "13/13 - 0s - loss: 185.1580\n",
      "Epoch 87/100\n",
      "13/13 - 0s - loss: 181.2294\n",
      "Epoch 88/100\n",
      "13/13 - 0s - loss: 176.8582\n",
      "Epoch 89/100\n",
      "13/13 - 0s - loss: 172.9136\n",
      "Epoch 90/100\n",
      "13/13 - 0s - loss: 169.4359\n",
      "Epoch 91/100\n",
      "13/13 - 0s - loss: 166.0690\n",
      "Epoch 92/100\n",
      "13/13 - 0s - loss: 162.4823\n",
      "Epoch 93/100\n",
      "13/13 - 0s - loss: 159.2503\n",
      "Epoch 94/100\n",
      "13/13 - 0s - loss: 154.9118\n",
      "Epoch 95/100\n",
      "13/13 - 0s - loss: 152.5691\n",
      "Epoch 96/100\n",
      "13/13 - 0s - loss: 148.5987\n",
      "Epoch 97/100\n",
      "13/13 - 0s - loss: 145.1593\n",
      "Epoch 98/100\n",
      "13/13 - 0s - loss: 142.3996\n",
      "Epoch 99/100\n",
      "13/13 - 0s - loss: 139.3435\n",
      "Epoch 100/100\n",
      "13/13 - 0s - loss: 135.2217\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f27b1545150>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "import pandas as pd\n",
    "import io\n",
    "import os\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "cars = df['name']\n",
    "\n",
    "# Handle missing value\n",
    "df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())\n",
    "\n",
    "# Pandas to Numpy\n",
    "x = df[['cylinders', 'displacement', 'horsepower', 'weight',\n",
    "       'acceleration', 'year', 'origin']].values\n",
    "y = df['mpg'].values # regression\n",
    "\n",
    "# Build the neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(10, activation='relu')) # Hidden 2\n",
    "model.add(Dense(1)) # Output\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x,y,verbose=2,epochs=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "82O2rvqiCtiR"
   },
   "source": [
    "### Introduction to Neural Network Hyperparameters\n",
    "\n",
    "If you look at the above code, you will see that the neural network contains four layers.  The first layer is the input layer because it contains the **input_dim** parameter that the programmer sets to be the number of inputs that the dataset has.  The network needs one input neuron for every column in the data set (including dummy variables).  \n",
    "\n",
    "There are also several hidden layers, with 25 and 10 neurons each. You might be wondering how the programmer chose these numbers. Selecting a hidden neuron structure is one of the most common questions about neural networks.  Unfortunately, there is not a right answer.  These are hyperparameters.  They are settings that can affect neural network performance, yet there are not a clearly defined means of setting them.\n",
    "\n",
    "In general, more hidden neurons mean more capability to fit complex problems.  However, too many neurons can lead to overfitting and lengthy training times.  Too few can lead to underfitting the problem and will sacrifice accuracy.  Also, how many layers you have is another hyperparameter.  In general, more layers allow the neural network to be able to perform more of its feature engineering and data preprocessing.  But this also comes at the expense of training times and the risk of overfitting.  In general, you will see that neuron counts start larger near the input layer and tend to shrink towards the output layer in a sort of triangular fashion. \n",
    "\n",
    "Some techniques use machine learning to optimize these values.  These will be discussed in [Module 8.3](t81_558_class_08_3_keras_hyperparameters.ipynb).\n",
    "\n",
    "### Controlling the Amount of Output\n",
    "\n",
    "The program produces one line of output for each training epoch.  You can eliminate this output by setting the verbose setting of the fit command:\n",
    "\n",
    "* **verbose=0** - No progress output (use with Jupyter if you do not want output)\n",
    "* **verbose=1** - Display progress bar, does not work well with Jupyter\n",
    "* **verbose=2** - Summary progress output (use with Jupyter if you want to know the loss at each epoch)\n",
    "\n",
    "### Regression Prediction\n",
    "\n",
    "Next, we will perform actual predictions.  The program assigns these predictions to the **pred** variable. These are all MPG predictions from the neural network.  Notice that this is a 2D array?  You can always see the dimensions of what Keras returns by printing out **pred.shape**.  Neural networks can return multiple values, so the result is always an array.  Here the neural network only returns one value per prediction (there are 398 cars, so 398 predictions).  However, a 2D range is needed because the neural network has the potential of returning more than one value.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HbErLyX_CtiR",
    "outputId": "d3ee0abe-e5b6-4575-8f7f-666a868e0e3f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (398, 1)\n",
      "[[22.539425]\n",
      " [27.995203]\n",
      " [25.851433]\n",
      " [25.711117]\n",
      " [23.701847]\n",
      " [31.893755]\n",
      " [35.556503]\n",
      " [34.45243 ]\n",
      " [36.27014 ]\n",
      " [31.358776]]\n"
     ]
    }
   ],
   "source": [
    "pred = model.predict(x)\n",
    "print(f\"Shape: {pred.shape}\")\n",
    "print(pred[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jk_lWnScCtiR"
   },
   "source": [
    "We would like to see how good these predictions are.  We know what the correct MPG is for each car, so we can measure how close the neural network was."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vrY0Vs9oCtiR",
    "outputId": "3be93e76-a814-4349-8907-ef9125938a7e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 11.552907365195134\n"
     ]
    }
   ],
   "source": [
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y))\n",
    "print(f\"Final score (RMSE): {score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i3yqZwhnCtiS"
   },
   "source": [
    "The number printed above is the average amount that the predictions were above or below the expected output. We can also print out the first ten cars, with predictions and actual MPG."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Tl7hv8NnCtiS",
    "outputId": "742d6cac-12d6-43a9-d442-64cd8e3e4523"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Car name: chevrolet chevelle malibu, MPG: 18.0, predicted MPG: [22.539425]\n",
      "2. Car name: buick skylark 320, MPG: 15.0, predicted MPG: [27.995203]\n",
      "3. Car name: plymouth satellite, MPG: 18.0, predicted MPG: [25.851433]\n",
      "4. Car name: amc rebel sst, MPG: 16.0, predicted MPG: [25.711117]\n",
      "5. Car name: ford torino, MPG: 17.0, predicted MPG: [23.701847]\n",
      "6. Car name: ford galaxie 500, MPG: 15.0, predicted MPG: [31.893755]\n",
      "7. Car name: chevrolet impala, MPG: 14.0, predicted MPG: [35.556503]\n",
      "8. Car name: plymouth fury iii, MPG: 14.0, predicted MPG: [34.45243]\n",
      "9. Car name: pontiac catalina, MPG: 14.0, predicted MPG: [36.27014]\n",
      "10. Car name: amc ambassador dpl, MPG: 15.0, predicted MPG: [31.358776]\n"
     ]
    }
   ],
   "source": [
    "# Sample predictions\n",
    "for i in range(10):\n",
    "    print(f\"{i+1}. Car name: {cars[i]}, MPG: {y[i]}, \" \n",
    "          + f\"predicted MPG: {pred[i]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U_dpgVtfCtiS"
   },
   "source": [
    "### Simple TensorFlow Classification: Iris\n",
    "\n",
    "Classification is the process by which a neural network attempts to classify the input into one or more classes.  The simplest way of evaluating a classification network is to track the percentage of training set items that were classified incorrectly.  We typically score human results in this manner. For example, you might have taken multiple-choice exams in school in which you had to shade in a bubble for choices A, B, C, or D.  If you chose the wrong letter on a 10-question exam, you would earn a 90%.  In the same way, we can grade computers; however, most classification algorithms do not merely choose A, B, C, or D.  Computers typically report a classification as their percent confidence in each class.  Figure 3.EXAM shows how a computer and a human might both respond to question number 1 on an exam.\n",
    "\n",
    "**Figure 3.EXAM: Classification Neural Network Output**\n",
    "![Classification Neural Network Output](images/class-multi-choice.png \"Classification Neural Network Output\")\n",
    "\n",
    "As you can see, the human test taker marked the first question as \"B.\" However, the computer test taker had an 80% (0.8) confidence in \"B\" and was also somewhat sure with 10% (0.1) on \"A.\" The computer then distributed the remaining points on the other two.  In the simplest sense, the machine would get 80% of the score for this question if the correct answer were \"B.\" The computer would get only 5% (0.05) of the points if the correct answer were \"D.\" \n",
    "\n",
    "What we just saw is a straightforward example of how to perform the Iris classification using TensorFlow.  The iris.csv file is used, rather than using the built-in data that many of the Google examples require.  \n",
    "\n",
    "**Make sure that you always run previous code blocks.  If you run the code block below, without the code block above, you will get errors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aLp65T9JCtiS",
    "outputId": "cc56ca71-b7cf-41ff-afbe-76cbb465dcca"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "5/5 - 0s - loss: 1.2338\n",
      "Epoch 2/100\n",
      "5/5 - 0s - loss: 0.9754\n",
      "Epoch 3/100\n",
      "5/5 - 0s - loss: 0.9132\n",
      "Epoch 4/100\n",
      "5/5 - 0s - loss: 0.8415\n",
      "Epoch 5/100\n",
      "5/5 - 0s - loss: 0.7983\n",
      "Epoch 6/100\n",
      "5/5 - 0s - loss: 0.7550\n",
      "Epoch 7/100\n",
      "5/5 - 0s - loss: 0.7182\n",
      "Epoch 8/100\n",
      "5/5 - 0s - loss: 0.6823\n",
      "Epoch 9/100\n",
      "5/5 - 0s - loss: 0.6457\n",
      "Epoch 10/100\n",
      "5/5 - 0s - loss: 0.6140\n",
      "Epoch 11/100\n",
      "5/5 - 0s - loss: 0.5891\n",
      "Epoch 12/100\n",
      "5/5 - 0s - loss: 0.5572\n",
      "Epoch 13/100\n",
      "5/5 - 0s - loss: 0.5335\n",
      "Epoch 14/100\n",
      "5/5 - 0s - loss: 0.5083\n",
      "Epoch 15/100\n",
      "5/5 - 0s - loss: 0.4858\n",
      "Epoch 16/100\n",
      "5/5 - 0s - loss: 0.4658\n",
      "Epoch 17/100\n",
      "5/5 - 0s - loss: 0.4442\n",
      "Epoch 18/100\n",
      "5/5 - 0s - loss: 0.4237\n",
      "Epoch 19/100\n",
      "5/5 - 0s - loss: 0.4071\n",
      "Epoch 20/100\n",
      "5/5 - 0s - loss: 0.3884\n",
      "Epoch 21/100\n",
      "5/5 - 0s - loss: 0.3738\n",
      "Epoch 22/100\n",
      "5/5 - 0s - loss: 0.3598\n",
      "Epoch 23/100\n",
      "5/5 - 0s - loss: 0.3479\n",
      "Epoch 24/100\n",
      "5/5 - 0s - loss: 0.3361\n",
      "Epoch 25/100\n",
      "5/5 - 0s - loss: 0.3204\n",
      "Epoch 26/100\n",
      "5/5 - 0s - loss: 0.3102\n",
      "Epoch 27/100\n",
      "5/5 - 0s - loss: 0.2999\n",
      "Epoch 28/100\n",
      "5/5 - 0s - loss: 0.2881\n",
      "Epoch 29/100\n",
      "5/5 - 0s - loss: 0.2833\n",
      "Epoch 30/100\n",
      "5/5 - 0s - loss: 0.2747\n",
      "Epoch 31/100\n",
      "5/5 - 0s - loss: 0.2630\n",
      "Epoch 32/100\n",
      "5/5 - 0s - loss: 0.2533\n",
      "Epoch 33/100\n",
      "5/5 - 0s - loss: 0.2457\n",
      "Epoch 34/100\n",
      "5/5 - 0s - loss: 0.2399\n",
      "Epoch 35/100\n",
      "5/5 - 0s - loss: 0.2328\n",
      "Epoch 36/100\n",
      "5/5 - 0s - loss: 0.2301\n",
      "Epoch 37/100\n",
      "5/5 - 0s - loss: 0.2241\n",
      "Epoch 38/100\n",
      "5/5 - 0s - loss: 0.2138\n",
      "Epoch 39/100\n",
      "5/5 - 0s - loss: 0.2093\n",
      "Epoch 40/100\n",
      "5/5 - 0s - loss: 0.2020\n",
      "Epoch 41/100\n",
      "5/5 - 0s - loss: 0.1973\n",
      "Epoch 42/100\n",
      "5/5 - 0s - loss: 0.1912\n",
      "Epoch 43/100\n",
      "5/5 - 0s - loss: 0.1863\n",
      "Epoch 44/100\n",
      "5/5 - 0s - loss: 0.1815\n",
      "Epoch 45/100\n",
      "5/5 - 0s - loss: 0.1769\n",
      "Epoch 46/100\n",
      "5/5 - 0s - loss: 0.1740\n",
      "Epoch 47/100\n",
      "5/5 - 0s - loss: 0.1700\n",
      "Epoch 48/100\n",
      "5/5 - 0s - loss: 0.1647\n",
      "Epoch 49/100\n",
      "5/5 - 0s - loss: 0.1622\n",
      "Epoch 50/100\n",
      "5/5 - 0s - loss: 0.1580\n",
      "Epoch 51/100\n",
      "5/5 - 0s - loss: 0.1552\n",
      "Epoch 52/100\n",
      "5/5 - 0s - loss: 0.1499\n",
      "Epoch 53/100\n",
      "5/5 - 0s - loss: 0.1494\n",
      "Epoch 54/100\n",
      "5/5 - 0s - loss: 0.1453\n",
      "Epoch 55/100\n",
      "5/5 - 0s - loss: 0.1414\n",
      "Epoch 56/100\n",
      "5/5 - 0s - loss: 0.1415\n",
      "Epoch 57/100\n",
      "5/5 - 0s - loss: 0.1360\n",
      "Epoch 58/100\n",
      "5/5 - 0s - loss: 0.1332\n",
      "Epoch 59/100\n",
      "5/5 - 0s - loss: 0.1305\n",
      "Epoch 60/100\n",
      "5/5 - 0s - loss: 0.1274\n",
      "Epoch 61/100\n",
      "5/5 - 0s - loss: 0.1252\n",
      "Epoch 62/100\n",
      "5/5 - 0s - loss: 0.1230\n",
      "Epoch 63/100\n",
      "5/5 - 0s - loss: 0.1226\n",
      "Epoch 64/100\n",
      "5/5 - 0s - loss: 0.1201\n",
      "Epoch 65/100\n",
      "5/5 - 0s - loss: 0.1177\n",
      "Epoch 66/100\n",
      "5/5 - 0s - loss: 0.1180\n",
      "Epoch 67/100\n",
      "5/5 - 0s - loss: 0.1158\n",
      "Epoch 68/100\n",
      "5/5 - 0s - loss: 0.1135\n",
      "Epoch 69/100\n",
      "5/5 - 0s - loss: 0.1121\n",
      "Epoch 70/100\n",
      "5/5 - 0s - loss: 0.1213\n",
      "Epoch 71/100\n",
      "5/5 - 0s - loss: 0.1095\n",
      "Epoch 72/100\n",
      "5/5 - 0s - loss: 0.1092\n",
      "Epoch 73/100\n",
      "5/5 - 0s - loss: 0.1059\n",
      "Epoch 74/100\n",
      "5/5 - 0s - loss: 0.1060\n",
      "Epoch 75/100\n",
      "5/5 - 0s - loss: 0.1033\n",
      "Epoch 76/100\n",
      "5/5 - 0s - loss: 0.1030\n",
      "Epoch 77/100\n",
      "5/5 - 0s - loss: 0.1097\n",
      "Epoch 78/100\n",
      "5/5 - 0s - loss: 0.1020\n",
      "Epoch 79/100\n",
      "5/5 - 0s - loss: 0.1021\n",
      "Epoch 80/100\n",
      "5/5 - 0s - loss: 0.0962\n",
      "Epoch 81/100\n",
      "5/5 - 0s - loss: 0.0995\n",
      "Epoch 82/100\n",
      "5/5 - 0s - loss: 0.0985\n",
      "Epoch 83/100\n",
      "5/5 - 0s - loss: 0.0965\n",
      "Epoch 84/100\n",
      "5/5 - 0s - loss: 0.0994\n",
      "Epoch 85/100\n",
      "5/5 - 0s - loss: 0.0949\n",
      "Epoch 86/100\n",
      "5/5 - 0s - loss: 0.0949\n",
      "Epoch 87/100\n",
      "5/5 - 0s - loss: 0.0913\n",
      "Epoch 88/100\n",
      "5/5 - 0s - loss: 0.0932\n",
      "Epoch 89/100\n",
      "5/5 - 0s - loss: 0.0915\n",
      "Epoch 90/100\n",
      "5/5 - 0s - loss: 0.0924\n",
      "Epoch 91/100\n",
      "5/5 - 0s - loss: 0.0901\n",
      "Epoch 92/100\n",
      "5/5 - 0s - loss: 0.0877\n",
      "Epoch 93/100\n",
      "5/5 - 0s - loss: 0.0869\n",
      "Epoch 94/100\n",
      "5/5 - 0s - loss: 0.0883\n",
      "Epoch 95/100\n",
      "5/5 - 0s - loss: 0.0869\n",
      "Epoch 96/100\n",
      "5/5 - 0s - loss: 0.0857\n",
      "Epoch 97/100\n",
      "5/5 - 0s - loss: 0.0853\n",
      "Epoch 98/100\n",
      "5/5 - 0s - loss: 0.0843\n",
      "Epoch 99/100\n",
      "5/5 - 0s - loss: 0.0851\n",
      "Epoch 100/100\n",
      "5/5 - 0s - loss: 0.0880\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f27afbe1210>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import io\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/iris.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x = df[['sepal_l', 'sepal_w', 'petal_l', 'petal_w']].values\n",
    "dummies = pd.get_dummies(df['species']) # Classification\n",
    "species = dummies.columns\n",
    "y = dummies.values\n",
    "\n",
    "\n",
    "# Build neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(50, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(25, activation='relu')) # Hidden 2\n",
    "model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "model.fit(x,y,verbose=2,epochs=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "QPM30WdBCtiS",
    "outputId": "a02c3d4e-239f-44b2-bb1f-f8904114ac29"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['Iris-setosa', 'Iris-versicolor', 'Iris-virginica'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "# Print out number of species found:\n",
    "\n",
    "print(species)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dM8-xyDxCtiS"
   },
   "source": [
    "Now that you have a neural network training, we would like to be able to use it. The following code makes use of our neural network. Exactly like before, we will generate predictions.  Notice that three values come back for each of the 150 iris flowers.  There were three types of iris (Iris-setosa, Iris-versicolor, and Iris-virginica).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3YzlVpw-CtiS",
    "outputId": "a253eb73-5a42-4276-87f7-c287bdc52edb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (150, 3)\n",
      "[[9.9768412e-01 2.3087766e-03 7.1474560e-06]\n",
      " [9.9349666e-01 6.4763017e-03 2.6995105e-05]\n",
      " [9.9618298e-01 3.7991456e-03 1.7790366e-05]\n",
      " [9.9207532e-01 7.8882594e-03 3.6453897e-05]\n",
      " [9.9791318e-01 2.0800228e-03 6.7602941e-06]\n",
      " [9.9684995e-01 3.1442614e-03 5.8112000e-06]\n",
      " [9.9547136e-01 4.5086881e-03 1.9946103e-05]\n",
      " [9.9625921e-01 3.7288493e-03 1.2040506e-05]\n",
      " [9.9011189e-01 9.8296851e-03 5.8434536e-05]\n",
      " [9.9447203e-01 5.5067884e-03 2.1272421e-05]]\n"
     ]
    }
   ],
   "source": [
    "pred = model.predict(x)\n",
    "print(f\"Shape: {pred.shape}\")\n",
    "print(pred[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JCay-JrvCtiS"
   },
   "source": [
    "If you would like to turn of scientific notation, the following line can be used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "nXZne7ZICtiS"
   },
   "outputs": [],
   "source": [
    "np.set_printoptions(suppress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nOvMqI7QCtiS"
   },
   "source": [
    "Now we see these values rounded up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "F9PSDexjCtiS",
    "outputId": "cce7841f-4c30-45d6-f80a-ee886986844b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]]\n"
     ]
    }
   ],
   "source": [
    "print(y[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qDC7hxqECtiS"
   },
   "source": [
    "Usually, the program considers the column with the highest prediction to be the prediction of the neural network.  It is easy to convert the predictions to the expected iris species.  The argmax function finds the index of the maximum prediction for each row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "367mbx_PCtiT",
    "outputId": "ea639c6e-7ba5-404e-afb9-8921e0891dd1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predictions: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1\n",
      " 1 1 1 1 1 1 1 1 1 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2]\n",
      "Expected: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
      " 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2]\n"
     ]
    }
   ],
   "source": [
    "predict_classes = np.argmax(pred,axis=1)\n",
    "expected_classes = np.argmax(y,axis=1)\n",
    "print(f\"Predictions: {predict_classes}\")\n",
    "print(f\"Expected: {expected_classes}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lrcy0Q4xCtiT"
   },
   "source": [
    "Of course, it is straightforward to turn these indexes back into iris species. We use the species list that we created earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NTORTvygCtiT",
    "outputId": "93b065ef-9c84-4103-992c-a77fe7be8519"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',\n",
      "       'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',\n",
      "       'Iris-setosa'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "print(species[predict_classes[1:10]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ljez1ZRACtiT"
   },
   "source": [
    "Accuracy might be a more easily understood error metric.  It is essentially a test score.  For all of the iris predictions, what percent were correct?  The downside is it does not consider how confident the neural network was in each prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zth2S2OcCtiT",
    "outputId": "245de0c3-22c8-4337-c631-a2967a8d44d8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9733333333333334\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "correct = accuracy_score(expected_classes,predict_classes)\n",
    "print(f\"Accuracy: {correct}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jY07aiLICtiT"
   },
   "source": [
    "The code below performs two ad hoc predictions.  The first prediction is simply a single iris flower, and the second predicts two iris flowers.  Notice that the argmax in the second prediction requires **axis=1**?  Since we have a 2D array now, we must specify which axis to take the argmax over.  The value **axis=1** specifies we want the max column index for each row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FZUSWVGnCtiT",
    "outputId": "3a38bf92-837c-469e-a672-8c7f53894650"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.00065001 0.17222181 0.8271282 ]]\n",
      "Predict that [[5. 3. 4. 2.]] is: Iris-virginica\n"
     ]
    }
   ],
   "source": [
    "sample_flower = np.array( [[5.0,3.0,4.0,2.0]], dtype=float)\n",
    "pred = model.predict(sample_flower)\n",
    "print(pred)\n",
    "pred = np.argmax(pred)\n",
    "print(f\"Predict that {sample_flower} is: {species[pred]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ENSZaaRICtiT"
   },
   "source": [
    "You can also predict two sample flowers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fdSSkkmwCtiT",
    "outputId": "2be33dec-832b-445f-dc82-35181e7932a3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.00065001 0.17222157 0.8271284 ]\n",
      " [0.9887937  0.01117751 0.00002886]]\n",
      "Predict that these two flowers [[5.  3.  4.  2. ]\n",
      " [5.2 3.5 1.5 0.8]] \n",
      "are: Index(['Iris-virginica', 'Iris-setosa'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "sample_flower = np.array( [[5.0,3.0,4.0,2.0],[5.2,3.5,1.5,0.8]],\\\n",
    "        dtype=float)\n",
    "pred = model.predict(sample_flower)\n",
    "print(pred)\n",
    "pred = np.argmax(pred,axis=1)\n",
    "print(f\"Predict that these two flowers {sample_flower} \")\n",
    "print(f\"are: {species[pred]}\")"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "colab": {
   "collapsed_sections": [],
   "name": "Copy of t81_558_class_03_2_keras.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3.9 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}