CIFAR10_CNN.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# CIFAR10_CNN\n",
        "An explatory notebook for training deep convolutional neural networks on the [CIFAR10 dataset](https://www.cs.toronto.edu/~kriz/cifar.html) to achieve a 90\\%< accuracy. Multiple image augementation techniques, learning rate curves, and architectures are tested."
      ],
      "metadata": {
        "id": "bgvbhB9ys5TM"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Imports"
      ],
      "metadata": {
        "id": "aW1IHUrBuFJK"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9QiYoJbHj-h7"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow_datasets as tfds\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Model Parameters"
      ],
      "metadata": {
        "id": "SEUpxXH1uHoJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "AUTOTUNE = tf.data.AUTOTUNE\n",
        "BATCH_SIZE = 128\n",
        "LEARNING_RATE = 80 ## This is used as LEARNING_RATE/BATCH_SIZE for a custom learning rate curve with SGD optmizer\n",
        "WARMUP =  10 ## number of epochs for the warmup phase\n",
        "WEIGHT_DECAY = 0.000125 ## SGD weight decay\n",
        "EPOCHS = 35"
      ],
      "metadata": {
        "id": "Yd6jnqcnejh2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load Dataset"
      ],
      "metadata": {
        "id": "p3A2oz2Wue0w"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ds_train = tfds.load('cifar10', split='train', shuffle_files=True, as_supervised=True)\n",
        "n_train = ds_train.cardinality().numpy() ## Total number of training images\n",
        "ds_test = tfds.load('cifar10', split='test', as_supervised=True)\n",
        "\n",
        "for image, label in ds_train:\n",
        "  IMG_SHAPE = image.shape ## Set the image shape\n",
        "  break"
      ],
      "metadata": {
        "id": "_uPekqDHkLDV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Preprocessing and Augmentation\n",
        "All the images are cast to `float32` data type and rescaled to 0 to 1 scale using `resize_rescale`.\n",
        "Implementations of `ReflectionPadding2D` and `masking` are also in this cell. These are used for data augmentation later on."
      ],
      "metadata": {
        "id": "0LUV1YRzupiM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def resize_rescale(image):\n",
        "    image = tf.cast(image, dtype=tf.float32)\n",
        "    return image/255.\n",
        "\n",
        "class ReflectionPadding2D(tf.keras.layers.Layer):\n",
        "  def __init__(self, pad=(1, 1)):\n",
        "    super(ReflectionPadding2D, self).__init__()\n",
        "    self.pad = pad\n",
        "\n",
        "  def __call__(self, x):\n",
        "    w_pad,h_pad = self.pad\n",
        "    return tf.pad(x, [[h_pad,h_pad], [w_pad,w_pad], [0,0]], mode='REFLECT')\n",
        "\n",
        "  \n",
        "def replace_slice(orig, replacement, begin):\n",
        "  inp_shape = tf.shape(orig) \n",
        "  rep_size = tf.shape(replacement)\n",
        "  padding = tf.stack([tf.squeeze(begin), tf.squeeze((inp_shape - (begin + rep_size)))], axis=1)\n",
        "  replacement_pad = tf.pad(replacement, padding) \n",
        "  mask = tf.pad(tf.ones_like(replacement, dtype=tf.bool), padding)\n",
        "  return tf.where(mask, replacement_pad, orig)\n",
        "  \n",
        "def masking(x, mask=(1, 1)):\n",
        "  w_mask,h_mask = mask\n",
        "  x = tf.expand_dims(x, 0)\n",
        "  shape = tf.shape(x)\n",
        "  x0 = tf.random.uniform([shape[0],], 0, shape[1] + 1 - h_mask, dtype=tf.int32)\n",
        "  y0 = tf.random.uniform([shape[0],], 0, shape[2] + 1 - w_mask, dtype=tf.int32)\n",
        "  begin = tf.pad(tf.stack([x0, y0], axis=1), [[0, 0], [1, 1]])\n",
        "  x = replace_slice(x, tf.zeros([shape[0], h_mask, w_mask, shape[-1]], dtype=tf.float32), begin)\n",
        "  return x[0]"
      ],
      "metadata": {
        "id": "TEBLDpmLeEGf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The function to be called on the training dataset:\n",
        "1. Resize and normalize\n",
        "2. Shuffle the training data\n",
        "3. Augmentations:\n",
        "  1. Reflection padding to prepare for random cropping\n",
        "  2. Random cropping\n",
        "  3. Random flipping (only horizontal)\n",
        "  4. Random cutouts: Randomly cutout an 8x8 section of the image\n",
        "4. Batch the dataset"
      ],
      "metadata": {
        "id": "buKfmEAS8j1h"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def prepare_data(data, shuffle=False, augment=False):\n",
        "    # resize and normalize\n",
        "    data = data.map(lambda x, y: (resize_rescale(x), y), num_parallel_calls=AUTOTUNE)\n",
        "\n",
        "    #shuffle\n",
        "    if shuffle:\n",
        "      data = data.shuffle(BATCH_SIZE)\n",
        "\n",
        "    #augment\n",
        "    augments  = tf.keras.Sequential([\n",
        "               ReflectionPadding2D((4, 4)),\n",
        "               tf.keras.layers.RandomCrop(IMG_SHAPE[0], IMG_SHAPE[1], IMG_SHAPE[2]),\n",
        "               tf.keras.layers.RandomFlip(\"horizontal\"),\n",
        "               ])\n",
        "    if augment:\n",
        "      data = data.map(lambda x, y: (augments(x, training=True), y), \n",
        "                num_parallel_calls=AUTOTUNE)\n",
        "      data = data.map(lambda x, y: (masking(x, mask=(8, 8)), y), \n",
        "                num_parallel_calls=AUTOTUNE)\n",
        "      \n",
        "    return data.batch(BATCH_SIZE).prefetch(buffer_size=AUTOTUNE)"
      ],
      "metadata": {
        "id": "TFiNJA3RvYKu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create the (augmented) train and test datasets"
      ],
      "metadata": {
        "id": "FNO0sZvF9hYd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ds_train = prepare_data(ds_train, shuffle=True, augment=True)\n",
        "\n",
        "ds_test = prepare_data(ds_test, shuffle=False, augment=False)"
      ],
      "metadata": {
        "id": "sLAi-_5IwjVT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Visualize a training set data point"
      ],
      "metadata": {
        "id": "yv0J0ISf-OuK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for image, label in ds_train:\n",
        "  print(label[0])\n",
        "  plt.imshow(image[1])\n",
        "  break"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "id": "wgluZzJTl5Sw",
        "outputId": "5408dc28-9aa0-49d6-97bf-15fe0af67e9f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tf.Tensor(7, shape=(), dtype=int64)\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD5CAYAAADhukOtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAcq0lEQVR4nO2da4yd13We33WucyWHFCmKInWhLpbkyJXEMqoFq4GTwIEqBJANFIb9w9APIwqKGKiBFKjgArUKFIVd1Db8o3BB10KUwvUlsR0LhdvGVozKSQzKlCtRlJTYIkVKooYcXodzOfdv9cc5RCllv2uGnJkzFPf7AATP7HX2t/fZ37fOd85+z1rL3B1CiKuf0npPQAgxHOTsQmSCnF2ITJCzC5EJcnYhMkHOLkQmVFbS2cweAvBVAGUA/9XdvxA9f3Jywrds2UyOxadiZul29GgfRyAphmojNxbEZFamfUqBzYP5A+nX3B+P2+hYzseKjxZZ+Wvj61jwHoEMXLDFB9Dr8WN2ul02GO1TLvN7YKdDjgegPjpKbRPjk/yY7VayvVqr0z7zC3PJ9jOnT2N+fj550i7b2a1/hf9nAB8B8BaAX5jZ0+7+CuuzZctmPPH5f520lavX0LFG6+lplnvpFwwAXXSorSiCC9/5hdMkJ7pc2Uj7jNS5rRPMv1TiF1y1WqM2J07R6yzQPpVK9O5XpZaiO0Ft5Up6HYtikfZpt7gjtRptajs7x1/b9MkzyfZywceaHB+jtrePH6e2O+6+l9oeuP+fUtv0m4eT7dtvuIX2+flzP0u2f/GL/4H2WcnH+PsBvObuh929DeDbAB5ZwfGEEGvISpx9B4A3L/r7rUGbEOIKZM036MzsMTPbb2b75+bm13o4IQRhJc5+DMANF/29c9D2Dtx9r7vvcfc9k5P8O54QYm1ZibP/AsDtZrbLzGoAPgHg6dWZlhBitbns3Xh375rZZwD8b/Q1mCfd/eW4VwleSssJzVBqSu8WV4zvnEeiVrfL3+NGjO8+V6vpuZer/BNLqczlGOvynWnykvsEtqKXfuWBOoVOI1AgmvyrV7M5yw9KVI3ZubO0y/wC31Vvd/hufDd4cdZL77rXgwW2Dj8vN25KS8cAcNsHdlPbhuvvoLaJ7WlbKVA9b/tAeq1GRrmSsCKd3d1/BOBHKzmGEGI46Bd0QmSCnF2ITJCzC5EJcnYhMkHOLkQmrGg3/nIoF2k9YXSUv+94t5lst24guZBxAMDbXHZpFVzi6VlaxvESD6potmaorUEilwCg2+GBPBZEbHXa6fm3A5mv6PG1KgXrWETSIVmTahAFuKnEL8eRcmCrjVDbWCl9jdTJuQSAReNS6vbdv01tO3fdQ22tTrDG5fT57HR5n5vJWLUal950ZxciE+TsQmSCnF2ITJCzC5EJcnYhMmGou/GdTgfHj/+DKFgAQFGk2wGgS4I7ovRSjUZ6Bx8AFprc1iK72QDfIQ+ztAXppUpBrrPRCg/IGStx2wTJW7aJZ7JCvcZfwUSZdxwt8SCfGnlppSAApRQF/wTBLkX30vMNLnCxA9WdN1Lbdb9xP59HJ0jhFaQ765FJWhDNVRRkLA92/fnhhBBXE3J2ITJBzi5EJsjZhcgEObsQmSBnFyIThiq9NRpNvHjwV0lbucIlg2olHTwRBVXUK1HgBLdNjfGSO6PldIDERCXqw+dYDbSVaon3q4XHTNuifGaRLSqj1SuCxHbkpfWCMk6taKzwBQQmklOwvnk77XNnUL1lYmKc2todfj57gUDb9fQ5q5T5+pbJArN8jYDu7EJkg5xdiEyQswuRCXJ2ITJBzi5EJsjZhciEFUlvZnYEwBz6QkvX3fdEzx8dHcE9d78vaYtUnGFGvc0utvg8OulSSO+FqLeRGl+rejCPOOotkgfT7UEqOVSDiK0w6i2Q8+DpXHMLM0dplxee+xm13b1pJ7WVSlyW6wZRb15On5tOkBuwQiRWD9ZwNXT233b3U6twHCHEGqKP8UJkwkqd3QH8pZk9b2aPrcaEhBBrw0o/xj/o7sfM7FoAPzazv3P3Zy9+wuBN4DEAmJrauMLhhBCXy4ru7O5+bPD/DIAfAPgHOXvcfa+773H3PePjfANDCLG2XLazm9m4mU1eeAzg9wAcXK2JCSFWl5V8jN8G4AdmduE4/93d/1fUoVqt4rrrdiRttfENtJ8XaamsRMpCAaFSg3abG0uBnMfLP3F5qtniYw2z/NPZqPxTUA6rFJTDWu3yT6NR+SeLSkNFMiUp/8S7YHHmDWo7/vJz1LZz94PUVurx+TMFswhE3VKZXB9B1NtlO7u7HwbAi1sJIa4oJL0JkQlydiEyQc4uRCbI2YXIBDm7EJkw1ISTANAjBb3mAjlsdGQk2W7GZSFnGQ8RR6nVjctoBZHYytUp2mfTpnSSSgBot05TWxAsh0qV60a9blry6gYyZUGiCgGg2eQSYLMZFCMjUV6zc2dpl7MLC9TW7gQ1+NqBTNlLr8dEIFGNEYkVAI7ve4bPY9Mktd16625qY0F71Sqf46FDLybb220uh+rOLkQmyNmFyAQ5uxCZIGcXIhPk7EJkwpB34wtYkc7xNkJKKwHACMmrVXb+XtUNduMrFb7zbyRnGQB0Omlb4XwZy0Hghwfln9yC9+GoEhIpDRVUEkK9FuTCG+XnpehyW5ms8XXFJtqn3eJr32rw3fizc3wXf/rkmWR7t+Bj+fgYtb1x/Di1jb70S2rbOsrDu6ffPJxs337DLbTPa2SsZkO78UJkj5xdiEyQswuRCXJ2ITJBzi5EJsjZhciE4Upv7vBOI2kaqaVLKwGAdck0jWtQ1SigJZC8ikCyq5LhSuAlo3ptXiynBB7oUA5OTbXgNiOSXYPIhgBQeBDQEtwOegjkK9JxMcjJ9/qhaWo7Mc2Dhs61gjJg3fS5uWMrz3l4y403UFtlNF1eCwAai1wC/LtX9lFb8/ibyfbZuZOXPJYHddR0ZxciE+TsQmSCnF2ITJCzC5EJcnYhMkHOLkQmLCm9mdmTAH4fwIy73z1o2wzgOwBuBnAEwMfdnScXu4A7jMg87RaX3srl9DTrdS6DRKFhjiAELKBMIspKQcK4KJdct8clu4LkTgOAdiDZlUiZpFIQRWfBenSd564rV/nlc2omfcy/eubntM/RN3jZpUgd7ARRe+Wx9Bytya+3O+66jdrqZT6RN6b5/CfG+Rw3b05HAp5ZmKV9TpxIj9UJcvUt587+JwAeelfb4wCecffbATwz+FsIcQWzpLMP6q2/Oyj4EQBPDR4/BeCjqzwvIcQqc7nf2be5+4WfOx1Hv6KrEOIKZsUbdO7uAP8SaWaPmdl+M9s/N89/TiiEWFsu19lPmNl2ABj8P8Oe6O573X2Pu++ZnOCpeYQQa8vlOvvTAB4dPH4UwA9XZzpCiLViOdLbtwB8GMAWM3sLwOcBfAHAd83s0wCOAvj4cgazchX1ifTXeyZrAUC3m5Y7ijKX3qLjFeR4AFAq8/e/KikZdH6Wq46tVjrKDwDG6ryM0xiRjACgUg4yThIVzSuBPBhIaEFgG46+fozanv0/6fJEv37tLdpnpJ4u8wUAvS4v8dQOzmd9dGOy/dR5Lr3NnTtPbV7i0la9Hsi9gXZ4cjadJLIURNixsYJA0KWd3d0/SUy/u1RfIcSVg35BJ0QmyNmFyAQ5uxCZIGcXIhPk7EJkwlATTlbrE7j29g8lbVbimkGZhI51OlyOOXuWy2FTU9dQ2+gIl3+6nXQU0slFXuOrFcxxkUguALC1wtdjvM6j3szTtlYgT82f5/M4fIjLa3/7swPUNn0i/Tur2sgo7VOtcqlpPogAa3S5vFnaPJlsbwa13lotLq/VJrlcCiLNAsDCAl/jQ6+9nmy/9bZdlz5WVAeQm4QQVxNydiEyQc4uRCbI2YXIBDm7EJkgZxciE4YqvVmphOpYusbWqdMnaL/XDv0q2X7nHXfSPm+f5nWy3nz7bWo7d4LP4/rr0pLduTNc5rOoHhqRyQDgjbd53bMtm6eorWpp+eqNozTlAA4c+DW1zRznr212lsth9ZF0rb16IL31ejy55eQGfqleO7aV2t6eS0e3+RRfw9dPnqO2LT0uD7YW+fybI9x2tpFO6tJs8z5sLNV6E0LI2YXIBTm7EJkgZxciE+TsQmTCUHfjAUepSAcZmPFAjQqZZb3Kf/X/vptvpLa/+aufUNuBfbw80cFSOq9d0eM7oBs38F3f0ckJamsG5YmmJ3lKbpbq7PQpvqveWODzL5d5YNDoKA/ycRK8VKmO0T7z83wX/L7fvIPabr+Nn+u/+MnfJtvPByW0jpx5d02U/888j2dBteBGL6XVCQAwUt7s3Lk52uccyZPX62o3XojskbMLkQlydiEyQc4uRCbI2YXIBDm7EJmwnPJPTwL4fQAz7n73oO0JAH8A4EK0yefc/UdLHatb9HB6MS1rVEd5bq8770oHvHSbLdrn2OtvUNvJt45TW9Hmksyxk+nglEaD50DbvJnnu7tmEw/g6HZ5AM3MMS5RsfI/UTmsEpHJAKBa5TYDL9TZ8fREWm2e361c5bLR5AYuAbYKLlPefseOZPvzLx2lfU6f4IFSY1M8EGbLNVxmXVjk18j8bHr+C2NcpqzW0vOIcjku587+JwAeSrR/xd3vHfxb0tGFEOvLks7u7s8C4L8yEEK8J1jJd/bPmNkBM3vSzDat2oyEEGvC5Tr71wDcCuBeANMAvsSeaGaPmdl+M9t//hz/yaYQYm25LGd39xPu3nP3AsDXAdwfPHevu+9x9z0bpvQBQIj14rKc3cy2X/TnxwAcXJ3pCCHWiuVIb98C8GEAW8zsLQCfB/BhM7sXgAM4AuAPlzNYs7GIV15JlwyqVbj0dvp4Op9cN5AzTk9zee3tI1yWOz3Dc7Wdm01LXl5wuS7KM9fucBmq2eT5x6KyV0a0twoLHQz6LEWtziO50E2/7maHy4bXbufH2zjFbZUyX+O7dqWlt+55Lm0WXb6+u+97H7X1enwdZ07yPe4770xLy61FHvV27dZtyfZqlfvRks7u7p9MNH9jqX5CiCsL/YJOiEyQswuRCXJ2ITJBzi5EJsjZhciEoSacbCzM4+V9zyZt7YJHZc3PpWWoUhAZ1p7jssWZk6eobW6BR1B1SbLMcpC8EAWX0NDj87dAzrOgxE9BbJ0eT+hZsijqjV8i5SBaDkjLUBs3bqQ9dt20mdrGA5kvkps6nfR63HhdugwZADQ6PJryzCy/ds4GST3LJR61t3v3bybbf/43P6V9jhxJz6Pd5nPXnV2ITJCzC5EJcnYhMkHOLkQmyNmFyAQ5uxCZMFTprdNZxLHpF5O2yU2jtN/ma9Jx8Ef/nkeoLcxwqSmKToIFkhd5a7SCj9Vp8/pfCwuzfB4kYSMAeDCeE+mt1+Ovi8l1ANBs8H5jQULEAmk5bCRILFolfQBg9iSXUitBxCRIHbXxUX69TQSvqxdImJUpnlwUzqXltw4dSrZvnuTyICyd7LNCXi+gO7sQ2SBnFyIT5OxCZIKcXYhMkLMLkQlD3Y0fHx/HA//kHydtU5t45tnGQjon2Osv8jI9jXm+Cz4yOkFt5SBXWwlkp7vNd2jPBwE5rRYPhKlVeeBHlDOuCHLeMTpBSaZo9xkW5HEju88bJvl5vvWmm6htrM4DSQC+090p0nNcXODXR6vFg0k2bZyktl43UGWCvHYg56wWrFWVKBCVMlcmdGcXIhPk7EJkgpxdiEyQswuRCXJ2ITJBzi5EJiyn/NMNAP4UwDb0yz3tdfevmtlmAN8BcDP6JaA+7u5hmdZuu4eZw+ngjzN1Lv8sNtK22Vle/qntPPdb1erUVinxJWEyVCvIhdcNcoJZIBmF8tpl5KBjUs1Sx7MgPx3LMwcAC/Pp8zx201baZ3ycB6BYEBhUBLYqOZ/RWPV6IF+V+DmLrp1ajV9zXSLLRQE+7BWXgutmOXf2LoA/dvf3A/gggD8ys/cDeBzAM+5+O4BnBn8LIa5QlnR2d592918OHs8BeBXADgCPAHhq8LSnAHx0rSYphFg5l/Sd3cxuBnAfgH0Atrn79MB0HP2P+UKIK5RlO7uZTQD4HoDPuvv5i23er0uc/M2fmT1mZvvNbH+zEeRQF0KsKctydjOrou/o33T37w+aT5jZ9oF9O4Bk2hh33+vue9x9z8ho9PtmIcRasqSzW39b+BsAXnX3L19kehrAo4PHjwL44epPTwixWiwn6u1DAD4F4CUze2HQ9jkAXwDwXTP7NICjAD6+1IHKpTI2bkhH8owEOcGmJtOS10tBKrmDr6fzel0N/MYttwfW9KJEEVlRfjry7axvCSLsmAS0c+f1tE+o8gXyWq3MIwS7RVrWqgXRYdUad4s4qJDLcuVAsltcTEvIY2PcJ3okD6GV+Dot6ezu/tfgst7vLtVfCHFloF/QCZEJcnYhMkHOLkQmyNmFyAQ5uxCZMNSEk5VKFds2XZe01epcmjg3m46gKvFgrauaO+64ldo6JALPe3x920Hiy2NvH6O2ZpP/IvKeez6QbL/rrrton1qV61rlMBqRmmDkGolkw0DRDYkiFXs9vsYlcsuN+7D79Mqi3oQQVwFydiEyQc4uRCbI2YXIBDm7EJkgZxciE4YqvRmAkqUlDyN11AAAnn5P6jmPXLqaefjhj1Abk5RKBZeazp7ldc/+4un/SW2nzp2itl27tifbR2v8/hLVlesFtdKMylBAhUUBBhJVJL0VQRRg0eVJU6N6euVyWhbtNPl5qdVIpJ9zP9KdXYhMkLMLkQlydiEyQc4uRCbI2YXIhKHuxsOAciW981it8qnQH/0bD+64milI/jGA7+xWgtxk0U53p8Nt1Spf/61br0m2F0FwR0RYoiqwdckOP1snAOgG5byi9YgIYmTQJiXCKhXuE6xklAdqge7sQmSCnF2ITJCzC5EJcnYhMkHOLkQmyNmFyIQlpTczuwHAn6JfktkB7HX3r5rZEwD+AMDJwVM/5+4/io7l7miTgIAoB10kQeRIp83ln16J5KALgkUWFnnARSuQmqY2pUt5AcD42FiyneXIWwqecw1oBLnwWuR6q9d5ySgL6lD1IpkvLLHFbSx4qRMcj0mHHgQ8LceLugD+2N1/aWaTAJ43sx8PbF9x9/+0jGMIIdaZ5dR6mwYwPXg8Z2avAtix1hMTQqwul/Sd3cxuBnAfgH2Dps+Y2QEze9LM+Gc6IcS6s2xnN7MJAN8D8Fl3Pw/gawBuBXAv+nf+L5F+j5nZfjPbv7DAvxsKIdaWZTm7mVXRd/Rvuvv3AcDdT7h7z90LAF8HcH+qr7vvdfc97r5nfDy9aSOEWHuWdHbrl7n4BoBX3f3LF7VfnHfoYwAOrv70hBCrxXJ24z8E4FMAXjKzFwZtnwPwSTO7F3057giAP1zySM5lhllS4gkAeqR0UbVaX3LIq5FOIDXV6+k14YIMcHb+PLU1Cy6V3XRNOrIN4Oc5KhkVyWtR1Fs7iNprE6lvvsG/UkYRcaMjo9TWDdIoLizw180iQXuB9DY+MZ5sj87zcnbj/xrpHHyhpi6EuLLQL+iEyAQ5uxCZIGcXIhPk7EJkgpxdiEwYajhZFPVWqfKMfHPz88n2douX1LmaabXSCQoBXrqoNMJlyjDBYo/LWuPjafkHAEqltJzUbgcRasHrKgUZG4sgmSMrDRVFUlaC5KftIAqwCCZSqfIoO7b+UR8WfRdJb7qzC5EJcnYhMkHOLkQmyNmFyAQ5uxCZIGcXIhOGLr2xWlkeRAy1mmlJZr6xsBrTes/BZK2IKNHjQoPLYWNjE9S2aWoDtXmRPs9R4kUmywKxPFgjkX4AUK5Vk+0WSHmlIOFk4Xz+RY8LX70ef209Im+acacolUbShkB7051diEyQswuRCXJ2ITJBzi5EJsjZhcgEObsQmTBU6a3wgiYcrBjXDDokoWDPuAyyffsN1LY1qFH24AMPUNvNN25OtneaXAIsIS39AMD4OJe1uj0uNUUSVZP08+Bt/dTpc9RWqxKJB8DEGE++2G41ku2LQV25KOFkpcwv1V7Br4PmAomyDKLeovUNhkKjwfs5uK1aTV8ji410tCcAGBkrkjZ1ZxciE+TsQmSCnF2ITJCzC5EJcnYhMmHJ3XgzGwHwLID64Pl/7u6fN7NdAL4N4BoAzwP4lLvHSeGcl/EpgredxmJ6Zzcqj8PKIAHA+BgvMHly5iS1XX9devc82gEdq/PdbA+iFsId4aAUkpOIokZQfmhmZobamu2g7FKb54xj+eRaYd5Avh5jwTlDENTSI2WoWHkqADRYCwAaDf6aZ8/NUdvoGL8e67VJMhbfjW8spkt2RQFDy7mztwD8jrvfg3555ofM7IMAvgjgK+5+G4CzAD69jGMJIdaJJZ3d+1x4i6kO/jmA3wHw54P2pwB8dE1mKIRYFZZbn708qOA6A+DHAA4BOOfuFz4zvAVgx9pMUQixGizL2d295+73AtgJ4H4Ady53ADN7zMz2m9n+xUb6u7cQYu25pN14dz8H4KcAHgAwZWYXNvh2AjhG+ux19z3uvmdslP+8Ugixtizp7Ga21cymBo9HAXwEwKvoO/0/HzztUQA/XKtJCiFWznICYbYDeMrMyui/OXzX3f+Hmb0C4Ntm9u8B/F8A31jqQA6nMlWvE+UmIzm6grHqFZ6nrR3kXHv98JvUdsOOa5PtWzbxMkgLCzzwI1AOYcH7sAWvnOVjOzM3S/t02lyuaS1y6e1sIDVdv21Lsr1kl54/DwC6nSBJYVQaqkuk3kC+jHL8VUvcZaY2pCU0AKjV+TGtSB9zcpx/Eq5X0qWhKmU+zpLO7u4HANyXaD+M/vd3IcR7AP2CTohMkLMLkQlydiEyQc4uRCbI2YXIBIuif1Z9MLOTAI4O/twC4NTQBudoHu9E83gn77V53OTuW1OGoTr7OwY22+/ue9ZlcM1D88hwHvoYL0QmyNmFyIT1dPa96zj2xWge70TzeCdXzTzW7Tu7EGK46GO8EJmwLs5uZg+Z2d+b2Wtm9vh6zGEwjyNm9pKZvWBm+4c47pNmNmNmBy9q22xmPzazXw/+5zWq1nYeT5jZscGavGBmDw9hHjeY2U/N7BUze9nM/uWgfahrEsxjqGtiZiNm9pyZvTiYx78btO8ys30Dv/mOmaVD3xjuPtR/AMrop7W6BUANwIsA3j/seQzmcgTAlnUY97cA7AZw8KK2/wjg8cHjxwF8cZ3m8QSAfzXk9dgOYPfg8SSAXwF4/7DXJJjHUNcE/ejticHjKoB9AD4I4LsAPjFo/y8A/sWlHHc97uz3A3jN3Q97P/X0twE8sg7zWDfc/VkAZ97V/Aj6iTuBISXwJPMYOu4+7e6/HDyeQz85yg4MeU2CeQwV77PqSV7Xw9l3ALg4Q8R6Jqt0AH9pZs+b2WPrNIcLbHP36cHj4wC2reNcPmNmBwYf89f868TFmNnN6OdP2Id1XJN3zQMY8pqsRZLX3DfoHnT33QD+GYA/MrPfWu8JAf13dkQVE9aWrwG4Ff0aAdMAvjSsgc1sAsD3AHzW3d9RBWGYa5KYx9DXxFeQ5JWxHs5+DMDFxdNpssq1xt2PDf6fAfADrG/mnRNmth0ABv/zMi1riLufGFxoBYCvY0hrYmZV9B3sm+7+/UHz0NckNY/1WpPB2Jec5JWxHs7+CwC3D3YWawA+AeDpYU/CzMbNbPLCYwC/B+Bg3GtNeRr9xJ3AOibwvOBcAz6GIayJmRn6OQxfdfcvX2Qa6pqweQx7TdYsyeuwdhjftdv4MPo7nYcA/Jt1msMt6CsBLwJ4eZjzAPAt9D8OdtD/7vVp9GvmPQPg1wB+AmDzOs3jvwF4CcAB9J1t+xDm8SD6H9EPAHhh8O/hYa9JMI+hrgmAf4R+EtcD6L+x/NuLrtnnALwG4M8A1C/luPoFnRCZkPsGnRDZIGcXIhPk7EJkgpxdiEyQswuRCXJ2ITJBzi5EJsjZhciE/wdkaj0/59iWBAAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Define steps per epoch and total __training__ steps"
      ],
      "metadata": {
        "id": "ST5MQWKq-QQH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "steps_per_epoch = n_train // BATCH_SIZE\n",
        "total_steps = steps_per_epoch * EPOCHS"
      ],
      "metadata": {
        "id": "InRA8lIr9Tbu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## First network\n",
        "A straight forward CNN with 4 blocks of convolutions and maxpooling. `Dropout` units are also added to prevent overfitting. The head unit is a 128 unit dense layer followed by a 10-unit output."
      ],
      "metadata": {
        "id": "iefmgUPD-ZJi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class CIFAR_CNN(tf.keras.Model):\n",
        "  def __init__(self):\n",
        "    super(CIFAR_CNN, self).__init__()\n",
        "    self.c1 = tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=1, padding='same', activation='elu', kernel_regularizer=tf.keras.regularizers.l2(1e-4))\n",
        "    self.bn1 = tf.keras.layers.BatchNormalization()\n",
        "    self.c2 = tf.keras.layers.Conv2D(filters=32, kernel_size=3, strides=1, padding='same', activation='elu', kernel_regularizer=tf.keras.regularizers.l2(1e-4))\n",
        "    self.bn2 = tf.keras.layers.BatchNormalization()\n",
        "    self.mp1 = tf.keras.layers.MaxPooling2D(pool_size=2)\n",
        "    self.d1 = tf.keras.layers.Dropout(0.2)\n",
        "\n",
        "    self.c3 = tf.keras.layers.Conv2D(filters=64, kernel_size=3, strides=1, padding='same', activation='elu', kernel_regularizer=tf.keras.regularizers.l2(1e-4))\n",
        "    self.bn3 = tf.keras.layers.BatchNormalization()\n",
        "    self.c4 = tf.keras.layers.Conv2D(filters=64, kernel_size=3, strides=1, padding='same', activation='elu', kernel_regularizer=tf.keras.regularizers.l2(1e-4))\n",
        "    self.bn4 = tf.keras.layers.BatchNormalization()\n",
        "    self.mp2 = tf.keras.layers.MaxPooling2D(pool_size=2)\n",
        "    self.d2 = tf.keras.layers.Dropout(0.3)\n",
        "\n",
        "    self.c5 = tf.keras.layers.Conv2D(filters=128, kernel_size=3, strides=1, padding='same', activation='elu', kernel_regularizer=tf.keras.regularizers.l2(1e-4))\n",
        "    self.bn5 = tf.keras.layers.BatchNormalization()\n",
        "    self.c6 = tf.keras.layers.Conv2D(filters=128, kernel_size=3, strides=1, padding='same', activation='elu', kernel_regularizer=tf.keras.regularizers.l2(1e-4))\n",
        "    self.bn6 = tf.keras.layers.BatchNormalization()\n",
        "    self.mp3 = tf.keras.layers.MaxPooling2D(pool_size=2)\n",
        "    self.d3 = tf.keras.layers.Dropout(0.4)\n",
        "\n",
        "    self.c7 = tf.keras.layers.Conv2D(filters=256, kernel_size=3, strides=1, padding='same', activation='elu', kernel_regularizer=tf.keras.regularizers.l2(1e-4))\n",
        "    self.bn7 = tf.keras.layers.BatchNormalization()\n",
        "    self.c8 = tf.keras.layers.Conv2D(filters=256, kernel_size=3, strides=1, padding='same', activation='elu', kernel_regularizer=tf.keras.regularizers.l2(1e-4))\n",
        "    self.bn8 = tf.keras.layers.BatchNormalization()\n",
        "    self.mp4 = tf.keras.layers.MaxPooling2D(pool_size=2)\n",
        "    self.d4 = tf.keras.layers.Dropout(0.5)\n",
        "\n",
        "    self.flat = tf.keras.layers.Flatten()\n",
        "    self.nn1 = tf.keras.layers.Dense(units=128, activation='relu')\n",
        "    self.d4 = tf.keras.layers.Dropout(0.5)\n",
        "    self.output_ = tf.keras.layers.Dense(10, activation='softmax')\n",
        "\n",
        "    self.model = None\n",
        "    self.history = None\n",
        "  \n",
        "  def call(self, input_):\n",
        "    x = self.c1(input_)\n",
        "    x = self.bn1(x)\n",
        "    x = self.c2(x)\n",
        "    x = self.bn2(x)\n",
        "    x = self.mp1(x)\n",
        "    x = self.d1(x)\n",
        "\n",
        "    x = self.c3(x)\n",
        "    x = self.bn3(x)\n",
        "    x = self.c4(x)\n",
        "    x = self.bn4(x)\n",
        "    x = self.mp2(x)\n",
        "    x = self.d2(x)\n",
        "\n",
        "    x = self.c5(x)\n",
        "    x = self.bn5(x)\n",
        "    x = self.c6(x)\n",
        "    x = self.bn6(x)\n",
        "    x = self.mp3(x)\n",
        "    x = self.d3(x)\n",
        "\n",
        "    x = self.c7(x)\n",
        "    x = self.bn7(x)\n",
        "    x = self.c8(x)\n",
        "    x = self.bn8(x)\n",
        "    x = self.mp4(x)\n",
        "    x = self.d4(x)\n",
        "\n",
        "    x = self.flat(x)\n",
        "    x = self.nn1(x)\n",
        "    x = self.d4(x)\n",
        "    x = self.output_(x)\n",
        "    return x"
      ],
      "metadata": {
        "id": "NzrNjQMgmeZ7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Learning rate scheduler\n",
        "A simple linear decay for the learning rate with a linear increase for the warmup period. Warmup prevents early over-fitting caused by clustered input images."
      ],
      "metadata": {
        "id": "lPJRhBdTA-08"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class MyScheduler(tf.keras.optimizers.schedules.LearningRateSchedule):\n",
        "\n",
        "  def __init__(self):\n",
        "    self.initial_learning_rate = 0.01\n",
        "\n",
        "  def __call__(self, step):\n",
        "    warmup_steps = steps_per_epoch * WARMUP\n",
        "    is_warmup = tf.cast(step < warmup_steps, tf.float32)\n",
        "    decay_lr = tf.keras.optimizers.schedules.PolynomialDecay(self.initial_learning_rate, (total_steps-(steps_per_epoch * WARMUP)), end_learning_rate=0.0, power=1, cycle=False)(step-(steps_per_epoch * WARMUP))\n",
        "    warmup_lr = self.initial_learning_rate * step / warmup_steps\n",
        "    return (1.0 - is_warmup) * decay_lr + is_warmup * warmup_lr"
      ],
      "metadata": {
        "id": "tk6K_5CU84sY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Plot of the learning rate curve"
      ],
      "metadata": {
        "id": "lztd_x7UBGXQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "xs = list(range(total_steps))\n",
        "ys = [MyScheduler()(i) for i in xs]\n",
        "plt.plot(xs, ys)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "3wuJVpXu9JX-",
        "outputId": "eb1a3b3e-4dfa-4f78-93c8-067c1fa9cb5e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f479a517f10>]"
            ]
          },
          "metadata": {},
          "execution_count": 64
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAD4CAYAAAD2FnFTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dfXxU9Zn38c+VhCSGAJIQnvMAglRALRieabdWu6KtYlddUbBorLRd7XZrd7t6733vdt1tu267265brVoTRGsFBNtlrVWrVmt4koCg8miaSYAYMUEIAjEhye/+Y050CAMZwiRnHr7v1ysvzpzzOzPXOSFzzcw53znmnENERKSzFL8LEBGR2KQGISIiYalBiIhIWGoQIiISlhqEiIiEleZ3Aadj0KBBrqioyO8yRETixsaNGxucc3ndWTeuGkRRUREVFRV+lyEiEjfMrKa76+ojJhERCUsNQkREwlKDEBGRsNQgREQkLDUIEREJK6IGYWZzzGynmVWa2V1hlmeY2TJv+XozK/Lm55rZH8zssJn9rNM6F5nZW94695mZRWODREQkOrpsEGaWCtwPXA6MB24ws/Gdht0KHHDOjQF+Atzrzf8I+H/A34a5658DtwFjvZ853dkAERHpGZG8g5gKVDrnqpxzLcBSYG6nMXOBJd70CuASMzPn3BHnXDnBRvExMxsG9HfOrXPB7xt/DLj6TDZE/PHc23VU1R/2uwwR6QGRNIgRwJ6Q23u9eWHHOOdagUYgt4v73NvFfQJgZovMrMLMKurr6yMoV3rL9rpDfP2Xm/j8f7zKxpoP/C5HRKIs5g9SO+ceds4VO+eK8/K6lRaXHrJ4dQCAnL7pzH9kPS9t3+dzRSISTZE0iFogP+T2SG9e2DFmlgYMAPZ3cZ8ju7hPiWENh5v5zeZ3WTC9gN9/+7OcO6Qfix7fyFMVe7peWUTiQiQNYgMw1sxGmVk6MA9Y1WnMKmChN30t8LI7xbVMnXN1wCEzm+6dvfQV4H9Ou3rxzS/X1dDS2s4ts0aRm53Br26bzsxzcvm7FW/y4Kt/QpeyFYl/XTYI75jCHcDzwHZguXNuq5ndY2ZXecNKgVwzqwTuBD4+FdbMqoH/BG42s70hZ0D9FfAIUAn8CfhddDZJelpzaxu/XFfDxePyOCcvG4DsjDRKF07hyguH82+/28H3f7ud9nY1CZF4FtG3uTrnngWe7TTvH0OmPwKuO8m6RSeZXwFMjLRQiR2rNr9Lw+EWbp09+rj56Wkp/Nf1nya3bzqPlAfYf6SFf7/2AvqkxvyhLhEJI66+7lv855yjtDzAuCH9mDXmxBPVUlKMf7pyPHn9MvjR8zv54EgLP18wmax0/VcTiTd6aSenZW3Vfna89yEls4s4WfjdzLj94jH821+cz2vv1HPjL9Zz4EhLL1cqImdKDUJOS1l5gNy+6cz9dNjYynHmTS3gwQUXsb3uENc+uIbag029UKGIRIsahEQs0HCEl3a8z/zphWT2SY1onT+fMJTHb53G+x82c80Da9i178MerlJEokUNQiL26OoAfVJSWDC94LTWmzoqh+Vfm0G7c1z34FqlrkXihBqERKSx6RhPbdzLlRcOZ3C/zNNe/7xh/Vn5jZlKXYvEETUIiciyDbs52tJGyeyibt9Hfk4WK74+Q6lrkTihBiFdam1rZ8maGqaPzmHC8AFndF8dqesZo5W6Fol1ahDSpee2vkftwaYTgnHdlZ2RRtnNSl2LxDqll6RLZeUBCnOz+PynBkftPpW6Fol9ahBySm/sPsCm3Qf53pXjSU2J7lVhlboWiW16uSanVLa6mn6ZaVxXnN/14G5Q6lokdqlByEm9e7CJZ9+qY96UfPpm9Oyr+nlTC/j5govYptS1SMxQg5CTWrK2GuccC2cW9crjXTZhKI+XTFXqWiRGqEFIWEdbWnly/W7mTBzKyIFZvfa400bnKnUtEiPUICSslRv3cuijVm6dParXH1upa5HYoAYhJ2hvd5StrubC/LOZXDDQlxqUuhbxnxqEnOCVXe8TaDhCyayTX/OhNyh1LeIvNQg5QWl5gGEDMrni/GF+l6LUtYiPlEiS4+x47xCrK/fz93M+FTOpZqWuRfyhBiHHKSsPcFafVG6Y2jPBuO5S6lqk9+klmHys4XAzv9n8LtdcNIKzs9L9LucESl2L9C41CPnYE+t209Lazi2zev/U1tOh1LVI71CDEACaW9t4fF0NF4/L45y8bL/L6ZJS1yI9Tw1CAFi1+V0aDjdH7ZoPvUGpa5GepQYhOBcMxo0b0o9ZY3L9Lue0dE5dv7xDqWuRaFGDENZW7Wd73SFKZvsbjOuu/Jwsnvr6DMYO7sdtj21kxca9fpckkhDUIISy8mpy+qYz99Mj/C6l2wZlZ/DkomDq+m+f2sJDr/7J75JE4p4aRJKrbjjCSzv2sWBaAZl9Uv0u54x0pK6/dMEwfvi7HXz/t9uUuhY5A0oZJbnFqwP0SUlhwYxCv0uJivS0FO6bN4lB2Rn84rUADYeVuhbpLjWIJNbYdIynNu7lyguHM7hfpt/lRI1S1yLREdHLKjObY2Y7zazSzO4KszzDzJZ5y9ebWVHIsru9+TvN7LKQ+d82s61m9raZPWlmifMMFSeWbdjN0ZY2SmYX+V1K1Cl1LXLmumwQZpYK3A9cDowHbjCz8Z2G3QoccM6NAX4C3OutOx6YB0wA5gAPmFmqmY0A/hoods5NBFK9cdJLWtvaWbKmhumjc5gwfIDf5fQYpa5Fui+SdxBTgUrnXJVzrgVYCsztNGYusMSbXgFcYsHzJecCS51zzc65AFDp3R8EP946y8zSgCzg3TPbFDkdz2/dR+3BJkpi/Gs1oiE0dX3tz5W6FolUJA1iBBB6Oa+93rywY5xzrUAjkHuydZ1ztcCPgd1AHdDonHsh3IOb2SIzqzCzivr6+gjKlUiUlldRmJvFJecN8buUXtGRum5tV+paJFK+nNphZgMJvrsYBQwH+prZgnBjnXMPO+eKnXPFeXl5vVlmwnpj9wE27T7ILTOLSE2Jv2Bcd503rD9PK3UtErFIGkQtEHpxgJHevLBjvI+MBgD7T7HupUDAOVfvnDsGPA3M7M4GyOkrW11Nv4w0ri2OrWs+9AalrkUiF0mD2ACMNbNRZpZO8GDyqk5jVgELvelrgZdd8OLBq4B53llOo4CxwOsEP1qabmZZ3rGKS4DtZ7450pV3Dzbx7Ft1zJuaT3ZGcp72qdS1SGS6bBDeMYU7gOcJPokvd85tNbN7zOwqb1gpkGtmlcCdwF3euluB5cA24Dngdudcm3NuPcGD2ZuAt7w6Ho7qlklYj62twTnHwplFfpfiK6WuRbpmwRf68aG4uNhVVFT4XUbcOtrSyowfvsysMbk8MP8iv8uJCe3tjnue2caja6r58qQRSl1LwjGzjc654u6sm5yfMSSplRv30th0jFtnJ/6prZHqSF0Pyk7nxy/s4sDRFh6Yr9S1COjL+pJGe7tj8epqLhw5gMkFA/0uJ6aYGXd8fiw//Ivz+eMupa5FOqhBJIlXdr1PVcMRSmaPistrPvSGG5S6FjmOGkSSKC0PMLR/JlecP8zvUmKaUtcin1CDSAI73jvE6sr9fGVmoQ7ARkCpa5EgPVskgbLyAGf1SeXGqQV+lxI3lLoWUYNIeA2Hm/nN5ne55qIRnJ2V7nc5cUWpa0l2ahAJ7ol1u2lpbeeWJPjW1p6g1LUkMzWIBNbc2sbj62q4eFwe5+Rl+11O3MrOSKP05mKlriXpKA2UwP53Sx0Nh5spUTDujGWkpXLfvEnk9k3nF68F2H+4hXuVupYEpwaRoJxzlJYHGDekH7PHDPK7nISQkmJ876oJ5PXL4Mcv7OIDpa4lwenlT4JaW7Wf7XWHKJldpGBcFCl1LclEDSJBlZVXk9M3nbmf7nzxP4mGG6YW8MB8pa4lsalBJKDqhiO8tGMfC6YVkNkn1e9yEtacicenrt9R6loSjBpEAnp0TTVpKcaCGYV+l5LwQlPX1z64lo01B/wuSSRq1CASTGPTMZZX7OHKC4czuF+m3+UkhY7U9cCsPsx/ZJ1S15Iw1CASzLINuzna0qZrPvSy/JwsVnxj5sep65VKXUsCUINIIK1t7SxZU8P00TlMGD7A73KSTkfqevroHL6j1LUkADWIBPL81n3UHmyiRF+r4Rtd61oSiRI+CaS0vIrC3CwuOW+I36UkNaWuJVGoQSSIN3YfYNPug/zTleNJTVEwzm9KXUsi0EuaBFG2upp+GWlcV5zvdyni6Zy6nv+IUtcSX9QgEkBdYxPPvlXHvKn5ZGfoFWqs6Uhdb333ENc9tJZ3lbqWOKEGkQCWrKnBOcdXZhT5XYqcxJyJQ3msZCr7Gj/iGqWuJU6oQcS5oy2tPPn6buZMHEp+Tpbf5cgpTB+dyzKlriWOqEHEuZWbamlsOqZgXJwYP1ypa4kfahBxrL3dsbg8wIUjBzC5YKDf5UiElLqWeKEGEcde2fU+VQ1HKJk9Std8iDNKXUs8UIOIY2Xl1Qztn8kV5w/zuxTpBqWuJdbpnMg4teO9Q5RXNvDdOeOU0I1jSl1LLIvof6GZzTGznWZWaWZ3hVmeYWbLvOXrzawoZNnd3vydZnZZyPyzzWyFme0ws+1mNiMaG5QsysoDZPZJ4capBX6XImeoI3X9nS+cy9Nv1LLosQqOtrT6XZZI1w3CzFKB+4HLgfHADWY2vtOwW4EDzrkxwE+Ae711xwPzgAnAHOAB7/4A/gt4zjn3KeBCYPuZb05yaDjczG82v8s1k0dydla63+VIFJgZ37wkmLp+ValriRGRvIOYClQ656qccy3AUmBupzFzgSXe9ArgEgseNZ0LLHXONTvnAkAlMNXMBgCfBUoBnHMtzrmDZ745yeGJdbtpaW2nRKe2JhylriWWRNIgRgB7Qm7v9eaFHeOcawUagdxTrDsKqAcWm9kbZvaImfUN9+BmtsjMKsysor6+PoJyE1tzaxuPr6vh4nF5nJOX7Xc50gOUupZY4deRsDRgMvBz59wk4AhwwrENAOfcw865YudccV5eXm/WGJP+d0sdDYeb9e4hwSl1LbEgkgZRC4R+RehIb17YMWaWBgwA9p9i3b3AXufcem/+CoINQ07BOUdpeYBzh2Qze8wgv8uRHqbUtfgtkgaxARhrZqPMLJ3gQedVncasAhZ609cCLzvnnDd/nneW0yhgLPC6c+49YI+ZjfPWuQTYdobbkvDWVX3A9rpDlMxSMC5ZKHUtfuqyQXjHFO4Anid4ptFy59xWM7vHzK7yhpUCuWZWCdyJ93GRc24rsJzgk/9zwO3OuTZvnW8CT5jZm8CngR9Eb7MSU2l5gJy+6Vw9qfMhIElkSl2LXyz4Qj8+FBcXu4qKCr/L8EV1wxEu/o9X+ObFY7jzz8d1vYIknObWNu5cvoXfvlnHbZ8Zxd2Xn0eKrh4oXTCzjc654u6sqyR1nHh0TTVpKcaC6YV+lyI+yUhL5b/nTWKQUtfSS9Qg4kBj0zGWV+zhyguHM7h/pt/liI86UteDsjP4j9/v4sDRFu7Xta6lh+ilRxxYvmEPR1vaKJmlU1tFqWvpPWoQMa61rZ1H11QzbVQOE0cM8LsciSFKXUtPU4OIcc9v3UftwSZdMU7CUupaepIaRIwrWx2gMDeLS84b4ncpEqOUupaeogYRwzbvOcjGmgPcPLOIVJ3OKKcwfnh/Vn5dqWuJLjWIGFZaHqBfRhrXFed3PViSXkFuMHU9ZnC2UtcSFWoQMaqusYln36rj+in5ZGfoFEaJzKDsDJ687ZPU9cN/VOpauk8NIkYtWVODc46FM4v8LkXiTL/MPpTdPIUvXjCMHzy7gx88u13XupZu0UvTGHS0pZUnX9/NnIlDyc/J8rsciUOhqeuH/1hFw+Fm7r1GqWs5PWoQMWjlploam44pGCdn5ITU9RGlruX06OVEjGlvdywuD3DhyAFcVDjQ73IkznWkrn/wZaWu5fSpQcSYV3fVU9VwhJLZuuaDRM+N0wp4YP5kpa7ltKhBxJjS8gBD+2dyxfnD/C5FEsycicOUupbTogYRQ3a8d4jyyga+MrNQBxOlR4Smrq97SKlrOTU9C8WQxeXVZPZJ4capBX6XIgmsI3V99lnB1PUfdrzvd0kSo9QgYkTD4WZ+vbmWayaP5OysdL/LkQRXkJvFU18Ppq6/+liFUtcSlhpEjHhi3W5aWtu5Rae2Si/J66fUtZyaGkQMaG5t4/F1NXxuXB5jBmf7XY4kEaWu5VSUmIkB/7uljobDzbrmg/giIy2V++ZNIlepa+lEDcJnzjnKygOcOySb2WMG+V2OJKnUFOOfvdT1fyp1LR69RPDZuqoP2FZ3iJJZCsaJv8yMv1bqWkKoQfistDxATt90rp40wu9SRAClruUTahA+qm44wks79jF/WgGZfVL9LkfkY3MmDmPJLZ+krivfV+o6GalB+OjRNdWkpRg3TS/0uxSRE8w4R9e6TnZqED5pbDrG8oo9XHnhcAb3z/S7HJGwlLpObmoQPlm+YQ9HW9p0zQeJeUpdJy81CB+0trXz6Jpqpo3KYeKIAX6XI9KljtT1tFFKXScTNQgfvLBtH7UHmxSMk7jSL7MPi2+ZwhfPV+o6WSgF44PS8gAFOVlcct4Qv0sROS0Zaancd8MkcrOVuk4GEf1WzWyOme00s0ozuyvM8gwzW+YtX29mRSHL7vbm7zSzyzqtl2pmb5jZM2e6IfFi856DbKw5wC2zikhNUTBO4k9H6vrOL5zL05tqWfRYBUdbWv0uS3pAlw3CzFKB+4HLgfHADWY2vtOwW4EDzrkxwE+Ae711xwPzgAnAHOAB7/46fAvYfqYbEU/KygP0y0jjuuJ8v0sR6TalrpNDJO8gpgKVzrkq51wLsBSY22nMXGCJN70CuMSC3xsxF1jqnGt2zgWASu/+MLORwBeBR858M+JDXWMTz75Vx/VT8snO0Kd7Ev+Uuk5skTSIEcCekNt7vXlhxzjnWoFGILeLdX8KfBdoP9WDm9kiM6sws4r6+voIyo1dS9bU0O4cC2cW+V2KSNQodZ24fDmyZGZfAt53zm3saqxz7mHnXLFzrjgvL68XqusZR1taefL13Vw2YSj5OVl+lyMSVTPOyWXp16ZzrE2p60QSSYOoBUI/MB/pzQs7xszSgAHA/lOsOwu4ysyqCX5k9Xkz+2U36o8bKzfV0th0TKe2SsKaMHwAT39jJgOUuk4YkTSIDcBYMxtlZukEDzqv6jRmFbDQm74WeNk557z587yznEYBY4HXnXN3O+dGOueKvPt72Tm3IArbE5Pa2x2LVwe4cOQALioc6Hc5Ij2mIDeLFV+fyTl5wdT105uUuo5nXTYI75jCHcDzBM84Wu6c22pm95jZVd6wUiDXzCqBO4G7vHW3AsuBbcBzwO3Oubbob0Zse3VXPVX1RyiZrWs+SOLL65fB0kXB1PWdy7fwiz9W+V2SdJMFX+jHh+LiYldRUeF3GadtwSPrqXz/MK/9/cUKFEnSaG5t485lW/jtW3Us+uxo7przKVKU/el1ZrbROVfcnXV1rmUP2/neh5RXNvDdOePUHCSpKHUd/9QgelhZeYDMPincOLXA71JEep2udR3f1Mp7UMPhZn69uZZrJo/k7Kx0v8sR8UVH6vr7X574cer64FGlruOBGkQP+tX63bS0tnOLrvkgwvxphcHUde0hrn1Qqet4oAbRQ5pb23hsbQ2fG5fHmMHZfpcjEhPmTBzGkhKlruOFGkQPeWZLHQ2HmxWME+mkc+p6026lrmOVGkQPcM5RWh7g3CHZzB4zyO9yRGJOaOr6xl8odR2r1CB6wLqqD9hWd4iSWQrGiZyMUtexTw2iB5StDpDTN52rJ3X+0lsRCaXUdWxTg4iy6oYjvLh9H/OnFZDZJ7XrFUSSXOi1rr//7HZ++Ox24ukbHhKZ0ipR9uiaatJSjJumF/pdikjc6Ehd5/RN56E/VtFwuIV/u+Z8pa59pgYRRY1Nx1hesYcrLxjO4P6ZfpcjEldSU4x75gZT1z95cRcHjrZw/42TOStd78T9ovYcRcs37OFoSxslOrVVpFvMjG9dGkxdv7LzfeY/sk6pax+pQURJa1s7j66pZtqoHCaOGOB3OSJxrSN1/bZS175Sg4iSF7bto/Zgk949iESJUtf+U4OIktLyAAU5WVx63hC/SxFJGEpd+0sNIgo27znIxpoD3DKriFRdEEUkqk5IXe9U6rq3qEFEQVl5gH4ZaVxXnO93KSIJKTR1fdsSpa57ixrEGaprbOLZt+q4fko+2Rk6a1ikp3Skrqcqdd1r1CDO0GNra2h3joUzi/wuRSThdaSurzh/qFLXvUAvec/A0ZZWfrV+N5dNGEp+Tpbf5YgkhYy0VP77hsnk9t2q1HUPU4M4Ays31dLYdEzXfBDpZUpd9w613G5qb3csXh3ggpEDuKhwoN/liCQdpa57nhpEN726q56q+iPcOlvXfBDxU2jq+roH11LXqNR1tKhBdFPZ6gBD+mdwxfnD/C5FJOl1pK7fa/yIax5Q6jpa1CC6Yed7H/LaOw18ZUaRDoyJxIiO1HWLUtdRo2e3bigrD5DZJ4X50wr8LkVEQkwYPoCV35ih1HWUqEGcpv2Hm/n15lqumTySs7PS/S5HRDopzO2r1HWUqEGcpifW76altZ1bZunUVpFYpdR1dKhBnIbm1jYeW1vD58blMWZwtt/liMgpKHV95iJqEGY2x8x2mlmlmd0VZnmGmS3zlq83s6KQZXd783ea2WXevHwz+4OZbTOzrWb2rWhtUE96ZksdDYebKdG7B5G40JG6vml6IQ/9sYq/fepNjrW1+11W3OgySW1mqcD9wBeAvcAGM1vlnNsWMuxW4IBzboyZzQPuBa43s/HAPGACMBx40czOBVqB7zjnNplZP2Cjmf2+033GFOccpeUBzh2SzWfGDvK7HBGJkFLX3RfJO4ipQKVzrso51wIsBeZ2GjMXWOJNrwAusWB6bC6w1DnX7JwLAJXAVOdcnXNuE4Bz7kNgOzDizDen56wPfMC2ukOUzFIwTiTedKSu//XqifxBqeuIRdIgRgB7Qm7v5cQn84/HOOdagUYgN5J1vY+jJgHrwz24mS0yswozq6ivr4+g3J5RWh4gp286V0+K6T4mIqewYHohD9yo1HWkfD1IbWbZwErgb5xzh8KNcc497Jwrds4V5+Xl9W6BnuqGI7y4fR/zpxWQ2UdvS0Xi2eXnD+PRkinUKXXdpUgaRC0Qeqm0kd68sGPMLA0YAOw/1bpm1odgc3jCOfd0d4rvLY+uqSYtxbhpeqHfpYhIFMw8ZxBLFyl13ZVIGsQGYKyZjTKzdIIHnVd1GrMKWOhNXwu87ILnk60C5nlnOY0CxgKve8cnSoHtzrn/jMaG9JRDHx3jqYo9XHnBcAb3z/S7HBGJkokjPkldz//FeqWuw+iyQXjHFO4Anid4MHm5c26rmd1jZld5w0qBXDOrBO4E7vLW3QosB7YBzwG3O+fagFnATcDnzWyz93NFlLctKpa9vocjLW2U6JoPIgmnI3U9Oq8vty2p4NdvKHUdyuIpOFJcXOwqKip67fFa29r5sx+9woiBZ7H8azN67XFFpHd9+NExvvb4Rtb8aT//94vn8dXPjPa7pKgxs43OueLurKsk9Sm8sG0ftQebdMU4kQQXmrr+198qdd1Blxw9hbLyAAU5WVx63hC/SxGRHtaRus7p+7aude1RgziJLXsOUlFzgH/80nhSUxSME0kGqSnGv8ydyKDsDH764jtJn7pO3tbYhdLyAP0y0vjLKfldDxaRhGFm/M2l5yp1jRpEWHWNTTz7Vh3XT8knO0NvskSSkVLXahBhPba2hnbnWDizyO9SRMRHyZ66VoPo5GhLK79av5vLJgwlPyfL73JExGedU9dvJFHqWg2ik6c31dLYdEzBOBH5WGjq+sYkSl2rQYRob3eUrQ5wwcgBFBcO9LscEYkhyZi6VoMI8equeqrqj3DrbF3zQURO1HGt6ylFOXx72RYeeS2xr3WtBhGibHWAIf0zuHziML9LEZEYlUypa53D6dn53oe89k4Df3fZONLT1DdF5OQy+yRH6loNwrN4dYDMPincOLXA71JEJA4kQ+o6sdpdN+0/3MzTb9TyF5NHMrBvut/liEic6Jy6XlC6PqFS12oQwBPrd9PS2k7JrCK/SxGRONSRun5rb2NCpa6TvkE0t7bx+Loa/uzcPMYM7ud3OSISpxIxdZ30DeKZLXXUf9isaz6IyBlLtNR1UjcI5xyl5QHGDs7mM2MH+V2OiCSAjtR1/8z4T10ndYNYH/iAbXWHKFEwTkSiqDC3Lyu/Ef+p66RuEKXlAXL6pvPlSSP8LkVEEkwipK6TtkHU7D/Ci9v3MX9aAZl9Eue8ZRGJHSekrn8XX6nrpA3KLV5dTVqKcdP0Qr9LEZEEdlzq+tUqGj6Mn9R1UjaIQx8d46mKPVx5wXAG98/0uxwRSXDxmrqO/RbWA5Zv2MORljZd80FEek08pq6TrkG0trWzeHU1U0flMHHEAL/LEZEks2B6IffHSeo66RrEC9v2UXuwScE4EfHNFXGSuk66BlFWHqAgJ4tLzxvidykiksTiIXWdVA1iy56DVNQc4OaZRaSmKBgnIv7qnLp+JcZS10nVIMpWB+iXkcZfTsn3uxQREeCT1PWoQX35aoylrpOmQbzX+BG/fbOOv5yST3ZGUp7dKyIxKq9fBsu+Fnup66RpEEvWVtPuHDfPLPK7FBGRE3Skri+fGDup64gahJnNMbOdZlZpZneFWZ5hZsu85evNrChk2d3e/J1mdlmk9xlNTS1t/Gr9bi6bMJT8nKyefCgRkW7L7JPKz26czPxpBTz0ahV/t+JNWtvafauny89azCwVuB/4ArAX2GBmq5xz20KG3QoccM6NMbN5wL3A9WY2HpgHTACGAy+a2bneOl3dZ9Ss3LSXxqZjCsaJSMxLTTH+9eqJ5PXzUtdHWviZT6nrSN5BTAUqnXNVzrkWYCkwt9OYucASb3oFcIkFvz97LrDUOdfsnAsAld79RXKfUdHe7ihbHeCCkQMoLhzYEw8hIhJVHanrf7l6Ii97qesjza29XvZrd90AAAbZSURBVEckR2tHAHtCbu8Fpp1sjHOu1cwagVxv/rpO63Z8t3ZX9wmAmS0CFgEUFBREUO7xjh5rY0phDp85d5Cu+SAiceWm6YXk9k1ndWUDWT68g4j503mccw8DDwMUFxef9hGb7Iw07r32gqjXJSLSG644fxhXnD/Ml8eO5COmWiA0ODDSmxd2jJmlAQOA/adYN5L7FBERH0XSIDYAY81slJmlEzzovKrTmFXAQm/6WuBlFzw/axUwzzvLaRQwFng9wvsUEREfdfkRk3dM4Q7geSAVKHPObTWze4AK59wqoBR43MwqgQ8IPuHjjVsObANagdudc20A4e4z+psnIiLdZX4HMU5HcXGxq6io8LsMEZG4YWYbnXPF3Vk3aZLUIiJyetQgREQkLDUIEREJSw1CRETCiquD1GZWD9R0c/VBQEMUy+lp8VYvqObeEG/1QvzVHG/1wqlrLnTO5XXnTuOqQZwJM6vo7pF8P8RbvaCae0O81QvxV3O81Qs9V7M+YhIRkbDUIEREJKxkahAP+13AaYq3ekE194Z4qxfir+Z4qxd6qOakOQYhIiKnJ5neQYiIyGlQgxARkbASvkGY2Rwz22lmlWZ2l8+15JvZH8xsm5ltNbNvefNzzOz3ZvaO9+9Ab76Z2X1e7W+a2eSQ+1rojX/HzBae7DGjVHeqmb1hZs94t0eZ2XqvrmXeV7bjfa37Mm/+ejMrCrmPu735O83ssh6u92wzW2FmO8xsu5nNiOV9bGbf9v4/vG1mT5pZZqztYzMrM7P3zeztkHlR26dmdpGZveWtc5/ZmV/+8SQ1/8j7f/Gmmf3azM4OWRZ2/53sOeRkv6No1huy7Dtm5sxskHe7d/axcy5hfwh+lfifgNFAOrAFGO9jPcOAyd50P2AXMB74d+Aub/5dwL3e9BXA7wADpgPrvfk5QJX370BvemAP1n0n8CvgGe/2cmCeN/0g8A1v+q+AB73pecAyb3q8t+8zgFHe7yS1B+tdAnzVm04Hzo7VfUzwErwB4KyQfXtzrO1j4LPAZODtkHlR26cErxMz3Vvnd8DlPVTznwNp3vS9ITWH3X+c4jnkZL+jaNbrzc8neGmEGmBQb+7jHvkDjZUfYAbwfMjtu4G7/a4rpJ7/Ab4A7ASGefOGATu96YeAG0LG7/SW3wA8FDL/uHFRrnEk8BLweeAZ7z9XQ8gf2cf72PtPPMObTvPGWef9HjquB+odQPAJ1zrNj8l9zCfXc8/x9tkzwGWxuI+BIo5/so3KPvWW7QiZf9y4aNbcadmXgSe86bD7j5M8h5zq7yDa9QIrgAuBaj5pEL2yjxP9I6aOP74Oe715vvM+GpgErAeGOOfqvEXvAUO86ZPV35vb9VPgu0C7dzsXOOicaw3z2B/X5S1v9Mb3Zr2jgHpgsQU/FnvEzPoSo/vYOVcL/BjYDdQR3Gcbie193CFa+3SEN915fk8rIfhKmi5qCzf/VH8HUWNmc4Fa59yWTot6ZR8neoOISWaWDawE/sY5dyh0mQu295g499jMvgS875zb6HctpyGN4Nv0nzvnJgFHCH788bEY28cDgbkEG9twoC8wx9eiuiGW9mkkzOwfCF7l8gm/azkZM8sC/g/wj37VkOgNopbg53cdRnrzfGNmfQg2hyecc097s/eZ2TBv+TDgfW/+yervre2aBVxlZtXAUoIfM/0XcLaZdVyuNvSxP67LWz4A2N+L9ULwldFe59x67/YKgg0jVvfxpUDAOVfvnDsGPE1wv8fyPu4QrX1a6013nt8jzOxm4EvAfK+x0UVt4ebv5+S/o2g5h+ALhy3e3+BIYJOZDe1Gvd3bx9H8jDLWfgi+mqzydnLHAaYJPtZjwGPATzvN/xHHH+z7d2/6ixx/IOp1b34Owc/ZB3o/ASCnh2v/HJ8cpH6K4w/O/ZU3fTvHH0Bd7k1P4PgDgFX07EHq14Bx3vT3vP0bk/sYmAZsBbK8GpYA34zFfcyJxyCitk858QDqFT1U8xxgG5DXaVzY/ccpnkNO9juKZr2dllXzyTGIXtnHPfaEEis/BI/27yJ4JsI/+FzLbIJvw98ENns/VxD8PPMl4B3gxZBfqAH3e7W/BRSH3FcJUOn93NILtX+OTxrEaO8/W6X3R5Lhzc/0bld6y0eHrP8P3nbsJApnqHRR66eBCm8//8b7Q4nZfQz8M7ADeBt43HuSiql9DDxJ8BjJMYLv0m6N5j4Fir3t/xPwMzqdZBDFmisJfkbf8ff3YFf7j5M8h5zsdxTNejstr+aTBtEr+1hftSEiImEl+jEIERHpJjUIEREJSw1CRETCUoMQEZGw1CBERCQsNQgREQlLDUJERML6/5FJ7JwkPpf2AAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Define the model, optimizer, and the losses\n",
        "Using an `Adam` optimizer with the scheduler defined above. The loss is a sparse categorical crossentropy."
      ],
      "metadata": {
        "id": "rEta_QEDBcT5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = CIFAR_CNN()\n",
        "for test_data in ds_train:\n",
        "  images, labels = test_data\n",
        "  run_through = model(images[0:1])\n",
        "  break\n",
        "optim = tf.keras.optimizers.Adam(learning_rate=MyScheduler())\n",
        "losses = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False)\n",
        "metrics = ['accuracy']\n",
        "model.compile(optimizer=optim,\n",
        "                       loss = losses,\n",
        "                       metrics = metrics\n",
        "                       )\n",
        "model.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w6j7UbbMmki2",
        "outputId": "2c9fad56-f37e-4228-999b-c36f172e8660"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"cifar_cnn_6\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " conv2d_60 (Conv2D)          multiple                  896       \n",
            "                                                                 \n",
            " batch_normalization_60 (Bat  multiple                 128       \n",
            " chNormalization)                                                \n",
            "                                                                 \n",
            " conv2d_61 (Conv2D)          multiple                  9248      \n",
            "                                                                 \n",
            " batch_normalization_61 (Bat  multiple                 128       \n",
            " chNormalization)                                                \n",
            "                                                                 \n",
            " max_pooling2d_29 (MaxPoolin  multiple                 0         \n",
            " g2D)                                                            \n",
            "                                                                 \n",
            " dropout_30 (Dropout)        multiple                  0         \n",
            "                                                                 \n",
            " conv2d_62 (Conv2D)          multiple                  18496     \n",
            "                                                                 \n",
            " batch_normalization_62 (Bat  multiple                 256       \n",
            " chNormalization)                                                \n",
            "                                                                 \n",
            " conv2d_63 (Conv2D)          multiple                  36928     \n",
            "                                                                 \n",
            " batch_normalization_63 (Bat  multiple                 256       \n",
            " chNormalization)                                                \n",
            "                                                                 \n",
            " max_pooling2d_30 (MaxPoolin  multiple                 0         \n",
            " g2D)                                                            \n",
            "                                                                 \n",
            " dropout_31 (Dropout)        multiple                  0         \n",
            "                                                                 \n",
            " conv2d_64 (Conv2D)          multiple                  73856     \n",
            "                                                                 \n",
            " batch_normalization_64 (Bat  multiple                 512       \n",
            " chNormalization)                                                \n",
            "                                                                 \n",
            " conv2d_65 (Conv2D)          multiple                  147584    \n",
            "                                                                 \n",
            " batch_normalization_65 (Bat  multiple                 512       \n",
            " chNormalization)                                                \n",
            "                                                                 \n",
            " max_pooling2d_31 (MaxPoolin  multiple                 0         \n",
            " g2D)                                                            \n",
            "                                                                 \n",
            " dropout_32 (Dropout)        multiple                  0         \n",
            "                                                                 \n",
            " conv2d_66 (Conv2D)          multiple                  295168    \n",
            "                                                                 \n",
            " batch_normalization_66 (Bat  multiple                 1024      \n",
            " chNormalization)                                                \n",
            "                                                                 \n",
            " conv2d_67 (Conv2D)          multiple                  590080    \n",
            "                                                                 \n",
            " batch_normalization_67 (Bat  multiple                 1024      \n",
            " chNormalization)                                                \n",
            "                                                                 \n",
            " max_pooling2d_32 (MaxPoolin  multiple                 0         \n",
            " g2D)                                                            \n",
            "                                                                 \n",
            " flatten_7 (Flatten)         multiple                  0         \n",
            "                                                                 \n",
            " dense_13 (Dense)            multiple                  131200    \n",
            "                                                                 \n",
            " dropout_34 (Dropout)        multiple                  0         \n",
            "                                                                 \n",
            " dense_14 (Dense)            multiple                  1290      \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 1,308,586\n",
            "Trainable params: 1,306,666\n",
            "Non-trainable params: 1,920\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Fit the training data"
      ],
      "metadata": {
        "id": "OXxZ2WyJBxlk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "hist = model.fit(x=ds_train, epochs=50)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "J8_GKKqmrRJg",
        "outputId": "d1f4df79-f410-43d7-b0d8-5234bfed6c34"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/50\n",
            "391/391 [==============================] - 43s 106ms/step - loss: 2.5384 - accuracy: 0.1956\n",
            "Epoch 2/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 1.8831 - accuracy: 0.3281\n",
            "Epoch 3/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 1.6868 - accuracy: 0.4126\n",
            "Epoch 4/50\n",
            "391/391 [==============================] - 36s 91ms/step - loss: 1.5677 - accuracy: 0.4761\n",
            "Epoch 5/50\n",
            "391/391 [==============================] - 36s 91ms/step - loss: 1.4770 - accuracy: 0.5309\n",
            "Epoch 6/50\n",
            "391/391 [==============================] - 36s 91ms/step - loss: 1.4068 - accuracy: 0.5755\n",
            "Epoch 7/50\n",
            "391/391 [==============================] - 38s 96ms/step - loss: 1.3444 - accuracy: 0.6155\n",
            "Epoch 8/50\n",
            "391/391 [==============================] - 36s 92ms/step - loss: 1.3414 - accuracy: 0.6366\n",
            "Epoch 9/50\n",
            "391/391 [==============================] - 36s 91ms/step - loss: 1.3884 - accuracy: 0.6482\n",
            "Epoch 10/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 1.4852 - accuracy: 0.6437\n",
            "Epoch 11/50\n",
            "391/391 [==============================] - 36s 91ms/step - loss: 1.5471 - accuracy: 0.6476\n",
            "Epoch 12/50\n",
            "391/391 [==============================] - 36s 91ms/step - loss: 1.5805 - accuracy: 0.6452\n",
            "Epoch 13/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 1.5807 - accuracy: 0.6502\n",
            "Epoch 14/50\n",
            "391/391 [==============================] - 36s 91ms/step - loss: 1.5748 - accuracy: 0.6585\n",
            "Epoch 15/50\n",
            "391/391 [==============================] - 37s 96ms/step - loss: 1.5863 - accuracy: 0.6560\n",
            "Epoch 16/50\n",
            "391/391 [==============================] - 36s 91ms/step - loss: 1.5528 - accuracy: 0.6642\n",
            "Epoch 17/50\n",
            "391/391 [==============================] - 36s 92ms/step - loss: 1.5432 - accuracy: 0.6644\n",
            "Epoch 18/50\n",
            "391/391 [==============================] - 36s 91ms/step - loss: 1.5150 - accuracy: 0.6712\n",
            "Epoch 19/50\n",
            "391/391 [==============================] - 36s 92ms/step - loss: 1.4706 - accuracy: 0.6840\n",
            "Epoch 20/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 1.4420 - accuracy: 0.6867\n",
            "Epoch 21/50\n",
            "391/391 [==============================] - 36s 92ms/step - loss: 1.4024 - accuracy: 0.6962\n",
            "Epoch 22/50\n",
            "391/391 [==============================] - 35s 91ms/step - loss: 1.3509 - accuracy: 0.7058\n",
            "Epoch 23/50\n",
            "391/391 [==============================] - 38s 97ms/step - loss: 1.3136 - accuracy: 0.7135\n",
            "Epoch 24/50\n",
            "391/391 [==============================] - 35s 91ms/step - loss: 1.2650 - accuracy: 0.7198\n",
            "Epoch 25/50\n",
            "391/391 [==============================] - 36s 91ms/step - loss: 1.2254 - accuracy: 0.7252\n",
            "Epoch 26/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 1.1792 - accuracy: 0.7354\n",
            "Epoch 27/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 1.1348 - accuracy: 0.7442\n",
            "Epoch 28/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 1.0839 - accuracy: 0.7545\n",
            "Epoch 29/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 1.0372 - accuracy: 0.7639\n",
            "Epoch 30/50\n",
            "391/391 [==============================] - 37s 95ms/step - loss: 0.9907 - accuracy: 0.7729\n",
            "Epoch 31/50\n",
            "391/391 [==============================] - 36s 91ms/step - loss: 0.9479 - accuracy: 0.7781\n",
            "Epoch 32/50\n",
            "391/391 [==============================] - 36s 91ms/step - loss: 0.8881 - accuracy: 0.7938\n",
            "Epoch 33/50\n",
            "391/391 [==============================] - 36s 91ms/step - loss: 0.8467 - accuracy: 0.8012\n",
            "Epoch 34/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 0.8010 - accuracy: 0.8132\n",
            "Epoch 35/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 0.7647 - accuracy: 0.8246\n",
            "Epoch 36/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 0.7562 - accuracy: 0.8247\n",
            "Epoch 37/50\n",
            "391/391 [==============================] - 35s 91ms/step - loss: 0.7608 - accuracy: 0.8230\n",
            "Epoch 38/50\n",
            "391/391 [==============================] - 37s 95ms/step - loss: 0.7574 - accuracy: 0.8225\n",
            "Epoch 39/50\n",
            "391/391 [==============================] - 35s 89ms/step - loss: 0.7545 - accuracy: 0.8257\n",
            "Epoch 40/50\n",
            "391/391 [==============================] - 36s 91ms/step - loss: 0.7570 - accuracy: 0.8249\n",
            "Epoch 41/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 0.7528 - accuracy: 0.8284\n",
            "Epoch 42/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 0.7548 - accuracy: 0.8259\n",
            "Epoch 43/50\n",
            "391/391 [==============================] - 35s 89ms/step - loss: 0.7514 - accuracy: 0.8262\n",
            "Epoch 44/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 0.7530 - accuracy: 0.8265\n",
            "Epoch 45/50\n",
            "391/391 [==============================] - 38s 97ms/step - loss: 0.7526 - accuracy: 0.8255\n",
            "Epoch 46/50\n",
            "391/391 [==============================] - 36s 91ms/step - loss: 0.7571 - accuracy: 0.8257\n",
            "Epoch 47/50\n",
            "391/391 [==============================] - 36s 91ms/step - loss: 0.7586 - accuracy: 0.8242\n",
            "Epoch 48/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 0.7582 - accuracy: 0.8251\n",
            "Epoch 49/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 0.7565 - accuracy: 0.8252\n",
            "Epoch 50/50\n",
            "391/391 [==============================] - 35s 90ms/step - loss: 0.7585 - accuracy: 0.8239\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Loss vs. Epochs graph"
      ],
      "metadata": {
        "id": "C4CG_n4IB0PP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fig, ax = plt.subplots(1, 1, dpi=100)\n",
        "ax.plot(hist.history['loss'])\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 362
        },
        "id": "qp9Iuaja4SzF",
        "outputId": "a9626b17-b21b-43e7-ace0-09f915b7dd68"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 600x400 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAg0AAAFZCAYAAAAb7xzoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU1f3/8ddnMsmErECQBFkEZKuAoriwVVHcat2w7nu1VhG/arWb36+21n5b+utibRXXuhS1alsVXOpSVFwAAVEBUUEWWQTCngSyzsz5/TETOuYLyUySyU1m3s/H4z4mc+fcmU9uU+fNueeca845RERERJri87oAERER6RgUGkRERCQuCg0iIiISF4UGERERiYtCg4iIiMRFoUFERETiotAgIiIicfF7XUBrMDMD9gcqvK5FRESkA8oHNrgmFm9KidBAJDCs97oIERGRDqwX8FVjDVIlNFQArFu3joKCAq9rERER6TDKy8vp3bs3xNFbnyqhAYCCggKFBhERkSTRQEgRERGJi0KDiIiIxEWhQUREROKi0CAiIiJxUWgQERGRuCg0iIiISFwUGkRERCQuCg0iIiISF4UGERERiYtCQyOq60KUV9d5XYaIiEi7oNCwD/fOWsGQ217lVy995nUpIiIi7YJCwz507pQFwLbdNR5XIiIi0j4oNOxDUV4kNGzdVetxJSIiIu1DQqHBzG4xswVmVmFmm81supkNbuKYy83MNdiqG7QxM7vDzDaaWZWZzTSzgc35hVpLtzz1NIiIiMRKtKfhGGAqMAo4AcgEXjez3CaOKwd6xGwHNHj9x8D1wDXAUcBu4DUzy06wvlZTlBsAYJt6GkRERADwJ9LYOXdy7HMzuxzYDIwE3mn8ULdpby+YmQE3Av/rnJsR3XcpUAqcCTydSI2tpf7yRGVtiMraIDlZCZ0qERGRlNPSMQ2F0cftTbTLM7M1ZrbOzGaY2dCY1/oBJcDM+h3OuTJgHjB6b29mZgEzK6jfgPzm/wr7KDjgJ8sfOT3qbRAREWlBaDAzH3AXMNs590kjTZcBVwBnABdHP3OOmfWKvl4SfSxtcFxpzGsN3QKUxWzrE/4FmmBmdMutH9eg0CAiItKSnoapwDDg/MYaOefmOuemOec+ds69DZwFbAGubsFnTyHSy1G/9Wq8efMU5dWPa9BgSBERkWZdqDeze4BTgaOdcwn9K985V2dmHwEDorvqxzoUAxtjmhYDH+/jPWqAPd/kkWERra9+XIMuT4iIiCQ+5dKigWEicJxzbnWiH2hmGcBw/hMQVhMJDhNi2hQQmUUxN9H3b031Myi2atqliIhIwj0NU4ELiYxPqDCz+jEHZc65KgAzmwZ85Zy7Jfr8Z8D7wAqgM/AjIlMu/wKRaRVmdhdwq5l9QSRE/BLYAExvwe/WYvVrNWxXT4OIiEjCoWFS9HFWg/3fBR6L/twHCMe81gV4iMigxh3AQmCMc+7TmDa/BXKBB4kEi/eAk51zX1sEqq3tuTyhgZAiIiIJr9PQ5OAB59z4Bs9/APygiWMc8LPo1m50rb88oYGQIiIiuvdEYzQQUkRE5D8UGhrRrX4paQ2EFBERUWhoTGxPQ+QKioiISPpSaGhE1+iKkMGwo7wq6HE1IiIi3lJoaER2Zgb5gchYUa3VICIi6U6hoQkaDCkiIhKh0NAE3X9CREQkQqGhCUXRcQ1btcCTiIikOYWGJqinQUREJEKhoQndNKZBREQEUGhoUv3lCS3wJCIi6U6hoQn1lye2qqdBRETSnEJDE+qnXG7XQEgREUlzCg1N6KaBkCIiIoBCQ5PqxzTsqKwjGAp7XI2IiIh3FBqa0DknC59Fft5eqUsUIiKSvhQampDhsz03rtK0SxERSWcKDXFQaBAREVFoiEtRbnQwpNZqEBGRNKbQEIf6aZdaq0FERNKZQkMcNO1SREREoSEuRRrTICIiotAQjz13utSYBhERSWMKDXHQmAYRERGFhrjsuT22ehpERCSNKTTEYc+US/U0iIhIGlNoiEP95YnK2hCVtUGPqxEREfGGQkMc8gJ+svyRU6XeBhERSVcJhQYzu8XMFphZhZltNrPpZja4iWOuMrN3zWxHdJtpZkc2aPOYmbkG26vN+YWSwczoFp12uX23QoOIiKSnRHsajgGmAqOAE4BM4HUzy23kmPHAU8CxwGhgXfSYng3avQr0iNkuSLC2pNK0SxERSXf+RBo7506OfW5mlwObgZHAO/s45qIGx3wP+A4wAZgW81KNc25TPHWYWQAIxOzKj+e4ltC0SxERSXctHdNQGH3cnsAxOUR6KBoeMz56yWOZmd1nZkWNvMctQFnMtj6Bz28WzaAQEZF01+zQYGY+4C5gtnPukwQO/X/ABmBmzL5XgUuJ9D78hMhlkFfMLGMf7zGFSGCp33olVn3i9qzVoPtPiIhImkro8kQDU4FhwLh4DzCznwLnA+Odc9X1+51zT8c0W2Jmi4GVRMZDvNHwfZxzNcCeb28zS7T2hBXtWeBJPQ0iIpKemtXTYGb3AKcCxzrn4ro0YGY/BH4KnOicW9xYW+fcKmArMKA59SVD1+jlia3qaRARkTSVUE+DRf5JfzcwkUhvweo4j/sx8D/ASc65D+Jo3wsoAjYmUl8y7elp0JgGERFJU4n2NEwFLgYuBCrMrCS6dapvYGbTzGxKzPOfAL8ErgC+jDkmL/p6npn9zsxGmVlfM5sAzABWAK+17NdrPd1yNeVSRETSW6KhYRKRgYeziPQC1G/nxbTpQ2SdhdhjsoB/Njjmh9HXQ8DBwAvAcuBhYCHwzejYhXYhtqfBOedxNSIiIm0v0XUamhxx6Jwb3+B53ybaVwEnJVKHF7pGV4QMhh3lVUEKczI9rkhERKRt6d4TccrOzCA/EMlYW3WJQkRE0pBCQwI0GFJERNKZQkMC9tx/QtMuRUQkDSk0JKAoOq5hqxZ4EhGRNKTQkID6nobtujwhIiJpSKEhAXvuP6GBkCIikoYUGhJQf3lCAyFFRCQdKTQkoP7yhO4/ISIi6UihIQG606WIiKQzhYYEdNOUSxERSWMKDQmoH9Owo7KOYCjscTUiIiJtS6EhAZ1zsvBF776xvVKXKEREJL0oNCQgw2d0ydEMChERSU8KDQnS/SdERCRdKTQkqCg3OhhSCzyJiEiaUWhIUH1Pw1b1NIiISJpRaEiQpl2KiEi6UmhIkJaSFhGRdKXQkKD6paQ1pkFERNKNQkOCtJS0iIikK4WGBHXTlEsREUlTCg0J2jPlUgMhRUQkzSg0JKj+8sTu2hBVtSGPqxEREWk7Cg0Jygv4yfJHTpsGQ4qISDpRaEiQmdFN0y5FRCQNKTQ0g6ZdiohIOlJoaAYtJS0iIukoodBgZreY2QIzqzCzzWY23cwGx3HcOWb2uZlVm9kSMzulwetmZneY2UYzqzKzmWY2MNFfpq38ZwaFQoOIiKSPRHsajgGmAqOAE4BM4HUzy93XAWY2BngKeBg4FJgOTDezYTHNfgxcD1wDHAXsBl4zs+wE62sT/1mrQZcnREQkffgTaeycOzn2uZldDmwGRgLv7OOwG4BXnXO/iz6/zcxOAK4DrjEzA24E/tc5NyP6vpcCpcCZwNOJ1NgWuuZqVUgREUk/LR3TUBh93N5Im9HAzAb7XovuB+gHlMS2cc6VAfNi2nyNmQXMrKB+A/KbUXuz1Q+E3KqeBhERSSPNDg1m5gPuAmY75z5ppGkJkV6DWKXR/cQ8NtamoVuAsphtfZxlt4oiLSUtIiJpqCU9DVOBYcD5rVRLIqYQ6eWo33q15Yd3y9WUSxERST8JjWmoZ2b3AKcCRzvnmvpX/iaguMG+4uh+Yh6LgY0N2ny8tzd0ztUAe76xI8Mi2k5sT4Nzrs0/X0RExAuJTrm0aGCYCBznnFsdx2FzgQkN9p0Q3Q+wmkhw2NMmOk7hqJg27Ur9QMhg2FFeFfS4GhERkbaR6OWJqcDFwIVAhZmVRLdO9Q3MbJqZTYk55k/AyWZ2s5kNMbPbgcOBewCcc47I2Ihbzex0MxsOTAM2EJme2e5kZ2aQH4h00ugShYiIpItEQ8MkImMIZhG5lFC/nRfTpg/Qo/6Jc24OkZDxfWARcDZwZoPBk78F7gYeBBYAecDJzrnqBOtrM3suUWjapYiIpIlE12lo8uK9c278Xvb9A/hHI8c44GfRrUMoygvw5bZKLfAkIiJpQ/eeaKaiXN1/QkRE0otCQzPtudOlQoOIiKQJhYZm2nP/CQ2EFBGRNKHQ0Ez1lyfU0yAiIulCoaGZdP8JERFJNwoNzaQplyIikm4UGpqp256BkOppEBGR9KDQ0Ez1Yxp2VNYRDIU9rkZERCT5FBqaqXNOFr7oUlfbK3WJQkREUp9CQzNl+IwuOZpBISIi6UOhoQXqxzVsLKvyuBIREZHkU2hogYN7FQIwZ8U2jysRERFJPoWGFhg/uDsAby3b7HElIiIiyafQ0ALjBnYjw2es3LKbddsrvS5HREQkqRQaWqCwUyaH9ekMwKzlWzyuRkREJLkUGlqo/hLF27pEISIiKU6hoYXGD94PgNkrtlFdF/K4GhERkeRRaGihg3oU0D0/QFVdiAVfbve6HBERkaRRaGghM+OYQZHehlnLNK5BRERSl0JDK6gf1zBL4xpERCSFKTS0Ak29FBGRdKDQ0AoKO2Uysk8XQL0NIiKSuhQaWskxgzWuQUREUptCQyupn3o5Z6WmXoqISGpSaGglmnopIiKpTqGhlZjZnt6Gtz7XJQoREUk9Cg2taM/Uy+UaDCkiIqkn4dBgZkeb2YtmtsHMnJmd2UT7x6LtGm5LY9rcvpfXP2/OL+SlsQMiUy9XaeqliIikoOb0NOQCi4DJcba/AegRs/UGtgP/aNBuaYN245pRm6c09VJERFJZwqHBOfeKc+5W59zzcbYvc85tqt+Aw4EuwKMNmgZj2znntiZaW3swfoimXoqISGryYkzDlcBM59yaBvsHRi95rDKzJ82sz77ewMwCZlZQvwH5Sa04AeMHRcY1zF65VVMvRUQkpbRpaDCz/YFvAX9p8NI84HLgZGAS0A9418z2FQZuAcpitvXJqLc5vtEjn+KCANV1Yeav1tRLERFJHW3d03AZsBOYHrszesnjH865xc6514BTgM7Auft4nylAYczWK3klJ0Z3vRQRkVTVZqHBzAy4AnjcOVfbWFvn3E5gOTBgH6/XOOfK6zegotULboFjNfVSRERSUFv2NBxDJAQ83FRDM8sDDgQ2JruoZBg7sBv+6NTLtds09VJERFJDc9ZpyDOzEWY2IrqrX/R5n+jrU8xs2l4OvRKY55z7ZC/v+XszO8bM+prZGOB5IAQ8lWh97UFBdiaHHRCdeqneBhERSRHN6Wk4HPgougHcGf35jujzHsDXZj6YWSHwHfbdy9CLSEBYBvwd2AaMcs512EEB43XXSxERSTH+RA9wzs0CrJHXL9/LvjIgp5Fjzk+0jvZu/KDu/PbVZcyJTr3MzszwuiQREZEW0b0nkkRTL0VEJNUoNCSJme1Z6OktLSktIiIpQKEhierHNcz8rJRQ2HlcjYiISMsoNCTR0YP2o7BTJuu2V/H60k1elyMiItIiCg1JlBvwc9noAwC4d9ZKnFNvg4iIdFwKDUl22Zi+ZGf6WPJVGbNXbPO6HBERkWZTaEiyorwA5x8RWbbi3lkrPK5GRESk+RQa2sBVR/fH7zPmrNzGx+t2el2OiIhIsyg0tIGenTtxxoieANyn3gYREemgFBrayKTx/TGD15aWsmJzu7opp4iISFwUGtrIgO75nHhQMQD3zVrlcTUiIiKJU2hoQ5PGDwBgxsdf8dXOKo+rERERSYxCQxsa0bszYw4sIhh2PPSOehtERKRjUWhoY9dGexueXrCWbbtqPK5GREQkfgoNbWzsgCIO7lVIdV2Yx+Z86XU5IiIicVNoaGNmxqRjDgTgr3O+ZFdN0OOKRERE4qPQ4IGThpbQf79cyquD/G3eGq/LERERiYtCgwd8PuOaaG/DX95dTU0w5HFF6aM2GGbJ+jKenLeGn/xzMVdN+4BZyzZ7XZaISIfg97qAdHXmiJ788d/L2VhWzXMffsUFR/bxuqR2oS4UZtrcNTw9fy1VdSGcA+ccDnAOwjE/B/w+igsClBRmU1LQiZLCAMUF2ZQUZFNSmM1++QHW76hi0bqdLF5fxuKvyvhsQzm1ofDXPvPfn5Zy0tBibjv1IHp1yfHk9xYR6QgsFW7XbGYFQFlZWRkFBQVelxO3h99bzS9f+pS+RTm8cfN4MnzmdUmemrtyG7e/sJRlpcldMbOwUyYH9yrk4F6F7K4J8fj7awiFHdmZPiaPH8BVR/cnOzMjqTWIiLQX5eXlFBYWAhQ658oba6vQ4KHK2iBjfvMmOyvr+NP5I/bcnyLdbCyr4tf/+pwXF20AoEtOJjedOJhh+xdgZvgMDMOMyBb9ubI2xObyajaWVVNaXs2m8mo2RX/eWFZNTTBMTlYGw3oWckivQob36swhvQrp0zUHs/8EtGWbKvjZjE+Yt3o7AAcU5fDz0w7iuCHFnpwPEZG2pNDQgdz9xhf84d/LKSnIZubNx5AXSJ8rRrXBMA+/t5q73/yCytoQPoOLjjqAm08cROecrBa9t3OO8uogeQF/XD04zjleWLSBX738GZsrIutnHP+N7vzs1KH0KdIlCxFJXQoNHUh1XYgT//gOa7dXcuW4ftx26kFel9Qm3lm+hdtfWMqqrbsBGHlAF35x+lCG9Sz0tK5dNUH+/MYXPPLeaoJhR5bfx/lH9GZYz0IGF+czsDiPnKz0CXYikvoUGjqYt5dv4bJH5uMzeOG6cZ5/cSZTdV2Im/++iJeXbASgW16AW741hLMO6/m1SwZe+6K0gp+/sJQ5K7d9bb8Z9O6Sw6DifAaX5EUf8xnUPR9fmo9JEZGOSaGhA7rubx/y0uKNHNKrkOeuHZuSgyKDoTDXPPEhMz8rJcNnXD6mLzccP5CC7EyvS9sr5xwzP9vM7BVbWV5awfLSCrbuqt1r20HFedwwYRDfGlai8CAiHYpCQwe0ubyaCX94m4qaIL88YyiXjO7rdUmtyjnHT59dwjMfrCPL72PaFUcyqn+R12UlbOuumkiA2FTB8s27WL6pgqUbyqmqi6y1Mbg4nxuOH8jJQxUeRKRjUGjooKbN/ZKfzVhKfsDPGz88hu752V6X1Gp+99rnTH1rJT6D+y4eyUlDS7wuqdWUVdbxyOzVPPLeaiqiy4IrPIhIR6HQ0EGFwo6J985m8foyTj9kf/58waFel9QqHp29ml+8+CkAvzlrOOen6EJWZZV1PDx7NY/GhIchJfncMGEgJyk8iEg7lUhoSHgZaTM72sxeNLMNZubM7Mwm2o+Ptmu4lTRoN9nMvjSzajObZ2ZHJlpbR5fhM349cXhkQOSiDbyzfIvXJbXYC4s2cMdLkcDwwxMHpWxgACjMyeSmEwbx3k+O4/oJA8kP+Pl8UwWTnvyQb9/9Hss2JXfRKhGRZGvOvSdygUXA5ASPGwz0iNn2LPhvZucBdwK/AA6Lvv9rZta9GfV1aMN6FnLZmL4A3DbjE6rrOu59Kd79Ygs3//1jnIPLRh/A5GMHeF1Sm6gPD+/+5FiuP24AeQE/n20sZ+K9s3lp8QavyxMRabaEQ4Nz7hXn3K3OuecTPHSzc25TzBZ7A4CbgIecc4865z4FrgEqgSsSrS8V3HTCIIoLAqzZVsm9b63wupxmWbx+J9c8vpC6kOPbB/fgZ6cNbVdTKttC55wsbjpxMG//aDzjBnSjsjbEdX/7iCn/+oxgg/tfiIh0BG15l8uPzWyjmf3bzMbW7zSzLGAkMLN+XzRQzARG7+2NzCxgZgX1G5Cf5NrbVH52Jj8/bSgA9729khWbd3lcUWJWb93Ndx9dwO7aEGMHFHHnuYek5BTSeBXlBXjsu0fsubPpA++s4rJH57N9996nb4qItFdtERo2Euk5+E50WwfMMrPDoq93AzKA0gbHlQL7GmJ/C1AWs61v5Zo9961hJRw7eD/qQo5bpy+howxY3VxezSUPz2Pb7lqG9Szg/otHEvDr5k/+DB8//dYQ7r3oMHKyMpi9Yhun3f0eS9aXeV2aiEjckh4anHPLnHMPOOcWOufmOOeuAOYAP2jB204BCmO2Xq1QartiZtxxxjCyM328v2o7z3/0ldclNck5x+S/fcj6HVUcUJTDo5cfSX47XbjJK6cM78H0yWPp1y2Xr3ZW8Z375/CPD9Z5XZaISFza8vJErPlA/ai4rUAIaHhLwWJg094Ods7VOOfK6zcgJYel9+6aw/UTBgJEb6RU7XFFjXth0QYWfLmDnKwMpl1xJPvlB7wuqV0aVJzPjOvGcvw3ulMbDPOjfy7mtumfUBvUOAcRad+8Cg0jiFy2wDlXCywEJtS/aGa+6PO5nlTXjnxvXH+GlOSzbXct1z7xYbv9YqmsDTLlX58DMPnYARxQlOtxRe1bQXYmD15yOD84fhBm8Pj7azhj6mze/Ly0w1yKEpH005x1GvLMbISZjYju6hd93if6+hQzmxbT/kYzO8PMBpjZMDO7CzgOmBrztncCV5nZZWb2DeA+IlM7H23uL5Yqsvw+7rt4JPnZfj5Ys4M7XlrqdUl7dd+slWwqr6Z3105cOa6f1+V0CD6fccPxA3n4ssMpyI5My7zisQ+YeO8c3lm+ReFBRNqd5vQ0HA58FN0g8oX/EXBH9HkPIHYFnyzgD8AS4G3gEOB459wb9Q2cc88AP4y+x8dEeiJOds41HByZlvp1y+VP54/ADJ54fy3PLFjrdUlfs257JQ+8swqA/znlILIzNfAxEccNKeatH47n6qP7k53p4+N1O7n0kfmc+8Bc5qzc6nV5IiJ7aBnpDuTuN77gD/9eTlaGj6evHsVhfbp4XRIAk55YyCufbGLMgUU8+b2j0m49hta0paKG+99eyRPvr6EmeilqVP+u3HziYI7o29Xj6kQkFeneEykqHHZMenIhry0tpbggwIv/Nc7zm1rNWbmVCx+ah8/gXzd8kyElqXv+21JpeTX3vrWCp+avoza6ENQ3B3bjtlMPYlBxSi1LIiIeS+q9J8Q7Pp/xh3NHMKB7HqXlNZ4PjAyGwtwRvRHVxaMOUGBoRcUF2fzijGHM+tF4LjqqD5kZxrtfbOW0u9/jsdmrNd5BRDyh0NDB5AX8PHjJfwZG/uJF7wZGPrVgHZ9vqqCwUyY/OH6QZ3Wksv07d+JXE4fz5s3jGT94P2qCYW5/8VMue3QBm8vb9xRcEUk9Cg0dUP/98vYMjHxy3lqent/2AyN3Vtbyh9eXAXDziYPokpvV5jWkk95dc3j08iP45RlDCfh9vLN8Cyfd9Q6vfrLXpUxERJJCoaGDOm5IMTdF/3X/sxlL+XDtjjb9/LtmfsHOyjoGF+dzYQrf7ro9MTMuGd2Xl68fx9D9C9hRWcc1TyzkJ/9czO6aoNfliUgaUGjowCYfO4CThhZTGwoz6YmFbbZi5LJNFTz+/hoAfnbaQfgz9GfUlgZ0z+f5a8dyzTEHYgbPfLCOU/78Lh+1cXAUkfSj/9p3YA0HRl41bSGVtcn9F6dzjjteWkoo7DhpaDFjB3RL6ufJ3mX5IzfAeuqqUexfmM2abZWcff9c/vzGF4TDGiQpIsmh0NDB1Q+MLOyUyaJ1O5n85IfUhZI3o+L1T0uZvWIbWX4f/3PKQUn7HInPqP5FvHLj0Zx+yP6Ewo47/72cm/+xKKl/AyKSvhQaUkD//fJ45PLDCfh9vLVsC//9XHJupV1dF+JXL38GwFXf7EefopxW/wxJXGGnTP58waH89uyDyfAZz3/0FVc/vpCq2pDXpYlIilFoSBEjD+jKPRcehs/gHwvX84fXl7fq+5dX1/H9xxeydnslxQUBrh0/oOmDpE2de3hvHrp0JNmZPt78fDMXPzyPsso6r8sSkRSi0JBCTjiomF9PHA7APW+t4K9zvmyV912zbTdnRW+i1Ckzg9+dfQi5AX+rvLe0ruOGFPPElUdRkO1n4ZodnPvAXEq1noOItBKFhhRz/pF9uOmEyFTM219cyr+WbGzR+72/ahtnTJ3Nis27KCnI5h/XjOboQfu1RqmSJIf37crfrxlN9/wAy0or+M59c1i9dbfXZYlIClBoSEH/ddwALjqqD87BjU9/zNyV25r1Ps8sWMslD89jZ2Udh/Qq5IXrxjKsZ2ErVyvJMKSkgGcnjaFvUQ7rd1Rxzv1z+OSrMq/LEpEOTqEhBZkZd5wxjJOHllAbCvP9aR/w2cZG70HyNaGw41cvf8pPnl1CXchx6sE9eObq0XQv8PbmWJKY3l1z+Mc1Yxi6fwFbd9Vy/oPvNztAioiAQkPKyvAZd50/giP7dqWiJsjlj85n/Y7KJo+rqK7jqmkf8NC7qwG4YcJA7r7gULIzM5JdsiTBfvkBnvr+KI7q15VdNUEue3Q+Lyza4HVZItJB6dbYKa6sqo5z75/LstIK+hblcNKwEvKy/ORl+8kN+MkPRB7zsv04B//93BKWlVYQ8Pv43TmHcPoh+3v9K0grqK4Lcf1TH/H6p6UAnHpwD35x+lCK8gIeVyYiXkvk1tgKDWlgY1kV37l3DhvK4htFv19+gIcuPZwRvTsnuTJpS8FQmD/OXM79b68iFHYU5WbxyzOHccrwHl6XJiIeUmiQ/2NjWRXPLlzPjso6dtcEqagJsju6VVQH2V0bZFd1kIP2L+D35xxCj8JOXpcsSbJ4/U5+9I/FLCutAODbw3twxxnqdRBJVwoNItKommCIe95cwb2zVhIKO7rmZvHLM4bx7YPV6yCSbhQaRCQuS9aX8aN/LuLzTZFeh1OGl3DHGcPopl4HkbSRSGjQ7AmRNDa8VyEvXDeO6ycMxO8z/rVkEyfc+TbvfrHF69JEpB1SaBBJc1l+HzedMIjpk8cypCSfHZV1fPfRBfx9wTqvSxORdkahQUQAGNazkBnXjeWMEfsTDDt+/Oxifv/asqTcMVVEOiaFBhHZI+DP4K7zRnD9cZG7mN7z1gpuePpjaoK6zbaIKDSISANmxk0nDua3Zx+M32e8sGgDl/xlPjt213pdmoh4TKFBRPbq3MN789crjiQ/4Gf+l9s56xK2UsAAABfpSURBVL45fKm7ZYqkNYUGEdmnsQO68c9JY+jZuROrt+7mrPvmsHDNdq/LEhGPJBwazOxoM3vRzDaYmTOzM5tof5aZ/dvMtphZuZnNNbOTGrS5PfpesdvnidYmIq1vcEk+z187huE9C9m+u5YLHprHS4t10yuRdNScnoZcYBEwOc72RwP/Bk4BRgJvAS+a2aEN2i0FesRs45pRm4gkQfeCbJ65ehTHf6M7tcEw1/3tI/77+SVU1ga9Lk1E2lCLVoQ0MwdMdM5NT/C4pcAzzrk7os9vB850zo1oZh1aEVKkDYTCjt+++jkPvLMKgP7dcrnr/BEc3Es3NxPpqNr1ipBm5gPygYYXRgdGL3msMrMnzaxPI+8RMLOC+i36fiKSZBk+45ZTvsGT3zuKkoJsVm3dzVn3zmHqWysIhbWeg0iq82Ig5A+BPODvMfvmAZcDJwOTgH7Au2a2rzBwC1AWs61PVrEi8n+NHdCNV2/8JqcMLyEYdvzutWWc98Bc1m2v9Lo0EUmiNr08YWYXAg8BZzjnZjbSrjOwBrjJOffwXl4PALF31MkH1uvyhEjbcs7x3Idf8fMXlrKrJkhewM8dZwxl4qE9MTOvyxOROLTLyxNmdj7wF+DcxgIDgHNuJ7AcGLCP12ucc+X1G1DR6gWLSJPMjO+M7MUrN3yTkQd0YVdNkJv+vojrnvqIsso6r8sTkVbWJqHBzC4AHgUucM69HEf7POBAYGOyaxORluvdNYdnvj+KH544CL/PeHnxRk6f+h5flCrPi6SS5qzTkGdmI8ysfqZDv+jzPtHXp5jZtJj2FwLTgJuBeWZWEt0KY9r83syOMbO+ZjYGeB4IAU+14HcTkTbkz/Bx3XEDeXbSGHp16cSabZVMvHcOb3xW6nVpItJKmtPTcDjwUXQDuDP68x3R5z2A2JkP3wf8wFQiPQf1259i2vQiEhCWERkguQ0Y5Zzb0oz6RMRDh/TuzIzJYzmyX1d21QT53rQPuG/WSt0tUyQFtGggZHuhdRpE2p/aYJjbX1zK3+atBWDioT2ZctZwsjMzPK5MRGK1y4GQIpJesvw+fnXmMH55xlAyfMbzH33FeQ++T2l5tdeliUgzKTSISNKYGZeM7svjVxxJ55xMFq3byen3vMeidTu9Lk1EmkGhQUSSbsyAbsyYPJaB3fMoLa/h3AfmMuPjr7wuS0QSpNAgIm3igKJcnrt2DBOGdKcmGOaGpz/mVy9/SjAU9ro0EYmTQoOItJn87EwevPRwJo0/EICH3l3NpY/MZ9uuGo8rE5F4KDSISJvK8Bk/OXkI9150GDlZGcxZuY3T7n6Pxes1zkGkvVNoEBFPnDK8BzMmj6V/t1w2lFVz9v1z+fuCdV6XJSKNUGgQEc8MLM5n+nVjOf4bxdQGw/z42cX8z/NLqA1qnINIe6TQICKeKsjO5MFLRnLzCYMwgyfnreW8B+eyqUzrOYi0NwoNIuI5n8/4rwkDeeSyIyjI9vPR2p2cevd7zF+93evSRCSGQoOItBvHDunOi/81jiEl+WzdVcPFf5nHy4t1s1uR9kKhQUTalfr1HE4eWkJtKMx1T33I43O/9LosEUGhQUTaoZwsP1MvOoyLjuqDc3DbjKXc+foy3SlTxGMKDSLSLmX4jP89cxg/OH4QAH9+cwX//fwnhMIKDiJeUWgQkXbLzLjh+IH8auIwfAZPzV/LtU8upLou5HVpImlJoUFE2r2LjjqAey86jKwMH68tLeXSR+ZTVlXndVkiaUehQUQ6hJOH9eCvVxxJfsDP/NXbOe+BuZSWay0Hkbak0CAiHcboA4t45urR7Jcf4PNNFXznvjms3LLL67JE0oZCg4h0KAftX8Bzk8bQtyiH9TuqmDh1Nu9+scXrskTSgkKDiHQ4vbvm8M9JYxh5QBfKq4Nc/ugC/jrnS03JFEkyhQYR6ZC65QX421VHcdZhPQmFHT9/YSm3Tv+EupBudiWSLAoNItJhBfwZ/OGcQ7jlW0P23Ozq0ofns2N3rdeliaQkhQYR6dDMjKuPOZCHLjmc3KwM5q7axpn3zmbF5gqvSxNJOQoNIpISjj+omGevHUOvLp1Ys62SiVPnMGvZZq/LEkkpCg0ikjKGlBQwY/JYjujbhYqaIFc8toBH3lutAZIirUShQURSSlFegCe+dxTnjOxF2MEdL33KLc8toTaoAZIiLaXQICIpJ+DP4LdnH8yt3/4GPoOnF6zj4ofnsV0DJEVaRKFBRFKSmfG9b/bn4cuO2LP09On3vMeyTRogKdJcCYcGMzvazF40sw1m5szszDiOGW9mH5pZjZmtMLPL99Jmspl9aWbVZjbPzI5MtDYRkYaOHdKd564dwwHRFSTPunc2Mz8t9boskQ6pOT0NucAiYHI8jc2sH/Ay8BYwArgL+IuZnRTT5jzgTuAXwGHR93/NzLo3oz4Rka8ZWJzP9GvHMrp/EbtrQ1z1+Afc//ZKDZAUSZC15P80ZuaAic656Y20+X/At51zw2L2PQ10ds6dHH0+D1jgnLsu+twHrAPuds79Zi/vGQACMbvygfVlZWUUFBQ0+/cRkdRWFwpz+wtLeXLeWgDOOrQnvz5rONmZGR5XJuKd8vJyCgsLAQqdc+WNtW2LMQ2jgZkN9r0W3Y+ZZQEjY9s458LR56P38Z63AGUx2/rWLVlEUlFmho9fTRzOL88YSobPeO6jr7jgoffZXKFbbIvEoy1CQwnQ8AJiKVBgZp2AbkDGPtqU7OM9pwCFMVuvVqtWRFLeJaP78tfvHklBtp+P1u7ktLvfY8GX270uS6Td65CzJ5xzNc658voN0HBoEUnIuIHdmHHdOAZ0z6O0vIbzH3yfBzTOQaRRbREaNgHFDfYVA+XOuSpgKxDaR5tNyS9PRNJVv265zJg8ljNH7E8o7JjyyudcNW0hZZV1Xpcm0i61RWiYC0xosO+E6H6cc7XAwtg20YGQE+rbiIgkS27Azx/PG8GvJg4jK8PHzM9KOfWed1myvszr0kTaneas05BnZiPMbER0V7/o8z7R16eY2bSYQ+4H+pvZb81siJldC5wL/DGmzZ3AVWZ2mZl9A7iPyNTOR5vzS4mIJMLMuOioA3ju2jH07tqJddur+M59c3ji/TW6XCESozk9DYcDH0U3iHzhfwTcEX3eA+hT39g5txr4NpHehUXAzcD3nHOvxbR5Bvhh9D0+JrKew8nOOa3AIiJtZljPQl667puccFAxtaEwt07/hBuf+ZjdNUGvSxNpF1q0TkN7YWYFQJnWaRCR1uCc46F3V/H/Xl1GKOwY0D2PRy47gj5FOV6XJtLq2ts6DSIiHYqZ8f2jD+Tp74+iuCDAis27uPAv77OpTOs5SHpTaBAR2Ycj+nblxevG0Td63wrdKVPSnUKDiEgjuhdk88T3jqJHYTYrNu/iskfmU1GtKZmSnhQaRESa0KtLDo9feRRFuVks+aqMK//6AVW1Ia/LEmlzCg0iInEY0D2Pv15xJPkBP/NXb2fSkwupDYa9LkukTSk0iIjEaVjPQh757hFkZ/qYtWwLP/j7x4TCHX8Gmki8FBpERBJwRN+uPHDJ4WRmGC8v3sj/PL9EC0BJ2lBoEBFJ0DGD9uNP5x+Kz+DpBev49b8+U3CQtKDQICLSDKcM78FvzjoYgIfeXc2f3vhCwUFSnkKDiEgznXtEb2479SAA7pr5BZP/9iFlVZqOKalLoUFEpAWuHNeP2087iMwM419LNvHtP7/LR2t3eF2WSFIoNIiItNDlY/vxz2vG0KdrZOXIc+6fy/1vrySsmRWSYhQaRERawSG9O/PS9eM49eAeBMOO37zyOZc/toCtu2q8Lk2k1Sg0iIi0koLsTO6+4FB+c9ZwsjN9vLN8C9/607vMXrHV69JEWoVCg4hIKzIzzj+yDy9cN45BxXlsqajh4ofn8fvXlhEMaQVJ6dgUGkREkmBQcT4zJo/jgiN74xzc89YK3SVTOjyFBhGRJOmUlcGUsw7m7gsOJTcrg/dXbef0e97js43lXpcm0iwKDSIiSXbaIfvz/OSxHFAUmV1x1r1zeGXJRq/LEkmYQoOISBuIXK4YyzcHdqOqLsSkJz/kzteXaVqmdCgKDSIibaRzThaPXn4EV47rB8Cf31zBNU8sZFdN0OPKROKj0CAi0ob8GT5uO/Ugfn/OIWRl+Hj901LOunc2a7bt9ro0kSYpNIiIeODskb145upRdM8PsLx0F6ffM5v3vtB6DtK+KTSIiHjk0D5dePG/xnFI786UVdVx6SPzmPLKZ1TXhbwuTWSvFBpERDxUXJDNM98fxTkjexF28MDbq/jWn97l/VXbvC5N5P+wVLj/u5kVAGVlZWUUFBR4XY6ISLP8+9NSbp2+hNLyyP0qLjyqDz/91hAKsjM9rkxSWXl5OYWFhQCFzrlGFxFRaBARaUfKq+v4zSuf87d5awEoKcjmf88cxvEHFXtcmaQqhQYRkQ7u/VXb+Omzi/lyWyUApx7cg9tPH0q3vIDHlUmqSSQ0NGtMg5lNNrMvzazazOaZ2ZGNtJ1lZm4v28sxbR7by+uvNqc2EZFUMKp/Ea/eeDRXH9OfDJ/x0uKNHH/n29zz5he8vnQTKzbvok43wJI2lnBPg5mdB0wDrgHmATcC5wCDnXOb99K+K5AVs6sIWAR8zzn3WLTNY0Ax8N2YdjXOuR1x1qSeBhFJWUvWl/HjZxf/n3tWZPiMPl1z6N8tl/775dJ/vzwO6JpDZW2I7btr2bq7hu27aqM/17I9+rwmGCY/209+diYFnfzkB6KP2ZkUZGeSn+0nL9tPXsBPbsBPXiCD3ICf3Kz/7Mvy+6gNhqmqDVFZF6SyNkRlTYjK2iCVdSGqakMY0CU3iy45WXTJzaRLThaZGR1j/H1NMMTOyjq2767ds1XVhqgJhakLhqkNhakNRrfoz3WhMGbgM8NnBtT/DD6fYYA/wwj4M8jy+wj4fQT8GZHHTN+e/cFQmKq6ENV1YarrQjFb/f4QZ4/sxaF9urTK75rUyxNmNg9Y4Jy7LvrcB6wD7nbO/SaO428E7gB6OOd2R/c9BnR2zp2ZUDH/eU+FBhFJaXWhME/NX8uCL3ewassuVm3ZTZWHUzN9Bs1ZATs/4KdzbiZdc7IozMnCObfnC7G6LrTny7KmLkR1MEQo7MgN+PeEmfrH+tCTn+0nw2fUBsPUNPwyjz6v75Gx6Be4EfkyN4vsMyLnd3tlHTt217Jjdy0V7XyVzt+efTDnHt67Vd4rkdDgT+SNzSwLGAlMqd/nnAub2UxgdJxvcyXwdH1giDHezDYDO4A3gVudc3udc2RmASD2wl5+nJ8tItIhZWb4uHR0Xy4d3RcA5xybyqtZtWU3q7bsYuWW3azaupv1OyrJC/gpys2ia26AorwsuuZmUZSbFf05QHamj13VQcqr66ioDlJeVUd5g+e7aoLsrgmyqybE7j0/B6kJRr6AYwOD32fkZGWQk+WPPAYyyMn0E3KOHZW17KysY2dlLWEHFTVBKmqCrNteFffvXlEdpKK67b/EM3xGl5zMaE9JFrlZkZ6ALH8GWRm+Pb0FWX4fmRm2pxcl7CL/+4Sdw7mvP68LOWpDYWrqwtQEQ9REw05N3X9+zsowApkZZGdm0CnTR3ZmBtn+DDplZRDI9NEpM4ODenjzD+SEQgPQDcgAShvsLwWGNHVwdOzDMCLBIdarwHPAauBA4NfAK2Y22jm3tyh9C/DzxEoXEUkdZkaPwk70KOzE2AHd2uxz60JhKmsiPQKdMiNfZFn+pi85hMOO8upId/+OaIjYUVlHhg+y/ZEvyED0C7JT9AszO9NHhlkkaFQHqaiuo7wq8rjneXWQsHN7vsT3bBmxX+g+zCAcBgeEnQMXeax/nunz0SU3i67RyyhFuQHys/34fJb0c9qRJBoaWupKYIlzbn7sTufc0zFPl5jZYmAlMB54Yy/vMwW4M+Z5PrC+dUsVEZGGMjN8FOb4KCSxtSN8PqNzThadc7KabtxA94SPkGRJdETKViBEZNBirGJgU2MHmlkucD7wcFMf4pxbFf2sAft4vcY5V16/ARVx1C4iIiItkFBocM7VAguBCfX7ogMhJwBzmzj8HCLjEJ5o6nPMrBeRWRYbE6lPREREkqc5c1/uBK4ys8vM7BvAfUAu8CiAmU0zsyl7Oe5KYHrDwY1mlmdmvzOzUWbW18wmADOAFcBrzahPREREkiDhMQ3OuWfMbD8i0yZLgI+Bk51z9YMj+wBfW3HEzAYD44AT9/KWIeBg4DKgM7ABeB24zTlXk2h9IiIikhxaRlpERCSNJX0ZaREREUk/Cg0iIiISF4UGERERiYtCg4iIiMRFoUFERETi0tbLSCdVeXmjgz5FRESkgUS+O1NlymVPdO8JERGRlujlnPuqsQapEhoM2J/WvwdF/Y2weiXhvdOZzmty6Lwmh85r8ujcJkdzzms+sME1EQpS4vJE9JdsNB01RySLAFDR1IIXEj+d1+TQeU0Ondfk0blNjmae17jaaSCkiIiIxEWhQUREROKi0NC4GuAX0UdpPTqvyaHzmhw6r8mjc5scSTuvKTEQUkRERJJPPQ0iIiISF4UGERERiYtCg4iIiMRFoUFERETiotAgIiIicVFo2Aczm2xmX5pZtZnNM7Mjva6pozGzo83sRTPbYGbOzM5s8LqZ2R1mttHMqsxsppkN9KrejsDMbjGzBWZWYWabzWy6mQ1u0CbbzKaa2TYz22Vmz5pZsVc1dxRmNsnMFptZeXSba2bfinld57UVmNlPo/89uCtmn85tgszs9uh5jN0+j3k9KedUoWEvzOw84E4i81wPAxYBr5lZd08L63hyiZy7yft4/cfA9cA1wFHAbiLnObttyuuQjgGmAqOAE4BM4HUzy41p80fgNOCcaPv9gefauM6OaD3wU2AkcDjwJjDDzIZGX9d5bSEzOwK4Gljc4CWd2+ZZCvSI2cbFvJacc+qc09ZgA+YB98Q89xG5t8VPva6to26AA86MeW7ARuCHMfsKgWrgfK/r7SgbsF/03B4dcw5rgbNj2gyJthnldb0dbQO2A1fqvLbKucwDlgPHA7OAu6L7dW6bdz5vBz7ex2tJO6fqaWjAzLKI/EtjZv0+51w4+ny0V3WloH5ACV8/z2VEApvOc/wKo4/bo48jifQ+xJ7Xz4G16LzGzcwyzOx8Ir1lc9F5bQ1TgZedczMb7Ne5bb6B0cu/q8zsSTPrE92ftHOaEne5bGXdgAygtMH+UiJJTVpHSfRxb+e5BGmSmfmAu4DZzrlPortLgFrn3M4GzXVe42Bmw4mEhGxgFzDROfepmY1A57XZogHsMOCIvbysv9nmmQdcDiwjcmni58C7ZjaMJJ5ThQaRjmsqMIyvX8eUllkGjCDSg3M28FczO8bbkjo2M+sN/Ak4wTlX7XU9qcI590rM08VmNg9YA5wLVCXrc3V54v/aCoSAhqNMi4FNbV9Oyqo/lzrPzWBm9wCnAsc659bHvLQJyDKzzg0O0XmNg3Ou1jm3wjm30Dl3C5GBvDeg89oSI4HuwIdmFjSzIJGBeddHfy5F57bFor0Ky4EBJPHvVaGhAedcLbAQmFC/L9oNPIFIt6W0jtVE/nhjz3MBkVkUOs/7EJ2meg8wETjOObe6QZOFQB1fP6+DgT7ovDaHDwig89oSbwDDifTg1G8fAE/G/Kxz20JmlgccSGSAedL+XnV5Yu/uJNIt+QEwH7iRyICoRz2tqoOJ/hEPiNnVL3pteLtzbm10nvatZvYFkRDxS2ADML3tq+0wpgIXAmcAFWZWf32yzDlX5ZwrM7OHgTvNbDtQDtwNzHXOve9NyR2DmU0BXiEyWCyfyHkeD5yk89p8zrkK4JPYfWa2G9hWPxZH5zZxZvZ74EUilyT2J7JEQAh4Kpl/rwoNe+Gce8bM9gPuIDJo5GPgZOdcw0F70rjDgbdint8ZffwrkQE8vyUSxh4EOgPvETnPuu65b5Oij7Ma7P8u8Fj05x8AYeBZIv9Kfg24tg1q6+i6A9OIDCorI7KWwEnOuX9HX9d5TR6d28T1Ap4CioAtRP77Oco5tyX6elLOqUXnb4qIiIg0SmMaREREJC4KDSIiIhIXhQYRERGJi0KDiIiIxEWhQUREROKi0CAiIiJxUWgQERGRuCg0iIiISFwUGkRERCQuCg0iIiISF4UGERERicv/B8DzIpwOHwaZAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The graph shows an increase in the value of loss at the end of the warmup period which probably indicates a large learning rate. The linear decay fixes the problem."
      ],
      "metadata": {
        "id": "3z6WATgqB_hq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Evaluate of test data"
      ],
      "metadata": {
        "id": "V_j1MCONCWWi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model.evaluate(x=ds_test)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K7Ki7ZAo6xd9",
        "outputId": "7e28b37f-8399-4923-afc0-4d55c9336695"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "79/79 [==============================] - 2s 19ms/step - loss: 0.6048 - accuracy: 0.8702\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.6048139929771423, 0.870199978351593]"
            ]
          },
          "metadata": {},
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "87\\% test accuracy. Not too bad..."
      ],
      "metadata": {
        "id": "D6sj3jy5CY1A"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Resnet Architecture\n",
        "Investigate the effect of residual connections on the performance of our network"
      ],
      "metadata": {
        "id": "eG8alfyzCgS0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class Resblock(tf.keras.layers.Layer):\n",
        "  def __init__(self, n_filters, kernel_size=3, strides=1):\n",
        "    super(Resblock, self).__init__()\n",
        "    self.c1 = tf.keras.layers.Conv2D(filters=n_filters, kernel_size=kernel_size, strides=strides, padding='same', activation=None)\n",
        "    self.bn1 = tf.keras.layers.BatchNormalization()\n",
        "    self.activation = tf.keras.layers.ReLU()\n",
        "  \n",
        "  def call(self, input_):\n",
        "    x = self.c1(input_)\n",
        "    x = self.bn1(x)\n",
        "    x = self.activation(x)\n",
        "    return x  "
      ],
      "metadata": {
        "id": "Tl_QJg8LVKby"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class Reslayer(tf.keras.layers.Layer):\n",
        "  def __init__(self, n_filters, kernel_size=3, strides=1):\n",
        "    super(Reslayer, self).__init__()\n",
        "    self.c1 = tf.keras.layers.Conv2D(filters=n_filters, kernel_size=kernel_size, strides=strides, padding='same', activation=None)\n",
        "    self.bn1 = tf.keras.layers.BatchNormalization()\n",
        "    self.a1 = tf.keras.layers.ReLU()\n",
        "    self.mp1 = tf.keras.layers.MaxPooling2D(pool_size=2)\n",
        "    self.res1 = Resblock(n_filters, kernel_size, strides)\n",
        "    self.res2 = Resblock(n_filters, kernel_size, strides)\n",
        "    self.add = tf.keras.layers.Add()\n",
        "  \n",
        "  def call(self, input_):\n",
        "    x = self.c1(input_)\n",
        "    x = self.bn1(x)\n",
        "    x = self.a1(x)\n",
        "    x = self.mp1(x)\n",
        "    y = self.res1(x)\n",
        "    y = self.res2(y)\n",
        "    x = self.add([x, y])\n",
        "    return x  "
      ],
      "metadata": {
        "id": "rYsRfPu_XPPk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## The new architecture\n",
        "This is based on the architecture used [here](https://github.com/fenwickslab/fenwicks/blob/master/tutorials/tutorial2_cifar10.ipynb) but not exactly the same."
      ],
      "metadata": {
        "id": "aSV021-gCtXA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class CIFAR_ResNet(tf.keras.Model):\n",
        "  def __init__(self):\n",
        "    super(CIFAR_ResNet, self).__init__()\n",
        "    # prep\n",
        "    self.c1 = tf.keras.layers.Conv2D(filters=64, kernel_size=3, strides=1, padding='same', activation=None)\n",
        "    self.bn1 = tf.keras.layers.BatchNormalization()\n",
        "    self.a1 = tf.keras.layers.ReLU()\n",
        "\n",
        "    # layer 1\n",
        "    self.reslayer1 = Reslayer(128, 3, 1)\n",
        "\n",
        "    # layer 2\n",
        "    self.c2 = tf.keras.layers.Conv2D(filters=256, kernel_size=3, strides=1, padding='same', activation=None)\n",
        "    self.bn2 = tf.keras.layers.BatchNormalization()\n",
        "    self.a2 = tf.keras.layers.ReLU()\n",
        "    self.mp2 = tf.keras.layers.MaxPooling2D(2)\n",
        "\n",
        "    # layer 3\n",
        "    self.reslayer2 = Reslayer(512, 3, 1)\n",
        "\n",
        "    # layer 4\n",
        "    self.c3 = tf.keras.layers.Conv2D(filters=512, kernel_size=3, strides=1, padding='same', activation=None)\n",
        "    self.bn3 = tf.keras.layers.BatchNormalization()\n",
        "    self.a3 = tf.keras.layers.ReLU()\n",
        "    self.mp3 = tf.keras.layers.MaxPooling2D(2)\n",
        "\n",
        "    # layer 5\n",
        "    self.reslayer3 = Reslayer(512, 3, 1)\n",
        "\n",
        "    # classifier\n",
        "    self.mpc = tf.keras.layers.GlobalMaxPool2D()\n",
        "    self.flatten = tf.keras.layers.Flatten()\n",
        "    self.fc = tf.keras.layers.Dense(10, activation='softmax')\n",
        "\n",
        "  \n",
        "  def call(self, input_):\n",
        "    x = self.c1(input_)\n",
        "    x = self.bn1(x)\n",
        "    x = self.a1(x)\n",
        "\n",
        "    x = self.reslayer1(x)\n",
        "\n",
        "    x = self.c2(x)\n",
        "    x = self.bn2(x)\n",
        "    x = self.a2(x)\n",
        "    x = self.mp2(x)\n",
        "\n",
        "    x = self.reslayer2(x)\n",
        "\n",
        "    x = self.c3(x)\n",
        "    x = self.bn3(x)\n",
        "    x = self.a3(x)\n",
        "    x = self.mp3(x)\n",
        "\n",
        "    x = self.reslayer3(x)\n",
        "\n",
        "    x = self.mpc(x)\n",
        "    x = self.flatten(x)\n",
        "    x = self.fc(x)\n",
        "    \n",
        "    return x"
      ],
      "metadata": {
        "id": "rCpKsR8yCTNi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Learning rate scheduler\n",
        "Similar to the previous one: Warmup + linear decay"
      ],
      "metadata": {
        "id": "3zZxOZjADDCf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class MyScheduler(tf.keras.optimizers.schedules.LearningRateSchedule):\n",
        "\n",
        "  def __init__(self):\n",
        "    self.initial_learning_rate = LEARNING_RATE/BATCH_SIZE\n",
        "\n",
        "  def __call__(self, step):\n",
        "    warmup_steps = steps_per_epoch * WARMUP\n",
        "    is_warmup = tf.cast(step < warmup_steps, tf.float32)\n",
        "    decay_lr = tf.keras.optimizers.schedules.PolynomialDecay(LEARNING_RATE/BATCH_SIZE, (total_steps-(steps_per_epoch * WARMUP)), end_learning_rate=0.0, power=1, cycle=False)(step-(steps_per_epoch * WARMUP))\n",
        "    warmup_lr = self.initial_learning_rate * step / warmup_steps\n",
        "    return (1.0 - is_warmup) * decay_lr + is_warmup * warmup_lr"
      ],
      "metadata": {
        "id": "qoyD8dPqA7Kt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "xs = list(range(total_steps))\n",
        "ys = [MyScheduler()(i) for i in xs]\n",
        "plt.plot(xs, ys)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "xBDxRLPCA-qj",
        "outputId": "797b3916-a4ec-4e7d-db47-672b9ea67a6c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f479a5406d0>]"
            ]
          },
          "metadata": {},
          "execution_count": 34
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD4CAYAAADvsV2wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3xU9Z3/8dcnCQkh4ZpEiAhJUKgGqYiR+7b+VqtoW2yrtiBWASutu27b7WXX/vp7+Nt1293tZW/dpfVGFKwKgtrVli7buloNCBJEEEQwZQJJ5BIN5ZaG3L77R050iAmZhJk5c2bez8cjD2bOnGTenCSfmZyZ9znmnENERJJfmt8BREQkPjTwRURShAa+iEiK0MAXEUkRGvgiIikiw687zs/Pd8XFxX7dvYhIIG3ZsuVd51xBfz7Xt4FfXFxMZWWlX3cvIhJIZravv5+rXToiIilCA19EJEVo4IuIpAgNfBGRFKGBLyKSIjTwRURShAa+iEiK0MCX9x061sQzW2tpb9chs0WSkW/FK0k8f/3Udl7cXc9Le97lhzd+lAHpej4gkkz0Gy0A1DQ08tKeegCe2VrHl5ZX0tjc6nMqEYkmDXwB4JEN1aSZsfE7V/KPn5vEy2/XM//BTTScbPY7mohEiQa+cLyphVWba/jkRwsZNXQg86aO5b5bLuOtA8e48b4N1P3hj35HFJEo0MAXVlfWcuJUK4tnlby/7OqJo3j09mnUHz/FDT/dwJ5Dx31MKCLRENHAN7M5ZrbbzKrM7O4e1vm8mb1pZjvN7PHoxpRYaWt3PLwhRFnRcC4ZM+y026aWjODJL8+g3Tluuu8Vtuxr8CmliERDrwPfzNKBpcC1QCkw38xKu6wzHvgOMMs5NxH4egyySgz85s1D1DT8kdtnl3R7+0WFQ3jqzpnk5WSy4KFNPL/rUJwTiki0RPIMfypQ5Zzb65xrBlYC13dZ5w5gqXPuCIBz7nB0Y0qslK8PMXpYNp8oHdnjOmNGDGL1V2YwYeRgljy6hdWVNXFMKCLREsnAHw2E/4bXesvCTQAmmNl6M9toZnO6+0JmtsTMKs2ssr6+vn+JJWp21B3l1VADi2YVk9HLe+7zcrN4/I7pzDw/j2+v2c59v/s9zqmgJRIk0XrRNgMYD1wBzAceNLNhXVdyzj3gnCtzzpUVFPTrDF0SReUVIXIy0/n85WMiWj83K4Nlt13Opy85l3/89Vt8/1e71MoVCZBImrZ1QPhEOM9bFq4W2OScawFCZraHjgeAzVFJKVF3+FgTz21/hwXTihgycEDEn5eZkca/fWEyeTmZPFQR4r2TzWrligREJL+lm4HxZlZiZpnAPODZLuv8go5n95hZPh27ePZGMadE2YpX9tHa7lg0q7jPn5uWZvz/T5fy7Ws+olauSID0OvCdc63AXcA6YBfwpHNup5nda2ZzvdXWAe+Z2ZvAC8C3nXPvxSq0nJ2mljYe27SPqy4aSVFeTr++hpnx5//ngvdbuTc/uIkjauWKJLSIDp7mnFsLrO2y7J6wyw74hvchCe6ZrXUcaWzp8a2YfTFv6liG52TyF09s5cb7NrDi9mmMHpYdhZQiEm3a8ZpinHOUV4SYeO4QppWMiMrXvGbiKB5dPJXDauWKJDQN/BTz8tvv8vbhEyyeVYKZRe3rThuXp1auSILTwE8xyypCFAzO4tOXnBv1r93Zyh2hVq5IQtLATyFVh4/zuz313Dq9iMyM2Hzrx4wYxBq1ckUSkgZ+CilfX01mRho3Txsb0/vpbOXOGKdWrkgi0cBPEUdONvP0a7V87tLR5OVmxfz+crMyKF+oVq5IItE5bVPE46/up6mlnUWzzv6tmJFSK1cksWjgp4Dm1nZWvFLNn4zP5yOjBsf1vjtbuQWDs/jRut00nGzmZ7dMYVCmfvRE4k1PtVLA2jcOcOjYKRZHoWjVH2rliiQGDfwk55yjfH2IcQU5fHy8v0conTd1LD+75TLe1LlyRXyhgZ/kKvcdYXvtURbPKiEtLXpFq/5SK1fEPxr4Sa68IsTQ7AHcMOU8v6O8T61cEX9o4CexmoZG1u08yM3TxpKdme53nNN0beX+z1tq5YrEmgZ+EntkQzVpZtw2o9jvKN3qPFfu+HMGc8eKLazZUut3JJGkpoGfpI43tbBqcw2f/Ggho4YO9DtOj/Jzs3hiSUcr91urt6mVKxJDGvhJanVlLSdOtbI4jkWr/lIrVyQ+1H5JQm3tjoc3hCgrGs4lYz50LvmEpFauSOxp4Ceh3+46RE3DH/nOtRf5HaVP1MoViS09fUpCyypCjB6WzdWlI/2O0mdq5YrEjgZ+ktlRd5RXQw0smlVMRoB3h6iVKxJ9wZ0I0q3yihA5mel8/vIxfkc5a+Gt3Bt/plauyNnSwE8ih4818dz2d7ipbAxDBg7wO05UdLZyW9vVyhU5Wxr4SeTRjftobXcsmlXsd5SouqhwCE+rlSty1iIa+GY2x8x2m1mVmd3dze0LzazezF73Pr4U/ahyJk0tbTy2aT9XXTSSorwcv+NEnVq5Imev14FvZunAUuBaoBSYb2al3ay6yjk32ft4KMo5pRfPbK2j4WQzt/t0zPt46NrKvf93v/c7kkigRPIMfypQ5Zzb65xrBlYC18c2lvSFc47yihClhUOYVjLC7zgx1dnK/dRHC/mHX7/F93/1plq5IhGKZOCPBmrCrtd6y7q6wcy2m9kaM+v2LSJmtsTMKs2ssr6+vh9xpTsvv/0ubx8+we2zSzDz/5j3sZaZkcZP5l3KwpnFPPhyiG+u3kZLW7vfsUQSXrRetH0OKHbOfRT4DbC8u5Wccw8458qcc2UFBf6efSmZLKsIUTA4i09dUuh3lLjpbOV+6+oJPLO1jjtWVNLY3Op3LJGEFsnArwPCn7Gf5y17n3PuPefcKe/qQ8Bl0Yknvak6fJzf7anni9OLyMpIrGPex5qZcdefjucfPjeJl/aolSvSm0gG/mZgvJmVmFkmMA94NnwFMwt/ajkX2BW9iHIm5eurycxIY8G0sX5H8c18tXJFItLrwHfOtQJ3AevoGORPOud2mtm9ZjbXW+2rZrbTzLYBXwUWxiqwfODIyWaefq2Wz106mrzcLL/j+EqtXJHemV8nmygrK3OVlZW+3HeyWPpCFT9at5t1X/8YHxk12O84CWHXgWPcWv4qza3tlC8s47Ki5H7XkqQeM9vinCvrz+eqaRtQza3trHilmj8Zn69hH0atXJGeaeAH1K93HODQsVOBOKNVvKmVK9I9DfwAcs6xrCLEuIIcPj5Bb2/tjlq5Ih+mgR9AlfuOsL32KItnlZCWlvxFq/7Kzcpg2cIytXJFPDp3XACVV4QYmj2Az03prvAs4bIy0vnJvEvJz83iwZdDvHtC58qV1KWBHzA1DY2s23mQL3/8fJ3rNUKdrdz83Ex+/N97ONLYzE8X6Fy5knr0NCdglm+oJs2MW2cU+R0lUNTKFdHAD5QTp1pZtbmG6yYVUjg02+84gaRWrqQyDfwAeXJzDcdPtbI4iY95Hw9dW7lvq5UrKUIDPyDa2h2PbKjmsqLhTB4zzO84gRd+rtwb73uFLfuO+B1JJOY08APit7sOsb+hManPaBVvna3c4YMGsOChjWrlStLTwA+IZRUhRg/L5urSkX5HSSpjRgxizZ0z1cqVlKCBHwA76o7yaqiBhTOLydD7x6Ous5U7fdwItXIlqWl6BEB5RYiczHS+MLXbM0dKFOhcuZIK1DxJcIePNfHc9ndYMK2IIQMH+B0nqXW2cvNyMnnw5RDvnWjmB2rlShLRwE9wj27cR2u7Y9GsYr+jpIS0NONv5k6kYHAWP/7vPTSolStJRE9dElhTSxuPbdrPVReNpCgvx+84KaNrK3fBQ2rlSnLQwE9gv9haR8PJZr0V0yfzp47lpwsuY+c7auVKctDAT1DOOcrXhygtHMK0Ep2mzy9zLh7FisVTOXxMrVwJPg38BPXy2++y59AJbp9dgpmOee+n6ePyWKVWriQBDfwEVb4+RH5uFp+6pNDvKAKUnqtWrgSfBn4Cqjp8nBd313PrjCKyMtL9jiOerq3cp9TKlYCJaOCb2Rwz221mVWZ29xnWu8HMnJmVRS9i6nl4fTWZGWksmDbW7yjSRXgr95tq5UrA9DrwzSwdWApcC5QC882stJv1BgNfAzZFO2QqOXKymadeq+Wzk0eTl5vldxzphlq5ElSRPMOfClQ55/Y655qBlcD13az3d8APgKYo5ks5j7+6n6aWdh3zPsF1tnJvm1HEgy+H+NbqbbS0tfsdS+SMIhn4o4GasOu13rL3mdkUYIxz7ldn+kJmtsTMKs2ssr6+vs9hk11LWzsrXqnmT8bn85FRg/2OI73obOV+8xMTeHprHXesqKSxudXvWCI9OusXbc0sDfhn4Ju9reuce8A5V+acKysoKDjbu046a984wKFjp1g8S8/ug8LM+Isr1cqVYIhk4NcB4YdpPM9b1mkwcDHwoplVA9OBZ/XCbd8451hWEWJcQQ4fn6AHw6AJb+XedP8rvKNWriSgSAb+ZmC8mZWYWSYwD3i280bn3FHnXL5zrtg5VwxsBOY65ypjkjhJbdl3hO21R1k0q4S0NBWtgqizlXvoaBM3qJUrCajXge+cawXuAtYBu4AnnXM7zexeM5sb64CpYllFiKHZA7hhyujeV5aEpVauJLKI9uE759Y65yY45853zn3fW3aPc+7Zbta9Qs/u+6amoZF1Ow9y87SxOgxvElArVxKVmrYJYPmGatLMuHVGkd9RJErUypVEpIHvsxOnWlm1uYbrJhVSODTb7zgSRWrlSqLRwPfZ6soajp9qVdEqSXW2cj+pVq4kAO0w9lFbu+Ph9dVcVjScyWOG+R1HYiQrI51/n3cp+TpXrvhMA99Hv911iP0Njdx97YV+R5EY62zl5udm8U+/2cORxmaW6ly5Emd6iuGj8ooQo4dlc3XpSL+jSByEt3J/p1au+EAD3yc76o6yKdTAwpnFZOhP+5SiVq74RZPGJ+UVIXIy0/nC1DG9ryxJR61c8YMGvg8OH2viue3vcFPZGIYMHOB3HPGJWrkSbxr4Pnh04z5a2x2LZhX7HUV8VnruEJ76ilq5Eh8a+HHW1NLGY5v2c9VFIynKy/E7jiSAsXlq5Up8aODH2S+21tFwslnHvJfTdG3lPvCSWrkSfRr4ceSco3x9iNLCIUwfN8LvOJJgwlu5f79WrVyJPrU+4qii6l32HDrBP910CWY65r18mFq5Eksa+HG0rCJEfm4Wn7qk0O8oksDUypVY0dOGOKk6fJwXd9dz64wisjLS/Y4jCa6zlfv3n1UrV6JHAz9OHl5fTWZGGjdPG+t3FAmQm6eN5acLpqiVK1GhgR8HR04289RrtXx28mjyc7P8jiMBM+fiQrVyJSo08OPgic37aWpp1zHvpd/CW7k33a9WrvSPBn6MtbS1s2LDPmZfkM9HRg32O44EWGcrd1i2WrnSPxr4Mbb2jQMcPNbE7Xp2L1HQ2cq94JxctXKlzzTwY8g5x7KKEOMKcvj4hAK/40iSyM/N4ok71MqVvtPAj6Et+46wvfYoi2aVkJamopVEz+CBA05r5f792l1q5UqvIhr4ZjbHzHabWZWZ3d3N7V8xszfM7HUzqzCz0uhHDZ5lFSGGZg/ghimj/Y4iSSgrI52fzLuUW2cU8cBLe/nWmm20tLX7HUsSWK8D38zSgaXAtUApML+bgf64c26Sc24y8EPgn6OeNGBqGhpZt/Mg86eOVUNSYiY9zfjbuRP5xicm8PRrdSxZUUljc6vfsSRBRfIMfypQ5Zzb65xrBlYC14ev4Jw7FnY1B0j5vy2Xb6gmzYzbZhb5HUWSnJnxVbVyJQKRDPzRQE3Y9Vpv2WnM7M/N7Pd0PMP/andfyMyWmFmlmVXW19f3J28gnDjVyqrNNVw3qZDCodl+x5EUoVau9CZqL9o655Y6584H/hr4fz2s84Bzrsw5V1ZQkLzvWlldWcPxU60qWkncqZUrZxLJwK8Dws+0fZ63rCcrgc+cTagga2t3PLy+msuKhjN5zDC/40gKUitXehLJwN8MjDezEjPLBOYBz4avYGbjw65+Eng7ehGD5fldh9jf0Kiilfiqayv3hbcO+x1JEkCvA9851wrcBawDdgFPOud2mtm9ZjbXW+0uM9tpZq8D3wBui1niBLesIsToYdlcXTrS7yiS4sbmDWL1VzpauV9aUalWrkR2AhTn3FpgbZdl94Rd/lqUcwXSjrqjbAo18N3rLiJDZyiSBFAwuKOV++VHt/DN1dt47+QplnzsfL9jiU80laKofH2IQZnpfP7yMb2vLBIngwcO4OFFl/PJSWrlpjo1gqLk8LEmntv2DgumFTE0e4DfcUROk5WRzk/mX0pebiYPvLSXd0+c4gc36Fy5qUYDP0oe3biP1nbHwpnFfkcR6VZnKzc/N4t//s0ejpzUuXJTjR7eo6CppY3HNu3nygtHUpyf43cckR6plZvaNPCj4Bdb62g42ay3YkpgqJWbmjTwz5JzjvL1IUoLhzB93Ai/44hEbM7FhSxf9EErt+qwWrnJTgP/LFVUvcueQydYPLsEMx3zXoJlxvl5rPzydFraHDfep1ZustPAP0vLKkLk52bx6UsK/Y4i0i8Tzx3K03fOZKhauUlPA/8sVB0+wYu76/ni9CKyMtL9jiPSb2PzBrFGrdykp4F/Fh5eHyIzI40F08f6HUXkrHW2cqeV6Fy5yUoDv5+OnGzmqddq+ezk0eTnZvkdRyQq1MpNbmpc9NMTm/fT1NLOotnFfkcRiSq1cpOXBn4/tLS1s2LDPmZfkM+Fo4b4HUck6tTKTU56yO6HtW8c4OCxJhWtJKl1tnK//9mL32/l/qFRrdwg08DvI+cc5RUhxhXk8PEJyXuaRpFOC6YVdbRy645x431q5QaZBn4fbdl3hG21R1k0q4S0NBWtJDXMubiQ5YvVyg06Dfw+Kl8fYmj2AG6YMtrvKCJxpVZu8Gng90FNQyP/teMg86eO1YtXkpLUyg02Dfw+WL6hGjPjtplFfkcR8U1nK/f8go5W7tOvqZUbFBr4ETpxqpVVm2u4blIhhUOz/Y4j4quCwVmsXNLRyv3Gk9t48KW9fkeSCGjgR2h1ZQ3HT7XqrZginvBW7vfX7uIf1u7CObVyE5l2REegrd3xyIZqLisazuQxw/yOI5IwOlu5I3Iyuf+lvbx7opl/vGGSWrkJSgM/As/vOsS+9xr5q2su9DuKSMJJTzPuvX4iBYM7WrkNJ0+plZugInoYNrM5ZrbbzKrM7O5ubv+Gmb1pZtvN7HkzS6pXNZdVhBg9LJtrJo70O4pIQlIrNxh6Hfhmlg4sBa4FSoH5ZlbaZbWtQJlz7qPAGuCH0Q7qlx11R9kUamDhzGIy9GeqyBmplZvYIplgU4Eq59xe51wzsBK4PnwF59wLzrlG7+pG4LzoxvRP+foQgzLT+fzlY/yOIhIIauUmrkgG/migJux6rbesJ7cDv+7uBjNbYmaVZlZZX18feUqfHD7WxHPb3uHzZWMYmj3A7zgigdG1lfvafrVyE0FU91GY2S1AGfCj7m53zj3gnCtzzpUVFCT+gcd+vnEfre2OhTOL/Y4iEjjhrdybH9zIC7vVyvVbJAO/Dgjfn3Get+w0ZnYV8F1grnPuVHTi+aeppY2fb9rPlReOpDg/x+84IoEU3sq9Y7lauX6LZOBvBsabWYmZZQLzgGfDVzCzS4H76Rj2SfEw/outdTScbFbRSuQsdbZyp6qV67teB75zrhW4C1gH7AKedM7tNLN7zWyut9qPgFxgtZm9bmbP9vDlAsE5R/n6EBcVDmH6uBF+xxEJPLVyE0NEzQjn3FpgbZdl94RdvirKuXxVUfUuew6d4Mc3XYKZjnkvEg1q5fpPVbhulFeEyM/N4tOXFPodRSSpdLZy83Oz+Jff7uFIYzNLb55Cdma639FSgh5au6g6fIIXdtfzxelFZGXoh1Ak2syMr13V0cp9cfdhFjy0Ua3cONHA7+Lh9SEyM9JYMH2s31FEklpnK3eHWrlxo4Ef5g+NzTz1Wi2fmXwu+blZfscRSXpq5caXBn6Yx1/dT1NLO4v1VkyRuFErN3408D0tbe2s2LCP2Rfkc+GoIX7HEUkpE88dylN3zlArN8Y08D1r3zjAwWNNLJ5d7HcUkZRUlJejVm6MaeDjFa0qQowryOGKCef4HUckZamVG1sa+MBr+4+wrfYoi2aVkJamopWInzpbuddNGqVWbpSpeEXHGa2GZg/ghilnOuqziMRLVkY6/z5/Cnk5O9XKjaKUH/g1DY38146DLPnY+ToHp0gCUSs3+lL+4XLFK9WYGbfNTKrT8Iokhc5W7vc+czEvqJV71lJ64J841crKV2u4blIhhUOz/Y4jIj24ZXoRP725o5V7032vcOCoWrn9kdIDf3VlDcdPteqY9yIBcO2kjlbuwaNN3PBTtXL7I2UHflu745EN1UwZO4zJY4b5HUdEItDZym1WK7dfUnbgP7/rEPvea+T22eP8jiIifRDeyl3w4Ca1cvsgZQd++foQo4dlc83EkX5HEZE+6mzljivIUSu3D1Jy4O985ygb9zZw28wiMvS+XpFAUiu371Jy2i2rCDEoM50vXK5j3osEmVq5fZNyTaPDx5t4bts7LJhWxNDsAX7HEZGz1NnKHZGzQ63cXqTcwP/5K/tobXcsnFnsdxQRiZL0NOPvrr+YgtyBauWeQUo9BDa1tPHzTfu58sKRFOfn+B1HRKJIrdzeRTTwzWyOme02syozu7ub2z9mZq+ZWauZ3Rj9mNHxn6/X0XCyWce8F0liauX2rNeBb2bpwFLgWqAUmG9mpV1W2w8sBB6PdsBocc6xrCLERYVDmDEuz+84IhJD104q5JHFl3NArdzTRPIMfypQ5Zzb65xrBlYC14ev4Jyrds5tB9pjkDEq1le9x55DJ7h9dglmOua9SLKbeX4+K5eolRsukoE/GqgJu17rLeszM1tiZpVmVllfX9+fL9Fvyyr2kp+bxacvKYzr/YqIfy4erVZuuLi+aOuce8A5V+acKysoKIjb/VYdPsELu+v54vQisjL0qr1IKunayn1ma+q2ciMZ+HXAmLDr53nLAuORDSEyM9JYMF1FK5FUFN7K/ctV23jo5dRs5UYy8DcD482sxMwygXnAs7GNFT1/aGzmqS11fGbyueTnZvkdR0R8Et7K/d6vUrOV2+vAd861AncB64BdwJPOuZ1mdq+ZzQUws8vNrBa4CbjfzHbGMnRfPP7qfv7Y0sZiHfNeJOV1tnJvmT6W+1/ay7dWb6elLWHfaxJ1ETVtnXNrgbVdlt0TdnkzHbt6EkpLWzsrNuxj1gV5XDhqiN9xRCQBdLZy83Oz+Nffvp1SrdykbtqufeMAB4816YxWInIaM+PrV01IuVZu0g585xzlFSHG5edwxYRz/I4jIgko1Vq5STvwX9t/hG21R1k0q5i0NBWtRKR7qdTKTdqBv6wixNDsAdxwWcK9tCAiCaZrK3drkrZyk3Lg1x5p5L92HGT+1LEMyky5I0CLSD+Et3JvTtJWblIO/OUbqjEzbp1R5HcUEQmQZG/lJt3AP3GqlZWv1nDdpELOHZbtdxwRCZjOVu7lxcnXyk26gb+msobjp1pZPKvY7ygiElDJ2spNqh3cbe2OhzdUM2XsMC4dO9zvOCISYAMHJN+5cpNq4D+/6xD73mvkr6650O8oIpIEkq2VG9yHqm6Urw8xelg210wc6XcUEUkSXVu5tyzbFNhWbtIM/J3vHGXj3gZum1lERoD/5BKRxNTZyn2j9mhgW7lJMxnLK6oZlJnOFy7XMe9FJDaC3spNioF/+HgTz217h5suO4+h2QP8jiMiSSzIrdykGPg/f2UfLe3tLJqlo2KKSOx1tnKHDAxWKzfwA7+ppY2fb9rPlReOpDg/x+84IpIiivJyeOrOmZTkB6eVG/iB/5+v19FwspnFs4v9jiIiKaZgcBarvhycVm6gB75zjmUVIS4qHMKMcXl+xxGRFNTZyr32Yq+V++vEbeUGeuCvr3qPPYdOsHhWMWY65r2I+GPggHT+4+YpLJg2lvt/t5dvr9lOawKeKzfQTdtlFXvJz81i7uRz/Y4iIikuPc343mcupmCw18o92cx/JFgrN7DP8H9ff4IXdtfzxelFZGUkzgYVkdQV3sr9nwRs5QZ24D+8PkRmRhoLpqtoJSKJ5ZbpRSxNwFZuIAf+HxqbeWpLHZ+ZfC75uVl+xxER+ZDrErCVG9HAN7M5ZrbbzKrM7O5ubs8ys1Xe7ZvMrDjaQcM98WoNf2xpY/FsFa1EJHElWiu314FvZunAUuBaoBSYb2alXVa7HTjinLsA+BfgB9EO2qmlrZ3lG6qZdUEeF44aEqu7ERGJiq6t3Bd9bOVG8gx/KlDlnNvrnGsGVgLXd1nnemC5d3kNcKXF6H2Sa984wMFjTdyuZ/ciEhBFeTmsuXMGJfk5fGl5Jb/c/o4vOSIZ+KOBmrDrtd6ybtdxzrUCR4EPNaHMbImZVZpZZX19fb8C52Zl8InSkVwx4Zx+fb6IiB/OGTyQVV+ezp9eeA7nF+T6kiGu78N3zj0APABQVlbWryralReN5MqLdIITEQmewQMH8MCtZb7dfyTP8OuAMWHXz/OWdbuOmWUAQ4H3ohFQRESiI5KBvxkYb2YlZpYJzAOe7bLOs8Bt3uUbgf9xiXowCRGRFNXrLh3nXKuZ3QWsA9KBcufcTjO7F6h0zj0LLAMeNbMqoIGOBwUREUkgEe3Dd86tBdZ2WXZP2OUm4KboRhMRkWgKZNNWRET6TgNfRCRFaOCLiKQIDXwRkRRhfr170szqgX39/PR84N0oxomHoGUOWl5Q5ngIWl4IXube8hY55wr684V9G/hnw8wqnXP+1dX6IWiZg5YXlDkegpYXgpc5lnm1S0dEJEVo4IuIpIigDvwH/A7QD0HLHLS8oMzxELS8ELzMMcsbyH34IiLSd0F9hi8iIn2kgS8ikiICN/B7O6F6HHOMMbMXzOxNM9tpZl/zlo8ws9+Y2dvev8O95WZmP/FybzezKWFf6zZv/bfN7Lae7jNKudPNbKuZ/dK7XuKdeO3eEDoAAAS+SURBVL7KOxF9pre8xxPTm9l3vOW7zeyaGOcdZmZrzOwtM9tlZjMCsI3/0vuZ2GFmT5jZwETbzmZWbmaHzWxH2LKobVczu8zM3vA+5ydmZ3fK0x7y/sj7udhuZs+Y2bCw27rddj3Nj56+P9HOHHbbN83MmVm+dz0+29g5F5gPOg7P/HtgHJAJbANKfcpSCEzxLg8G9tBxkvcfAnd7y+8GfuBdvg74NWDAdGCTt3wEsNf7d7h3eXgMc38DeBz4pXf9SWCed/k+4E7v8p8B93mX5wGrvMul3nbPAkq870d6DPMuB77kXc4EhiXyNqbjdJ8hIDts+y5MtO0MfAyYAuwIWxa17Qq86q1r3udeG4O8VwMZ3uUfhOXtdttxhvnR0/cn2pm95WPoONz8PiA/nts4Jr+ksfoAZgDrwq5/B/iO37m8LP8JfALYDRR6ywqB3d7l+4H5Yevv9m6fD9wftvy09aKc8TzgeeBPgV96Pyjvhv3SvL99vR/IGd7lDG8967rNw9eLQd6hdAxP67I8kbdx5/mdR3jb7ZfANYm4nYFiTh+gUdmu3m1vhS0/bb1o5e1y22eBx7zL3W47epgfZ/o9iEVmYA1wCVDNBwM/Lts4aLt0Ijmhetx5f4ZfCmwCRjrnDng3HQQ6T8DbU/Z4/p/+FfgroN27ngf8wXWceL7rffd0Yvp45i0B6oGHrWM31ENmlkMCb2PnXB3wY2A/cICO7baFxN7OnaK1XUd7l7suj6XFdDzLpZdc3S0/0+9BVJnZ9UCdc25bl5viso2DNvATjpnlAk8BX3fOHQu/zXU89CbE+17N7FPAYefcFr+z9EEGHX8S/8w5dylwko5dDe9LpG0M4O33vp6OB6tzgRxgjq+h+iHRtuuZmNl3gVbgMb+znImZDQL+L3BPb+vGStAGfiQnVI8bMxtAx7B/zDn3tLf4kJkVercXAoe95T1lj9f/aRYw18yqgZV07Nb5N2CYdZx4vut993Ri+nh+D2qBWufcJu/6GjoeABJ1GwNcBYScc/XOuRbgaTq2fSJv507R2q513uWuy6POzBYCnwIWeA9S/cn7Hj1/f6LpfDqeCGzzfg/PA14zs1H9yNy/bRzNfYKx/qDjGd9eb6N1vugy0acsBqwA/rXL8h9x+gtfP/Quf5LTX5R51Vs+go791MO9jxAwIsbZr+CDF21Xc/qLVX/mXf5zTn8x8Unv8kROf0FsL7F90fZl4CPe5b/xtm/CbmNgGrATGOTlWA78RSJuZz68Dz9q25UPv6B4XQzyzgHeBAq6rNfttuMM86On70+0M3e5rZoP9uHHZRvHbKjE6oOOV7P30PFq+3d9zDGbjj95twOvex/X0bE/8HngbeC3Yd8cA5Z6ud8AysK+1mKgyvtYFIfsV/DBwB/n/eBUeT/0Wd7ygd71Ku/2cWGf/13v/7Gbs3z3RQRZJwOV3nb+hfdDn9DbGPhb4C1gB/CoN3gSajsDT9DxGkMLHX9J3R7N7QqUef//3wP/QZcX3qOUt4qO/dudv3/39bbt6GF+9PT9iXbmLrdX88HAj8s21qEVRERSRND24YuISD9p4IuIpAgNfBGRFKGBLyKSIjTwRURShAa+iEiK0MAXEUkR/wtEjlUzRUgeGgAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Define the model\n",
        "This time we are using `SGD` instead of `Adam`. Everything else is the same."
      ],
      "metadata": {
        "id": "vvbcMIeODLMX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = CIFAR_ResNet()\n",
        "for test_data in ds_train:\n",
        "  images, labels = test_data\n",
        "  run_through = model(images[0:1])\n",
        "  break\n",
        "#optim = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
        "optim = tf.keras.optimizers.SGD(learning_rate=MyScheduler(), momentum=0.9, nesterov=True, decay = WEIGHT_DECAY*BATCH_SIZE)\n",
        "losses = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False)\n",
        "metrics = ['accuracy']\n",
        "\n",
        "model.compile(optimizer=optim,\n",
        "                       loss = losses,\n",
        "                       metrics = metrics\n",
        "                       )\n",
        "model.summary()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SHeewl8nbcG4",
        "outputId": "d3dcd427-bc7c-4138-8915-b05417bd6e03"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"cifar__res_net\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " conv2d_32 (Conv2D)          multiple                  1792      \n",
            "                                                                 \n",
            " batch_normalization_32 (Bat  multiple                 256       \n",
            " chNormalization)                                                \n",
            "                                                                 \n",
            " re_lu (ReLU)                multiple                  0         \n",
            "                                                                 \n",
            " reslayer (Reslayer)         multiple                  370560    \n",
            "                                                                 \n",
            " conv2d_36 (Conv2D)          multiple                  295168    \n",
            "                                                                 \n",
            " batch_normalization_36 (Bat  multiple                 1024      \n",
            " chNormalization)                                                \n",
            "                                                                 \n",
            " re_lu_4 (ReLU)              multiple                  0         \n",
            "                                                                 \n",
            " max_pooling2d_17 (MaxPoolin  multiple                 0         \n",
            " g2D)                                                            \n",
            "                                                                 \n",
            " reslayer_1 (Reslayer)       multiple                  5905920   \n",
            "                                                                 \n",
            " conv2d_40 (Conv2D)          multiple                  2359808   \n",
            "                                                                 \n",
            " batch_normalization_40 (Bat  multiple                 2048      \n",
            " chNormalization)                                                \n",
            "                                                                 \n",
            " re_lu_8 (ReLU)              multiple                  0         \n",
            "                                                                 \n",
            " max_pooling2d_19 (MaxPoolin  multiple                 0         \n",
            " g2D)                                                            \n",
            "                                                                 \n",
            " reslayer_2 (Reslayer)       multiple                  7085568   \n",
            "                                                                 \n",
            " global_max_pooling2d (Globa  multiple                 0         \n",
            " lMaxPooling2D)                                                  \n",
            "                                                                 \n",
            " flatten_4 (Flatten)         multiple                  0         \n",
            "                                                                 \n",
            " dense_8 (Dense)             multiple                  5130      \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 16,027,274\n",
            "Trainable params: 16,018,698\n",
            "Non-trainable params: 8,576\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Fit to training data"
      ],
      "metadata": {
        "id": "_fmywCrvDbIn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "hist = model.fit(x=ds_train, epochs=EPOCHS)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HxiP6ICWblD1",
        "outputId": "9ae9e805-6a25-435f-d3d1-1003c855abf0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/35\n",
            "391/391 [==============================] - 58s 142ms/step - loss: 1.6141 - accuracy: 0.4526\n",
            "Epoch 2/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 1.0419 - accuracy: 0.6323\n",
            "Epoch 3/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 0.8372 - accuracy: 0.7054\n",
            "Epoch 4/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 0.7289 - accuracy: 0.7456\n",
            "Epoch 5/35\n",
            "391/391 [==============================] - 53s 134ms/step - loss: 0.6448 - accuracy: 0.7748\n",
            "Epoch 6/35\n",
            "391/391 [==============================] - 54s 138ms/step - loss: 0.5891 - accuracy: 0.7945\n",
            "Epoch 7/35\n",
            "391/391 [==============================] - 53s 136ms/step - loss: 0.5464 - accuracy: 0.8108\n",
            "Epoch 8/35\n",
            "391/391 [==============================] - 52s 134ms/step - loss: 0.5070 - accuracy: 0.8239\n",
            "Epoch 9/35\n",
            "391/391 [==============================] - 52s 133ms/step - loss: 0.4713 - accuracy: 0.8360\n",
            "Epoch 10/35\n",
            "391/391 [==============================] - 54s 137ms/step - loss: 0.4441 - accuracy: 0.8458\n",
            "Epoch 11/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 0.4083 - accuracy: 0.8582\n",
            "Epoch 12/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 0.3756 - accuracy: 0.8682\n",
            "Epoch 13/35\n",
            "391/391 [==============================] - 53s 134ms/step - loss: 0.3450 - accuracy: 0.8786\n",
            "Epoch 14/35\n",
            "391/391 [==============================] - 52s 133ms/step - loss: 0.3165 - accuracy: 0.8901\n",
            "Epoch 15/35\n",
            "391/391 [==============================] - 54s 137ms/step - loss: 0.2941 - accuracy: 0.8955\n",
            "Epoch 16/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 0.2721 - accuracy: 0.9043\n",
            "Epoch 17/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 0.2538 - accuracy: 0.9118\n",
            "Epoch 18/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 0.2420 - accuracy: 0.9153\n",
            "Epoch 19/35\n",
            "391/391 [==============================] - 55s 139ms/step - loss: 0.2214 - accuracy: 0.9223\n",
            "Epoch 20/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 0.2108 - accuracy: 0.9259\n",
            "Epoch 21/35\n",
            "391/391 [==============================] - 53s 136ms/step - loss: 0.1980 - accuracy: 0.9291\n",
            "Epoch 22/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 0.1834 - accuracy: 0.9352\n",
            "Epoch 23/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 0.1732 - accuracy: 0.9386\n",
            "Epoch 24/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 0.1660 - accuracy: 0.9402\n",
            "Epoch 25/35\n",
            "391/391 [==============================] - 54s 138ms/step - loss: 0.1563 - accuracy: 0.9448\n",
            "Epoch 26/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 0.1463 - accuracy: 0.9489\n",
            "Epoch 27/35\n",
            "391/391 [==============================] - 52s 134ms/step - loss: 0.1434 - accuracy: 0.9491\n",
            "Epoch 28/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 0.1346 - accuracy: 0.9529\n",
            "Epoch 29/35\n",
            "391/391 [==============================] - 54s 137ms/step - loss: 0.1264 - accuracy: 0.9559\n",
            "Epoch 30/35\n",
            "391/391 [==============================] - 53s 136ms/step - loss: 0.1191 - accuracy: 0.9582\n",
            "Epoch 31/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 0.1195 - accuracy: 0.9577\n",
            "Epoch 32/35\n",
            "391/391 [==============================] - 54s 137ms/step - loss: 0.1133 - accuracy: 0.9611\n",
            "Epoch 33/35\n",
            "391/391 [==============================] - 53s 136ms/step - loss: 0.1094 - accuracy: 0.9614\n",
            "Epoch 34/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 0.1108 - accuracy: 0.9609\n",
            "Epoch 35/35\n",
            "391/391 [==============================] - 53s 135ms/step - loss: 0.1070 - accuracy: 0.9634\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Loss vs. Epoch graph"
      ],
      "metadata": {
        "id": "odxJmS4ZDeUH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fig, ax = plt.subplots(1, 1, dpi=100)\n",
        "ax.plot(hist.history['loss'])\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 362
        },
        "id": "mVkzsz7RcFYC",
        "outputId": "2d6599ee-ac58-4d72-d95d-0b6d7749dfdb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 600x400 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgQAAAFZCAYAAADn+jciAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxcdb3/8ddnJvveJG3SNE1XWgoFKoVi2bVwQeHKIgKiF/EqihaX63bF5aq41BX0CuLjqhdFRfHnRYQrpexc1pbd7gt0b5amafZ95vv7YybpNCZpJp2Zk5l5Px+P85jMyTmZz+G05N3vdsw5h4iIiKQ3n9cFiIiIiPcUCERERESBQERERBQIREREBAUCERERQYFAREREUCAQERERIMPrAsbCzAyoAtq8rkVERCQJFQL73CiLDyVFICAUBvZ4XYSIiEgSqwb2jvTNZAkEbQC7d++mqKjI61pERESSRmtrK9OnT4cjtLInSyAAoKioSIFAREQkDjSoUERERBQIRERERIFAREREUCAQERERFAhEREQEBQIRERFBgUBERERQIBAREREUCERERIQ0DgSBoKOhrdvrMkRERCaEqAOBmZ1tZg+Y2T4zc2Z26RjOyTazb5vZTjPrMbMdZvav4yv56O1t7mL+V1Zy5veeIBgc8cFPIiIiaWM8zzLIB14H/hu4d4zn/AmoAD4EbAOm4mHrxOSCbALO0d/vONDRy+TCbK9KERERmRCiDgTOuZXASgAzO+LxZnYhcA4w2znXFN69I9rPjaWsDB8VhTnUtXazt7lLgUBERNJeIv6V/i7gJeALZrbXzLaY2Q/NLHekE8JdDEUDG1AY66KqSnIA2NfcFesfLSIiknQSEQhmA2cCC4HLgE8DVwA/G+Wcm4CWiG1PrIuaNikPgL0HFQhEREQSEQh8gAPe55xb45x7EPgM8IFRWglWAMURW3WsixpoIdirFgIREZFxDSqMVi2w1znXErFvI2CEftFvHXqCc64H6Bl4P5axCtGqLgllEQUCERGRxLQQPAtUmVlBxL55QJA4dAWMVVU4EGgMgYiIyPjWISgws0Vmtii8a1b4fU34+yvM7K6IU+4GDgB3mtlxZnY28APgv51znv02njZJLQQiIiIDxtNCcArwangDuCX89c3h91OBmoGDnXPtwPlACaHZBr8HHgA+Ob6SY2OghaC5s4+Onn4vSxEREfHceNYheJJQ//9I379umH2bCIWCCaMoJ5PCnAzauvvZ19zFMRUxn9koIiKSNNL2WQYA08KtBHvUbSAiImlOgQANLBQREUnvQDAwsFCLE4mISJpL60CgqYciIiIhaR0IpmlxIhERESDNA8GhFoJujysRERHxVloHgurwGIK61m76A0GPqxEREfFOWgeCyQXZZPqNQNBR39Zz5BNERERSVFoHAp/PmFqsmQYiIiJpHQjg0GOQNdNARETSWdoHgmkleYBmGoiISHpTIAi3ECgQiIhIOlMg0GqFIiIiCgQDXQYaQyAiIuks7QNBVUSXgXPO42pERES8oUAQXq2wszdAS1efx9WIiIh4I+0DQU6mn/KCLAD2aByBiIikqbQPBKCHHImIiCgQoMcgi4iIKBAQ0UKgLgMREUlTCgREtBC0KBCIiEh6UiBAixOJiIgoEBA5qLDb40pERES8oUDAoUDQ2N5Dd1/A42pEREQST4EAKMnLJDfTD0Bti1oJREQk/UQdCMzsbDN7wMz2mZkzs0ujOPcMM+s3s9ei/dx4MjONIxARkbQ2nhaCfOB1YHk0J5lZCXAX8Ng4PjPupmktAhERSWMZ0Z7gnFsJrITQv6yj8HPgbiAAjLlVIVEGph7uUSAQEZE0lJAxBGb2QWA28I0xHp9tZkUDG1AY1wKB6klqIRARkfQV90BgZscA3wXe75zrH+NpNwEtEdueOJU3aPAxyBpDICIiaSiugcDM/IS6Cb7mnNsSxakrgOKIrToO5R1mWkkeoNUKRUQkPUU9hiBKhcApwFvM7LbwPh9gZtYP/JNz7vGhJznneoCegfdRjlUYl4EWgtrmboJBh88X/88UERGZKOIdCFqBE4bs+zjwduAKYHucP3/MKoty8Bn0BoLsb++hoijH65JEREQSJupAYGYFwNyIXbPMbBHQ5JzbZWYrgGnOuWudc0Fg3ZDzG4Bu59xh+72W4fdRWZTDvpZu9jZ3KRCIiEhaGc8YglOAV8MbwC3hr28Ov58K1Bx9aYmnxYlERCRdjWcdgieBETvYnXPXHeH8rwNfj/ZzEyG0FsFBTT0UEZG0o2cZRDj01EMFAhERSS8KBBGqtHyxiIikKQWCCANjCPZoDIGIiKQZBYII1WohEBGRNKVAEGGgy6C1u5+27j6PqxEREUkcBYII+dkZlORlArCvudvjakRERBJHgWCIquKBmQadHlciIiKSOAoEQwwuTqQWAhERSSMKBEMMrkWgmQYiIpJGFAiGmKaZBiIikoYUCIao0mqFIiKShhQIhhgYQ6AWAhERSScKBENUlYQee1zf2k1fIOhxNSIiIomhQDBEeX42WRk+gg7qWjTTQERE0oMCwRA+n1FVHGol0DgCERFJFwoEwxhci0BTD0VEJE0oEAxDUw9FRCTdKBAMQ1MPRUQk3SgQDGOaAoGIiKQZBYJhKBCIiEi6USAYRuTiRM45j6sRERGJPwWCYVSGpx129wVp6uj1uBoREZH4UyAYRnaGnymF2QDs02OQRUQkDSgQjODQTINOjysRERGJPwWCEQwuTqQWAhERSQMKBCMYnGmg1QpFRCQNRB0IzOxsM3vAzPaZmTOzS49w/OVm9oiZ7TezVjN73swuGH/JiaHVCkVEJJ2Mp4UgH3gdWD7G488GHgHeCSwGngAeMLO3jOOzE0arFYqISDrJiPYE59xKYCWAmY3l+E8P2fUlM7sE+Gfg1eHOMbNsIDtiV2G0dR4tLU4kIiLpJOFjCMzMR+gXfNMoh90EtERsexJQ2mEGBhU2dfTS1RtI9MeLiIgklBeDCj8HFAB/GuWYFUBxxFadgLoOU5STQUF2qAFFrQQiIpLqEhoIzOwa4GvAlc65hpGOc871OOdaBzagLWFFhpmZBhaKiEjaSFggMLOrgV8SCgOPJupzj0ZVSWgJY7UQiIhIqktIIDCz9wJ3Au91zv0tEZ8ZC5EPORIREUllUc8yMLMCYG7Erllmtghocs7tMrMVwDTn3LXh468BfgN8ClhtZpXh87qccy1HV358VWlxIhERSRPjaSE4hdB0wYEpg7eEv745/H4qUBNx/EcIBY/bgdqI7Sfj+OyE0tRDERFJF+NZh+BJYMQFCJxz1w15f260nzFRKBCIiEi60LMMRjEwhqCupZtA0HlcjYiISPwoEIxiSmEOGT6jP+hoaNNTD0VEJHUpEIzC7zMqi0NTDzXTQEREUpkCwREMzDTYo5kGIiKSwhQIjqB6cLVCdRmIiEjqUiA4goGBhXubOz2uREREJH4UCI6gSi0EIiKSBhQIjmCaVisUEZE0oEBwBFURixM5p7UIREQkNSkQHMFAC0F7Tz+t3f0eVyMiIhIfCgRHkJvlpzQ/C1C3gYiIpC4FgjGYVqLHIIuISGpTIBiDqpLQaoV6yJGIiKQqBYIxmFaSB6iFQEREUpcCwRgMtBDsUSAQEZEUpUAwBtWTNIZARERSmwLBGFRpcSIREUlxCgRjMDDLoKGth57+gMfViIiIxJ4CwRiU5meRkxn6T1XXomcaiIhI6lEgGAMzO2wJYxERkVSjQDBGesiRiIikMgWCMZpdng/AK7uaPa5EREQk9hQIxuj84yoBWLW+jv5A0ONqREREYkuBYIzeOruU0vwsmjp6eeHNJq/LERERiSkFgjHK8Pu44PhQK8Hf1tZ6XI2IiEhsRR0IzOxsM3vAzPaZmTOzS8dwzrlm9oqZ9ZjZNjO7blzVeuyiE6YC6jYQEZHUM54WgnzgdWD5WA42s1nA34AngEXAj4FfmtkF4/hsT6nbQEREUlXUgcA5t9I59xXn3F/GeMoNwHbn3Gedcxudc7cBfwb+LdrP9pq6DUREJFUlYgzBUuDRIftWhfcPy8yyzaxoYAMK41lgNNRtICIiqSgRgaASqB+yrx4oMrPcEc65CWiJ2PbEr7zoqNtARERS0USdZbACKI7Yqr0t5xB1G4iISCpKRCCoAyqG7KsAWp1zw64D7Jzrcc61DmxAW7yLjIa6DUREJNUkIhA8Dywbsu/88P6kpG4DERFJNeNZh6DAzBaZ2aLwrlnh9zXh768ws7siTvk5MNvMvm9mx5rZx4ErgVuPunqPqNtARERSzXhaCE4BXg1vALeEv745/H4qUDNwsHNuO3ARoVaB14HPAh92zq0aZ80TgroNREQklWREe4Jz7knARvn+dSOc85ZoP2siG9ptcOYx5V6XJCIiMm4TdZbBhKduAxERSSUKBEdB3QYiIpIqFAiOgmYbiIhIqlAgOArqNhARkVShQHCU1G0gIiKpQIHgKKnbQEREUoECwVFSt4GIiKQCBYIYULeBiIgkOwWCGFC3gYiIJDsFghhQt4GIiCQ7BYIYufhEdRuIiEjyUiCIkdNmqdtARESSlwJBjBzebbDP42pERESio0AQQ4e6DerVbSAiIklFgSCG1G0gIiLJSoEghtRtICIiyUqBIMbUbSAiIslIgSDG1G0gIiLJSIEgxtRtICIiyUiBIA7UbSAiIslGgSAO1G0gIiLJRoEgDtRtICIiyUaBIE4Gug3uf20f+5q7PK5GRERkdAoEcbJ0dhmLZ0yiozfA1+5f73U5IiIio1IgiBOfz1hx+Qlk+o1HNtTz0Lo6r0sSEREZkQJBHM2rKOSjZ88B4Gv3r6O1u8/jikRERIY3rkBgZsvNbIeZdZvZajNbcoTjP21mm82sy8x2m9mtZpYzvpKTy41vn8vMsjzqW3v44arNXpcjIiIyrKgDgZldBdwCfAM4GXgdWGVmU0Y4/hrgu+HjFwAfAq4CvjPOmpNKTqaf71x2AgC/fWEnL+886HFFIiIi/2g8LQSfAX7hnLvTObcBuAHoBP51hONPB551zt3tnNvhnHsY+AMwYquCmWWbWdHABhSOo84J4/S55bz75Gqcgy/du5Y+LVYkIiITTFSBwMyygMXAowP7nHPB8PulI5z2HLB4oFvBzGYD7wQeHOWjbgJaIrY90dQ5EX35ogVMystkc30bv3j6Ta/LEREROUy0LQTlgB+oH7K/Hqgc7gTn3N3AfwDPmFkf8AbwpHNutC6DFUBxxFYdZZ0TTml+Fl+9+DgAfvLoVnYe6PC4IhERkUPiPsvAzM4FvgR8nNCYg8uBi8zsqyOd45zrcc61DmxAW7zrTITL3jKNM+aW0dMf5Mt/WYdzzuuSREREgOgDQSMQACqG7K8ARppo/03gt865Xzrn1jrn/kIoINxkZmk17dHM+PalJ5Cd4eOZbY385dW9XpckIiICRBkInHO9wMvAsoF94V/qy4DnRzgtDxg6ii4wcHo0n58KZpbn88llxwDwrb9tpKmj1+OKRERExtdlcAtwvZl9wMwWAHcA+cCdAGZ2l5mtiDj+AeBjZna1mc0ys/MJtRo84JwLDP3h6eAjZ89mfkUhTR29fOfBjV6XIyIiEn0gcM7dA3wOuBl4DVgEXOicGxhoWANMjTjlW8CPwq8bgF8Bq4CPjr/s5Jbp9/Gdy0/ADP788h6e29bodUkiIpLmLBkGtoXXImhpaWmhqKjI63Ji5qv3reO3L+xkZlkeD336bHIy/V6XJCIiKaa1tZXi4mKA4vBA/WGl1aC+iebzF85nSmE2Ow50ctvj27wuR0RE0pgCgYeKcjK5+ZLjAfj5U2+wpT4lZleKiEgSUiDw2AXHV3Leggr6g46b7l1LMDjxu3BERCT1KBB4zMy4+ZLjyc/y8/LOg9y9ZpfXJYmISBpSIJgAqkpy+dwF8wH43spN1LZ0eVyRiIikGwWCCeLapTM5qbqYtp5+bvjty3T3peUSDSIi4hEFggnC7zN++t6TKcnL5PU9LXzp3rV61oGIiCSMAsEEUlOWx+3XnIzfZ9z76l5+9cx2r0sSEZE0oUAwwZwxt5wvv3MBAN95cCPPbNUqhiIiEn8KBBPQB8+YybtPriboYPndr7DzQIfXJYmISIpTIJiAzIxvX7aQk6aX0NLVx/V3vUR7T7/XZYmISApTIJigcjL9/Ne/LGZyYTZb6tv57J9e06JFIiISNwoEE1hFUQ4/f/9isvw+Vq2v5z8f3+p1SSIikqIUCCa4xTMm8a3LFgLw40e3smp9nccViYhIKlIgSAJXnjKd606fCcBn7nlND0ESEZGYUyBIEl++aAFLZ5fR0Rvg+rteormz1+uSREQkhSgQJIlMv4/b33cy1ZNy2Xmgk0/84VX6A0GvyxIRkRShQJBESvOz+MW1p5Cb6efprY18d+Umr0sSEZEUoUCQZBZMLeKH7zkJgF8+s517X9njcUUiIpIKFAiS0EUnTuUTb58LwBfvXctru5s9rkhERJKdAkGS+rfz5nHegin09gd53y9e4GFNRxQRkaOgQJCkfD7j1qsWDc48+MhvX+anj23VI5NFRGRcFAiSWGFOJnd9aAkfWDoDgB89soUb736Vzl4990BERKKjQJDkMv0+vnHJQlZcfgKZfuNva2u54o7n2XOw0+vSREQkiSgQpIj3Lqnh7uvfSnlBFhtqW7nktmdZ/eYBr8sSEZEkMa5AYGbLzWyHmXWb2WozW3KE40vM7HYzqzWzHjPbYmbvHF/JMpJTZ5by1xvP5PiqIg509PK+X67mdy/s9LosERFJAlEHAjO7CrgF+AZwMvA6sMrMpoxwfBbwCDATuAKYD1wP7B1fyTKaaSW5/PmG07n4xKn0Bx1fuW8dX/7LWnr7taqhiIiMzKIdlW5mq4EXnXM3ht/7gN3AT51z3x3m+BuAzwPHOuf6xlWkWRHQ0tLSQlFR0Xh+RNpxzvGzJ9/ghw9vxjlYMquUO953MmUF2V6XJiIiCdTa2kpxcTFAsXOudaTjomohCP9rfzHw6MA+51ww/H7pCKe9C3geuN3M6s1snZl9ycz8o3xOtpkVDWxAYTR1CpgZy982l19eewoF2Rms2d7Eu257lg37RvyzICIiaSzaLoNywA/UD9lfD1SOcM5sQl0FfuCdwDeBzwJfGeVzbgJaIjatzztOyxZUcN/y05lZlsfe5i7efcdz/O3vtV6XJSIiE0wiZhn4gAbgI865l51z9wDfBm4Y5ZwVQHHEVh33KlPY3CmF/HX5mZx1TDldfQGW3/0Kn/jDqzS0dXtdmoiITBDRBoJGIABUDNlfAYy0dm4tsMU5F4jYtxGoDHdB/APnXI9zrnVgA9qirFOGKM7L5M7rTuWGc+bgM3jg9X0s+9FT/H71ToJBrW4oIpLuogoEzrle4GVg2cC+8KDCZYTGCQznWWBu+LgB84Da8M+TBMnw+/jiO47lr8vPZOG0Itq6+/nyX9Zxxc+fY3OdMpeISDobT5fBLcD1ZvYBM1sA3AHkA3cCmNldZrYi4vg7gFLgJ2Y2z8wuAr4E3H50pct4nVBdzH0fP4P/uPg48rP8vLKrmYv+82m+99AmunoDR/4BIiKScqKedghgZjcSmkpYCbwGfNI5tzr8vSeBHc656yKOXwrcCiwitP7Ar4DvDelGGO3zNO0wTmpbuvj6/etZtT40TnR6aS7fvGQh584fdlkJERFJMmOddjiuQJBoCgTx9/D6Or52/3pqW0IDDf/5pCq+evECphTmeFyZiIgcDQUCiVp7Tz+3PrKFO5/dTtBBYU4G/37hsVyzpAafz7wuT0RExkGBQMZt3d4Wbrp3LWv3tgBwck0J37r0BI6r0n97EZFko0AgRyUQdPz2+R38YNVmOnoD+AyuXlLDZ8+fp+WPRUSSiAKBxERtSxff+tvGwdUNC3My+PR587h26Qwy/Xp6tojIRKdAIDG1ZnsT33hgPevDz0KYMzmfr158nGYjiIhMcAoEEnOBoOP/vbSbH6zazIGO0JpSbz92Cl+5aAGzJxd4XJ2IiAxHgUDipqWrj58+tpVfP7eD/qAjw2d88IyZfGLZMRTlZHpdnoiIRFAgkLh7Y3873/7bRh7f1ABAWX4Wn79gPu85ZTp+TVMUEZkQFAgkYZ7Y3MA3/3cDb+7vAOD4qiL+4+LjOG12mceViYiIAoEkVG9/kLue38FPHttKW3c/AOfOn8zn/mk+C6cVe1uciEgaUyAQTzS293DLI1u458XdBMKPVb7ohKn82/nzmDtFAw9FRBJNgUA8taOxg1sf3cL9r+/DOfAZXH5yNZ9adgzTS/O8Lk9EJG0oEMiEsKmulR89vIVHNoSeppjpN65ZUsPyt8/Vg5NERBJAgUAmlFd3HeSHD2/m2W0HAMjJ9HHd6bO44ZzZlORleVydiEjqUiCQCem5bY384OHNvLqrGYDC7Aw+cvZsPnjmLAqyMzyuTkQk9SgQyITlnOPxTQ38YNVmNtW1AVCan8V7l0zn6lNrNMZARCSGFAhkwgsGHf+7tpZbH9nC9sbQGgZmcNYxk7lmyXSWLajQA5RERI6SAoEkjb5AkIfX1/PHF3fx9NbGwf3lBdlceUo1V59aQ02ZWg1ERMZDgUCS0s4DHdzz4m7+9NIeGtt7BvefdUw5711Sw3kLKsjKUKuBiMhYKRBIUusLBHlsYz13r9nN01v3M/DHtLwgiysWT+fqU6czszzf2yJFRJKAAoGkjN1Nndzz4m7ueWk3+9sOtRq8/dgpfOb8eVoaWURkFAoEknL6AkEe39TAH9bs4qkth1oNLjpxKp85fx5zJmtpZBGRoRQIJKXtaOzgx49u4a8RSyNfsbiaT503j2kluV6XJyIyYSgQSFrYWBtaGvnRjaGlkbP8Pt731hqWv20u5QXZHlcnIuI9BQJJK6/sOsgPHtrM82+GlkbOy/Lzr2fM4vqzZ1Ocm+lxdSIi3hlrIBjX/C0zW25mO8ys28xWm9mSMZ53tZk5M7tvPJ8rMpKTayZx9/Wn8bsPncZJ1cV09ga47YltnP39J7jjyTfo7O33ukQRkQkt6hYCM7sKuAu4AVgNfBp4DzDfOdcwynkzgWeAN4Em59ylUXymWghkzJxzPLyhnh+u2szWhnYAJhdmc+Pb5nLVqdPJyfR7XKGISOLErcvAzFYDLzrnbgy/9wG7gZ865747wjl+4P+A/wbOAkoUCCTeAkHHX1/by62PbmF3UxcAUwqz+cjZs3nvkhry9TAlEUkDcQkEZpYFdAJXOOfui9j/G0K/5C8Z4bxvACc65y4zs19zhEBgZtlA5IiwQmCPAoGMR29/kHte3MXPnnyD2pZuACblZfLBM2bxgdNnaoyBiKS0eI0hKAf8QP2Q/fVA5XAnmNmZwIeA66P4nJuAlohtT5R1igzKyvDxL0tn8tTn38b33n0CM8vyONjZxy2PbOGM7z7O9x7adNgyySIi6Siui8KbWSHwW+B651zjkY6PsAIojtiq41CepJmsDB9XnVrDo585h59cvYj5FYW09/Rzx5NvcMZ3H+fr969nX3OX12WKiHgirl0GZrYIeBUIROweCCFBQgMR3xjD52oMgcRcMOh4dGM9tz+xjdf3tACQ6Tcuf0s1Hzt3jp6VICIpId6DCtc45z4Rfu8DdgG3DR1UaGY5wNwhP+JbhMYEfArY4pzrHcNnKhBI3DjneHbbAW57YisvvNkEhFY+vPjEKj527hwWTNWfORFJXvEMBFcBvwE+CqwhNO3wSuBY51y9md0F7HXO3TTC+b9Gswxkgnp5ZxO3Pb6NJzbvH9x35txyPnzWLM6ZNxkz87A6EZHojTUQRD3vyjl3j5lNBm4mNJDwNeBC59zAQMMaQt0BIkln8YxS7vzgEtbtbeGOp95g5dpantnWyDPbGjlmSgEfPmsWlyyaprUMRCTlaOlikVHsburk18/t4I9rdtHRGxoKU16QxbVLZ/L+t86gND/L4wpFREanZxmIxFBrdx9/XLOLO5/dMbiWQXaGj3cvruZDZ87So5dFZMJSIBCJg75AkAfX1vLLp7ezdm/L4P7zFkzhw2fN5rRZpRpnICITigKBSBw551izvYlfPL2dxzbVM/DXaOG0Iq5dOpN3nVSlcQYiMiEoEIgkyJv72/nVM9v5n1f20N0XGk9bkpfJladM5/2nzaCmLM/jCkUknSkQiCRYU0cvf3ppN797YSd7DoZWPDSDc+dN5tqlMzln3mR8PnUniEhiKRCIeCQQdDy5uYG7nt/JU1sOrWdQU5rH+99aw5WnTKckT7MTRCQxFAhEJoAdjR387oWd/Oml3bR29wOh2QnvOqmKa5fO5ITqYo8rFJFUp0AgMoF09Qa4//W93PX8TtbvO/T38aTpJVxx8jQuOL6SKUU5HlYoIqlKgUBkAnLO8cquZn77/A4eXFtHbyA0CNEMTp1RyjtOqOTChZVMLc71tlARSRkKBCITXGN7D395ZS8Prqvl1V3Nh33v5JoS3rFwKhcurGR6qWYpiMj4KRCIJJF9zV08tK6OletqeWnnQSL/Wp5YXcw7Fk7lHQsr9UhmEYmaAoFIkqpv7WbV+joeXFvLmu1NBCP+ii6YWsQ7F1byjhMqmTul0LsiRSRpKBCIpID9bT08vKGOlWvreP7NAwQi0sGcyfmD3QrHVxVpyWQRGZYCgUiKaero5ZENdaxcV8ez2xrpCxz6uzu9NJcLj6/kwoVTecv0Ei2AJCKDFAhEUlhrdx+Pb2xg5bpantqyf3DJZICKomwuOD40W2HJzFIy/D4PKxURrykQiKSJzt5+ntq8n4fW1/HYxgbae/oHv1ean8X5Cyo4d/5kls4p0wqJImlIgUAkDfX0B3hu2wFWrqvlkQ31HOzsG/yeGZw4rZgz5pZz5txyFs+cRHaGnsgokuoUCETSXH8gyJrtTTyysZ5ntjaytaH9sO/nZPo4dWYpZ84t58xjyllQWaSxByIpSIFARA5T39rNM1sbeXZbI89sa6Shreew75fmZ3H6nDLOOqacxTNKmVmWp/EHIilAgUBERuScY2tDO0+HA8ILbx6gszdw2DFZGT6OmVLA/IpC5lcWMq+ykGMrC6ksytEUR5EkokAgImPW2x/ktcrsX7kAAA7DSURBVN3NPLMtFBA27Gulqy8w7LFFORmhgFARCgih1yKK8zITXLWIjIUCgYiMWzDo2H2wk011bWyua2Nzfeh1e2PHYYsjDTCDRdNLOG9BBcsWTGF+RaFaEUQmCAUCEYm5nv4AbzR0sKW+jU11bWwJB4W9zV2HHTetJJdlC6awbEEFb51dqtkMIh5SIBCRhKlr6eaxTfU8trGBZ7c10tN/aKGk/Cw/Zx0zmWULpvD2Y6dQVpDtYaUi6UeBQEQ80dnbz7PbDvDYxnoe29TA/ojZDGbwluklLFtQwSkzJjFnSgFl+VnqXhCJIwUCEfFcMOhYu7eFxzbW8+jGBjbU/uP/i4pzM5kzOZ85kwuYM6Ug9Do5n5pSTXsUiYW4BgIzWw58HqgEXgc+4ZxbM8Kx1wPXAgvDu14GvjTS8SP8DAUCkRSwr7mLxzY18NTmBjaFxx6M9L+gTL8xoyx/MCzMnVLAcVVFzJlcQKaCgsiYxS0QmNlVwF3ADcBq4NPAe4D5zrmGYY7/PfAs8BzQDfw7cBlwvHNu7xg/U4FAJAV19QbY3tjBG/vbw1sHbzS082Zj+2EPbIqU5fcxr7KA46YWhbaqYhZMLaQwR9MeRYYTz0CwGnjROXdj+L0P2A381Dn33TGc7wcOAjc65+4a42cqEIikkWDQUdvazRsN7YNhYXNdGxtr2w57eFOkmtK8cEAo4viq0KsWURKJUyAwsyygE7jCOXdfxP7fACXOuUvG8DMKgQbgPc65/x3hmGwgcihyIbBHgUAkvQWDjj0Hu9hQ28KGfa1sqG1lw75W9rV0D3t8eUE2i6YXs2h6CYumT+LE6cUUqSVB0ky8AkEVsBc43Tn3fMT+7wPnOOdOG8PP+BlwAaEug2H/FpvZ14GvDd2vQCAiwznY0cvG2kMBYUNtK1sb2oddRGnO5HwWTZ/EopoSFlWXcOzUQo1JkJQ2IQOBmX0R+AJwrnPu76McpxYCETkq3X0B1u9r4bXdLby2u5nXdh9kd1PXPxyXneFj4bRiTqou4YTqImaXFzCzPJ/iXLUkSGoYayDIiPLnNgIBoGLI/gqgbrQTzexzwBeB80YLAwDOuR5gcPKy+gBFJFo5mX4Wzyhl8YzSwX2N7T38fU8zr+1q5tXdzby+u5nW7n5e3nmQl3cePOz88oIsZpXnh7cCZpXnM3tyPjPK8rTyoqSk8Q4qXOOc+0T4vQ/YBdw20qBCM/sC8GXgAufcC1EXqUGFIhIHzjm2N3aEWxCaB5/XMPTR0JHMoHpSbigklOUxtSSXqcU5VBTlUFmUQ2VxDjmZCgwyccR72uFvgI8CawhNO7wSONY5V29mdwF7nXM3hY//d+Bm4BpC0w8HtDvn2sf4mQoEIpIw7T397AhPh9ze2DG4vbm/Y8RZDpFK8jKpLAqFhMGwUBwKDNWTcplemqfQIAkTry4DnHP3mNlkQr/kK4HXgAudc/XhQ2qAyAnEHwOygD8P+VHfAL4e7eeLiMRbQXYGC6cVs3Ba8WH7nXPsb+9h+/5wSDjQQX1LN3Wt3dSFX7v7gjR39tHc2cemurZhf74ZTC3KYWZ5PjPK8plZlhd6Lc9jRmk+uVkKC5J4WrpYRCRGnHO0dvWHAkJrN3UtXdS19FDX2k19aze1Ld3saeqk7QitDJVFOcwoy2NmWT4zyvOYHR7HMKNMLQsSPT3LQERkAnLO0dTRy44Dnew80MGOA53saOxg54FQq0Nr98hhwQyqinMHBzvOLM8Ph4V8qifl6tkPMiwFAhGRJNTc2cv2xg52Huhkx4EOdjR2sP1AJ2/ub6dtlLCQ4TNqSvOYVZ5PaX4WeVl+crMyyMvyk5flJyfTP/j1wP7cTD+5WX6KcjIpL9BTJ1OVAoGISAo51LIQGty4vbFj8OsdBzpGfPbDWGVn+AYHPE6flMf00lyqJx36ujg3U4EhSSkQiIikiWDQUd/WHRrseKCD5s4+unoDdPUF6OwN0NXbH3odfD/wdWh/e0//iE+dHFCYncG0iMAwbVIu00pyqQ6/luQpMExUCgQiIjImvf1Balu62N3UxZ6Dnew+2Mnupi52H+xkz8Eu9o+yLsOA/Cw/VSW5g0Hh8MCQx5TCbHw+BQYvKBCIiEhMdPUG2NvcGREYuth7MPT13uYuGtt7j/gzMnxGRVEOFUXZTC3OPbRGQ3HotbIohylF2VoFMg4UCEREJCG6+wLsbQ6FhOFe61q7h33Q1HDK8rOoCIeDDJ8RdKHxEw4Ofe3A4QgGw68OcJCT5ae8IIvJBdmUF2RTVpBFefjr8sIsSvOy0nImhgKBiIhMCP2BIA1tPYcWcGo5tC5D5KJOvf1HNzDySMygNC9rMCyUFWRTlJNBYU4mhTkZh7bsTArCXxflZFKQHfp6aJgIBh19wSB9AUd/IPwaDNIfcPQFgvSHQ1B++Pz8rAz8HnSbxG2lQhERkWhk+H1UleRSVZI74jHOOZo7+6gNh4WGtm6cC/0SNzMM8JlhdujVzPAZGKH3HT39NLb30tjeM7gdCL8/0NGLc3Cgo5cDHb1QP2IpI8rN9OMz6AuGAsAYGz0Ok5/lpyAng4LsDApyMikMh4XQ+wwKszOYUZbPuxdXR//Dj5ICgYiIeM7MmJSfxaT8LI6rin1LcCAYmrY5NCy0dffT1t1Pe0/f4NdtPf20dffRHn7f1RcAGHwdTYbPyPAbmT4fGf5Qa0B7Tz99gVB66OgN0NEboJ6RB2oumVmqQCAiIhIPfp8xuTCbyYXZUZ/bFwjS0RMKB0HnyPD7yPQZGX7fYb/8M3w24tTLnv7AYMBo7zn02t4TDh4D+7r7qSnNO9rLHRcFAhERkVFk+n2U5GVRkpc17p+RneEnu8BPWUH0gSRR0m+4pYiIiPwDBQIRERFRIBAREREFAhEREUGBQERERFAgEBERERQIREREBAUCERERQYFAREREUCAQERERkmzp4tbWEZ/aKCIiIsMY6+9Oc24cz29MMDObBuzxug4REZEkVu2c2zvSN5MlEBhQBbTF8McWEgoZ1TH+uROdrlvXnS7S9dp13brukY7b50b5pZ8UXQbhCxgx1YxHxCMq25xzadMXoevWdaeLdL12XbeuewRH/G+iQYUiIiKiQCAiIiLpHQh6gG+EX9OJrju9pOt1Q/peu647vcTsupNiUKGIiIjEVzq3EIiIiEiYAoGIiIgoEIiIiIgCgYiIiKBAICIiIqRpIDCz5Wa2w8y6zWy1mS3xuqZ4M7Ovm5kbsm3yuq5YM7OzzewBM9sXvsZLh3zfzOxmM6s1sy4ze9TMjvGq3lgZw3X/epj7/5BX9caKmd1kZi+aWZuZNZjZfWY2f8gxOWZ2u5kdMLN2M/sfM6vwquZYGON1PznMPf+5VzXHgpl9zMz+bmat4e15M3tHxPdT7l7DmK47Jvc67QKBmV0F3EJo3ubJwOvAKjOb4mlhibEemBqxneltOXGRT+ieLh/h+18APgncAJwGdBC6/zmJKS9ujnTdAA9x+P1/bwLqirdzgNuBtwLnA5nAw2aWH3HMrcA/A+8JH18F3JvgOmNtLNcN8AsOv+dfSGSRcbAH+CKwGDgFeBz4q5kdH/5+Kt5rOPJ1QyzutXMurTZgNXBbxHsfoeckfNHr2uJ83V8HXvO6jgRfswMujXhvQC3wuYh9xUA3cLXX9cbrusP7fg3c53VtCbj2yeHrPzvi/vYCV0Qcc2z4mLd6XW+8rju870ngx17XloBrbwI+lC73euh1x/Jep1ULgZllEUpYjw7sc84Fw++XelVXAh0TblJ+08x+b2Y1XheUYLOASg6//y2EQmI63P9zw83Lm83sDjMr87qgOCgOvzaFXxcT+tdz5D3fBOwite750Ose8D4zazSzdWa2wszyEl1YvJiZ38yuJtQ69jxpcq+Hue4BR32vk+JphzFUDviB+iH76wklyVS2GrgO2EyoOelrwNNmttA5ly6PCq0Mvw53/ytJbQ8RajrdDswBvgOsNLOlzrmAp5XFiJn5gB8Dzzrn1oV3VwK9zrnmIYenzD0f4boB7gZ2AvuAE4HvAfOByxNeZAyZ2QmEfhHmAO3AZc65DWa2iBS+1yNdd/jbMbnX6RYI0pZzbmXE27+b2WpCf4CuBH7lTVWSKM65P0a8XWtmfwfeAM4FHvOkqNi7HVhIao6NGc2w1+2c+6+It2vNrBZ4zMzmOOfeSGSBMbYZWESoVeQK4Ddmdo63JSXEsNftnNsQq3udVl0GQCMQAIaOOq0A6hJfjnfCKXoLMNfrWhJo4B7r/jv3JqG/Dylx/83sNuBi4G3OuT0R36oDssysZMgpKXHPR7nu4awOvyb1PXfO9TrntjnnXnbO3URoMO2nSPF7Pcp1D2dc9zqtAoFzrhd4GVg2sC/c3LaMw/tiUp6ZFRBqOq71upYE2k7ofwyR97+I0GyDdLv/1UAZSX7/w9NIbwMuA97unNs+5JCXgT4Ov+fzgRqS+J6P4bqHsyj8mtT3fBg+IJsUvdejGLju4YzrXqdjl8EthJpaXgLWAJ8mNDjjTk+rijMz+yHwAKFugipC0y4DwB+8rCvWwkEnMhXPCvctNjnndpnZj4GvmNlWQgHhm4T63e5LfLWxM9p1h7evAf9DKBDNAb4PbANWJbjUWLsduAa4BGgzs4G+4hbnXJdzrsXMfgXcYmZNQCvwU+B559wL3pQcE6Net5nNCX//QeAAoX7lW4H/c8793YuCY8HMVgArCQ0ULCR0jecCF6TwvR71umN6r72eOuHRdI0bCf1i7CHUtHKa1zUl4Jr/SOgXXw+hOa1/BOZ4XVccrvNcQtOMhm6/Dn/fgJsJ/WLsJjQieZ7XdcfzuoFcQr/4GwhNy9oB/BdQ4XXdMbju4a7ZAddFHJND6BdoE6F1J+4FKr2uPZ7XDUwHngr/gugGthIKgUVe136U1/2r8J/fnvCf50eB81P5Xh/pumN5ry38A0VERCSNpdUYAhERERmeAoGIiIgoEIiIiIgCgYiIiKBAICIiIigQiIiICAoEIiIiggKBiIiIoEAgIiIiKBCIiIgICgQiIiIC/H9p/X4/963aJwAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Smooth and plateu-ing. Cool."
      ],
      "metadata": {
        "id": "Zd79iwzADhw9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Evaluate of test data"
      ],
      "metadata": {
        "id": "JG7rU8WADnGC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model.evaluate(x=ds_test)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "p-DdRbzYbvrS",
        "outputId": "57f35f8c-dc45-4fa0-8f99-b36be80b97ee"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "79/79 [==============================] - 3s 28ms/step - loss: 0.3009 - accuracy: 0.9120\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.3009231686592102, 0.9120000004768372]"
            ]
          },
          "metadata": {},
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "91\\% test accuracy! Pretty good!"
      ],
      "metadata": {
        "id": "YkpbUu91DpLU"
      }
    }
  ]
}