04_deep_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GTY-Msjal7zU"
   },
   "source": [
    "# Creative Machine Learning - Deep learning\n",
    "\n",
    "### Author: Philippe Esling (esling@ircam.fr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Z3M5ezRyrjA3"
   },
   "source": [
    "In this course we will cover\n",
    "1. A brief introduction to [deep learning](#learning) and problems that we face\n",
    "2. A formal presentation of [Auto-Encoders](#ae) (AEs)\n",
    "3. An explanation of how to [implement AEs](#implement) and [layer-wise pretraining](#implement)\n",
    "4. An [more modern application](#application) of AEs as generative models\n",
    "4. An practical exemple of [convolutional denoising AEs](#denoising) for image data **(exercise)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep learning\n",
    "\n",
    "Deep learning is currently the most successful subfield of machine learning that deals with algorithms inspired by the structure and function of the brain through artificial neural networks. It enables computers to learn from vast amounts of data and has revolutionized various fields such as computer vision, natural language processing and speech recognition.\n",
    "\n",
    "## Why deep architectures?\n",
    "Deep architectures are essential because they allow models to learn hierarchical representations from raw input data. These hierarchical representations can capture increasingly complex and abstract features as the network depth increases. This capability enables deep learning models to achieve state-of-the-art results in many tasks, as they can automatically learn to extract relevant features from the input data without relying on hand-engineered features.\n",
    "\n",
    "**References:** \n",
    "- Goodfellow, I., Bengio, Y., & Courville, A. (2016). [Deep Learning](http://www.deeplearningbook.org). MIT Press.\n",
    "\n",
    "## Challenges in deep architectures\n",
    "\n",
    "As seen in the course, there are several impediments in learning deep architectures, namely\n",
    "1. Vanishing gradient\n",
    "2. Overfitting issues\n",
    "3. Multiple local minima\n",
    "4. Instability in the training\n",
    "Here, we briefly discuss the two major problems, namely _vanishing gradient_ and _overfitting_.\n",
    "\n",
    "#### Vanishing gradient\n",
    "One of the main challenges in training deep architectures is the vanishing gradient problem. It occurs when gradients of the loss function with respect to the model parameters become too small as they backpropagate through the layers. This issue leads to slow convergence and poor performance, as the weights in the earlier layers do not get updated significantly during training.\n",
    "\n",
    "#### Overfitting\n",
    "Another challenge in deep learning is overfitting, where the model learns to perform well on the training data but fails to generalize to unseen data. Regularization techniques, such as L1 and L2 regularization, dropout, and weight decay, are used to prevent overfitting by adding constraints to the model or by making the training process more robust.\n",
    "\n",
    "**References:**\n",
    "- Pascanu, R., Mikolov, T., & Bengio, Y. (2013). [On the difficulty of training recurrent neural networks](https://proceedings.mlr.press/v28/pascanu13.html). In International conference on machine learning (pp. 1310-1318).\n",
    "- Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). [Dropout: A simple way to prevent neural networks from overfitting](http://jmlr.org/papers/v15/srivastava14a.html). Journal of Machine Learning Research, 15(1), 1929-1958.\n",
    "\n",
    "### Layer-wise Pretraining\n",
    "To address the vanishing gradient problem and improve the training of deep architectures, layer-wise pretraining was proposed. In this approach, each layer of the deep architecture is pretrained independently, usually with unsupervised learning methods like autoencoders, before fine-tuning the entire network using supervised learning.\n",
    "\n",
    "**References:**\n",
    "- Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). [A fast learning algorithm for deep belief nets](https://www.cs.toronto.edu/~hinton/absps/fastnc.pdf). Neural computation, 18(7), 1527-1554.\n",
    "- Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P. A. (2008). [Extracting and composing robust features with denoising autoencoders](https://dl.acm.org/doi/10.1145/1390156.1390294). In Proceedings of the 25th international conference on Machine learning (pp. 1096-1103).\n",
    "\n",
    "\n",
    "#### Autoencoders\n",
    "_Autoencoders_ are unsupervised learning models that learn to reconstruct their input data. They consist of an encoder that maps the input data to a lower-dimensional representation and a decoder that reconstructs the input data from the lower-dimensional representation. Autoencoders can be used to learn useful features from the data and initialize the weights of a deep architecture during layer-wise pretraining. As we have seen, an interesting variant of autoencoders are _denoising autoencoders_ that learn to reconstruct the input data from a corrupted version of it. By learning to recover the original data from the corrupted version, denoising autoencoders can capture more robust and meaningful features. They can be used in layer-wise pretraining to improve the training of deep architectures further."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A brief refresher in Pytorch\n",
    "\n",
    "As we have seen previously, `Pytorch` is a Python-based scientific computing package targeted at deep learning, which provides a very large flexibility and easeness of use for GPU calculation. `Pytorch` is constructed around the concept of `Tensor`, which is very similar to `numpy.ndarray`, but can be seamlessly run on GPU. Here, we provide a more detailed refresher on concepts of `Pytorch`.\n",
    "\n",
    "Here are some examples of different `Tensor` creation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "# Create a 5 x 3 Tensor of zeros\n",
    "x = torch.empty(5, 3)\n",
    "# Create a 64 x 3 x 32 x 32 random Tensor\n",
    "x = torch.rand(64, 3, 32, 32)\n",
    "# Create a Tensor of zeros with _long_ type\n",
    "x = torch.zeros(10, 10, dtype=torch.long)\n",
    "# Construct a Tensor from the data\n",
    "x = torch.tensor([5.5, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or create a tensor based on an existing tensor. These methods\n",
    "will reuse properties of the input tensor, e.g. dtype, unless\n",
    "new values are provided by user\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([8, 2])\n",
      "8\n",
      "torch.Size([8, 2])\n"
     ]
    }
   ],
   "source": [
    "x = x.new_ones(8, 2, dtype=torch.double)      # new_* methods take in sizes\n",
    "y = torch.randn_like(x, dtype=torch.float)    # override dtype!\n",
    "print(x.size())\n",
    "print(x.shape[0])\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Arithmetic operations\n",
    "\n",
    "Tensors provide access to a transparent library of arithmetic operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.4085, 0.2284],\n",
      "        [0.0620, 0.6781],\n",
      "        [0.8168, 0.1245],\n",
      "        [0.3624, 1.1728],\n",
      "        [0.0470, 0.5424],\n",
      "        [0.9522, 0.7917],\n",
      "        [0.7642, 0.4602],\n",
      "        [0.9652, 0.4182]])\n",
      "tensor([[0.6538, 0.8344, 0.8655, 1.0896],\n",
      "        [0.3575, 0.5421, 0.4529, 0.6375],\n",
      "        [0.8396, 0.8638, 1.1605, 1.2986],\n",
      "        [0.8479, 1.0034, 1.1409, 1.3749],\n",
      "        [0.7885, 1.0248, 1.0393, 1.3231],\n",
      "        [1.0653, 1.2531, 1.4353, 1.7238],\n",
      "        [1.1275, 1.1190, 1.5680, 1.7239],\n",
      "        [1.1508, 1.1790, 1.5918, 1.7774]])\n"
     ]
    }
   ],
   "source": [
    "x = torch.rand(8, 2)\n",
    "y = torch.rand(8, 2)\n",
    "z = torch.rand(2, 4)\n",
    "# Equivalent additions\n",
    "a = (x + y)\n",
    "b = (torch.add(x, y))\n",
    "# Add in place\n",
    "x.add_(y)\n",
    "# Put in target Tensor\n",
    "result = torch.empty(8, 2)\n",
    "torch.add(x, y, out=result)\n",
    "# Element_wise multiplication\n",
    "print(x * y)\n",
    "# Matrix product\n",
    "print(x @ z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Slicing and resizing\n",
    "\n",
    "You can slice tensors using the usual Python operators. For resizing and reshaping tensor, you can use ``torch.view`` or ``torch.reshape``"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([16, 32, 32])\n",
      "torch.Size([64, 32, 32])\n",
      "torch.Size([4, 4]) torch.Size([16]) torch.Size([2, 8])\n",
      "torch.Size([2, 2, 2, 2])\n"
     ]
    }
   ],
   "source": [
    "#print(x[:, 1])\n",
    "import torch\n",
    "x = torch.zeros(16, 32, 32)\n",
    "x2 = torch.zeros(64, 32, 32)\n",
    "y = x.view(-1, 32, 32)\n",
    "y2 = x2.view(-1, 32, 32)\n",
    "print(y.size())\n",
    "print(y2.size())\n",
    "x = torch.randn(4, 4)\n",
    "y = x.view(16)\n",
    "z = x.view(-1, 8)  # the size -1 is inferred from other dimensions\n",
    "print(x.size(), y.size(), z.size())\n",
    "\n",
    "z = torch.rand(4, 4)\n",
    "z = z.view(-1, 2, 2, 2)\n",
    "print(z.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have a one element tensor, use ``.item()`` to get the value as a\n",
    "Python number\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-1.8315])\n",
      "-1.83152174949646\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(1)\n",
    "print(x)\n",
    "print(x.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensors have more than **100 operations**, including _transposing, indexing, slicing, mathematical operations, linear algebra, random numbers_, which are all described at [https://pytorch.org/docs/torch](https://pytorch.org/docs/torch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Numpy bridge\n",
    "\n",
    "Converting a Torch Tensor to a Numpy array and vice versa is extremely simple. Note that the Pytorch Tensor and Numpy array **will share their underlying memory locations** (if the Tensor is on CPU), and changing one will change the other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([2., 2., 2., 2., 2.])\n",
      "[2. 2. 2. 2. 2.]\n"
     ]
    }
   ],
   "source": [
    "a = torch.ones(5)\n",
    "b = a.numpy()\n",
    "a.add_(1)\n",
    "print(a)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Going GPU\n",
    "\n",
    "Tensors can be moved onto any device using the ``.to`` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# let us run this cell only if CUDA is available\n",
    "# We will use ``torch.device`` objects to move tensors in and out of GPU\n",
    "if torch.cuda.is_available():\n",
    "    device = torch.device(\"cuda\")          # a CUDA device object\n",
    "    y = torch.ones_like(x, device=device)  # directly create a tensor on GPU\n",
    "    x = x.to(device)                       # or just use strings ``.to(\"cuda\")``\n",
    "    z = x + y\n",
    "    print(z)\n",
    "    print(z.to(\"cpu\", torch.double))       # ``.to`` can also change dtype together!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computation Graphs\n",
    "\n",
    "The concept of a computation graph is essential to efficient deep learning programming, because it allows you to not have to write the back propagation gradients yourself. A computation graph is simply a specification of how your data is combined to give you the output (the forward pass). Since the graph totally specifies what parameters were involved with which operations, it contains enough information to compute derivatives. \n",
    "\n",
    "The fundamental flag ``requires_grad`` allows to specify which variables are going to need differentiation in all these operations. If ``requires_grad=True``, the Tensor object keeps track of how it was created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1., 2., 3.], requires_grad=True)\n",
      "tensor([5., 7., 9.], grad_fn=<AddBackward0>)\n",
      "<AddBackward0 object at 0x12025ec40>\n",
      "tensor([0.3333, 0.3333, 0.3333])\n"
     ]
    }
   ],
   "source": [
    "# Tensor factory methods have a ``requires_grad`` flag\n",
    "x = torch.tensor([1., 2., 3], requires_grad=True)\n",
    "# With requires_grad=True, we can still do all the operations \n",
    "y = torch.tensor([4., 5., 6], requires_grad=True)\n",
    "print(x)\n",
    "z = x + y\n",
    "out = z.mean(-1)\n",
    "print(z)\n",
    "# But z now knows something extra.\n",
    "print(z.grad_fn)\n",
    "out.backward()\n",
    "print(y.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore, `z` knows that it is the direct result of an addition. Furthermore, if we keep following z.grad_fn, we can even find back both `x` and `y`. But how does that help us compute a gradient?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(21., grad_fn=<SumBackward0>)\n",
      "<SumBackward0 object at 0x1202c9c40>\n",
      "<AddBackward0 object at 0x102e864c0>\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# Lets sum up all the entries in z\n",
    "s = z.sum()\n",
    "print(s)\n",
    "print(s.grad_fn)\n",
    "print(z.grad_fn)\n",
    "print(x.grad_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now, what is the derivative of this sum with respect to the first\n",
    "component of x? In math, we want\n",
    "\n",
    "\\begin{align}\\frac{\\partial s}{\\partial x_0}\\end{align}\n",
    "\n",
    "\n",
    "\n",
    "Well, s knows that it was created as a sum of the tensor z. z knows\n",
    "that it was the sum x + y. So\n",
    "\n",
    "\\begin{align}s = \\overbrace{x_0 + y_0}^\\text{$z_0$} + \\overbrace{x_1 + y_1}^\\text{$z_1$} + \\overbrace{x_2 + y_2}^\\text{$z_2$}\\end{align}\n",
    "\n",
    "And so s contains enough information to determine that the derivative we want is 1. We can have Pytorch compute the gradient, and see that we were right:\n",
    "\n",
    "**Note** : If you run this block multiple times, the gradient will increment. That is because Pytorch *accumulates* the gradient into the .grad property, since for many models this is very convenient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1.3333, 1.3333, 1.3333])\n"
     ]
    }
   ],
   "source": [
    "# calling .backward() on any variable will run backprop, starting from it.\n",
    "s.backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Understanding what is going on in the block below is crucial for being a\n",
    "successful programmer in deep learning.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False False\n",
      "None\n",
      "<AddBackward0 object at 0x102e7a7f0>\n",
      "True\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(2, 2)\n",
    "y = torch.randn(2, 2)\n",
    "# By default, user created Tensors have ``requires_grad=False``\n",
    "print(x.requires_grad, y.requires_grad)\n",
    "z = x + y\n",
    "# So you can't backprop through z\n",
    "print(z.grad_fn)\n",
    "# ``.requires_grad_( ... )`` changes an existing Tensor's ``requires_grad``\n",
    "x = x.requires_grad_()\n",
    "y = y.requires_grad_()\n",
    "# z contains enough information to compute gradients, as we saw above\n",
    "z = x + y\n",
    "print(z.grad_fn)\n",
    "# If any input to an operation has ``requires_grad=True``, so will the output\n",
    "print(z.requires_grad)\n",
    "# Now z has the computation history, which we can **detach**\n",
    "new_z = z.detach()\n",
    "# Which means that we have no gradient attached anymore\n",
    "print(new_z.grad_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also stop autograd from tracking history on Tensors\n",
    "with ``.requires_grad=True`` by wrapping the code block in\n",
    "``with torch.no_grad():``\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "print(x.requires_grad)\n",
    "print((x ** 2).requires_grad)\n",
    "with torch.no_grad():\n",
    "\tprint((x ** 2).requires_grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining networks \n",
    "\n",
    "Here, we briefly recall that in `PyTorch`, the `nn` package provides higher-level abstractions over raw computational graphs that are useful for building neural networks. The `nn` package defines a set of `Modules`, which are roughly equivalent to neural network layers. A `Module` receives input `Tensors` and computes output `Tensors`, but may also hold internal state such as `Tensors` containing learnable parameters. In the following example, we use the `nn` package to show how easy it is to instantiate a three-layer network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "# Define the input dimensions\n",
    "in_size = 1000\n",
    "# Number of neurons in a layer\n",
    "hidden_size = 100\n",
    "# Output (target) dimension\n",
    "output_size = 10\n",
    "# Use the nn package to define our model and loss function.\n",
    "model = torch.nn.Sequential(\n",
    "    nn.Linear(in_size, hidden_size),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(hidden_size, hidden_size),\n",
    "    nn.Tanh(),\n",
    "    nn.Linear(hidden_size, output_size),\n",
    "    nn.Softmax()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have seen in the slides, we can as easily mix between pre-defined modules and arithmetic operations. Here, we will define our very own *residual* layer, and then combine them in a more complex network. To do so, we first define our own `ResidualLayer` (by subclassing `nn.Module`). This type of layer can now be combined inside a full network by simply using `nn.Sequential`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ResidualLayer(nn.Module):\n",
    "    def __init__(self, dim, dim_res=32):\n",
    "        super().__init__()\n",
    "        self.block = nn.Sequential(\n",
    "            nn.Conv2d(dim, dim_res, 3, 1, 1),\n",
    "            nn.ReLU(True),\n",
    "            nn.Conv2d(dim_res, dim, 1),\n",
    "            nn.ReLU(True)\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        # Perform main residual action\n",
    "        return x + self.block(x)\n",
    "\n",
    "model = nn.Sequential(\n",
    "\tResidualLayer(64, 32),\n",
    "\tResidualLayer(64, 32),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Autoencoders and layer-wise training\n",
    "\n",
    "Here, we discuss the implementation of a deep autoencoder using PyTorch, focusing on layer-wise training. Autoencoders are unsupervised learning models that can learn useful features from the input data. They consist of an encoder, which maps the input data to a lower-dimensional representation, and a decoder, which reconstructs the input data from the lower-dimensional representation. _Deep autoencoders_ are multi-layer autoencoders that can learn more abstract and complex features from the input data. To learn such models, _layer-wise training_ was historically used as a technique to help overcome the vanishing gradient problem and initializes the network with better weights, leading to improved performance. Hence, we aim to _pretrain each layer individually_ in an **unsupervised** manner, followed by fine-tuning the entire network with _supervised learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"generative\"></a>\n",
    "## Types of learning\n",
    "\n",
    "### Supervised refresher\n",
    "\n",
    "Until now, we have mostly discussed models that are developed for _supervised_ and _discriminative_ tasks. To formalize this problem, we have a set of data $\\{\\mathbf{x}_{i}, \\mathbf{y}_{i}\\}_{i\\in[1,n]}$, where the $\\mathbf{x}_{i}$ and $\\mathbf{y}_{i}$ are linked. Therefore, we want to approximate this relation through\n",
    "\\begin{equation}\n",
    "    \\mathbf{\\hat{y}} = \\mathcal{F}_\\mathbf{\\theta}(\\mathbf{x})  \n",
    "\\end{equation}\n",
    "where we train the parameters $\\mathbf{\\theta}$ so that $\\mathbf{\\hat{y}}\\approx\\mathbf{y}$. The existence of a label $\\mathbf{y}$ (\"correct answer\") defines a _supervised_ problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Going unsupervised\n",
    "\n",
    "In some cases, we might only have a set of data $\\{\\mathbf{x}_{i}\\}_{i\\in[1,n]}$, and still be interested in learning some underlying properties or structure of this set. In that case, the problem is _unsupervised_ as we have to learn without a given answer. \n",
    "\n",
    "Here, we can turn to _generative_ models [[1](#reference1)], which allows to create new data instances based on the observation of existing examples. Although these models are more naturally defined in a _probabilistic way_ (as we will see later when we dwelve in generative models), we will assume that we have some simple _code_ $\\mathbf{z}$, which allows to control the properties of the generation, and need to learn\n",
    "\\begin{equation}\n",
    "    \\mathbf{\\hat{x}} = \\mathcal{F}_\\mathbf{\\theta}(\\mathbf{z})  \n",
    "\\end{equation}\n",
    "where we still need to learn $\\mathbf{\\theta}$, so that $\\mathbf{\\hat{x}}$ have similar properties to that of the examples in $\\{\\mathbf{x}_{i}\\}_{i\\in[1,n]}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Kg83oWxMm92f"
   },
   "source": [
    "<a id=\"ae\"></a>\n",
    "\n",
    "## Auto-encoders\n",
    "\n",
    "One way to understand a set of data is to try to _compress_, or _simplify_ the corresponding dataset. So the idea is to learn simultaneously how to _encode_ our unlabeled input $\\{\\mathbf{x}_{i}\\}_{i\\in[1,n]}$ and to _decode_ the corresponding representation. This idea give rise to the notion of **auto-encoder**. \n",
    "\n",
    "### Architecture \n",
    "\n",
    "The auto-encoder is an unsupervised architecture originally proposed to perform _dimensionality reduction_ [[3](#reference3)]. As its name indicates, we will try to train this model to learn an efficient _encoding_ $\\mathbf{z}$ of unlabeled input data $\\mathbf{x}$. The only way to learn efficient parameters is to also learn a _decoding_ function to _reconstruct_ $\\mathbf{x}$ from $\\mathbf{z}$.\n",
    "\n",
    "<img src=\"images/auto_encoder.png\"/>\n",
    "\n",
    "As shown here, a first model $\\mathcal{E}_\\phi$ _encodes_ the input into a _latent code_ $\\mathbf{z}$ in order to provide a low-dimensional representation of the data. A second model $\\mathcal{D}_\\theta$ designated as the _decoder_ aims to generate outputs from $\\mathbf{z}$ that are as close to the original inputs as possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Formal definition\n",
    "\n",
    "The latent code $\\mathbf{z}$ can be seen as a compressed abstract representation, and may be used as an intermediate space for analysis or generation. This helps to govern the distribution of the data through a simpler and higher-level representation, while enhancing the _expressiveness_ of the generative model.\n",
    "The behaviour of an auto-encoder can be formalized as:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mathbf{z} &= \\mathcal{E}_\\phi(\\mathbf{x}) \\\\\n",
    "\\mathbf{\\hat{x}} &= \\mathcal{D}_\\theta(\\mathbf{z})  \n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "with the _encoder_ $\\mathcal{E}_\\phi$ and _decoder_ $\\mathcal{D}_\\theta$ functions parameterized respectively by $\\phi$ and $\\theta$. As we can see this defines the reconstruction relationship\n",
    "$$\n",
    "    \\mathbf{\\hat{x}} = \\mathcal{D}_\\theta(\\mathcal{E}_\\phi(\\mathbf{x}))  \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "The training of an auto-encoder consists in finding the optimal functions of encoding $\\mathcal{E}^*$ and decoding $\\mathcal{D}^*$ by evaluating the \\textit{reconstruction error} $\\mathcal{L}$ between $\\mathbf{x}$ and $\\mathbf{\\hat{x}}$, such that\n",
    "\\begin{equation}\n",
    "    \\mathcal{E}^*, \\mathcal{D}^* = arg\\,min_{ \\phi, \\theta}{\\mathcal{L}}(\\mathbf{x}, \\mathcal{D}_\\theta(\\mathcal{E}_\\phi(\\mathbf{x})))\n",
    "\\end{equation}\n",
    "\n",
    "As the latent space usually has a smaller dimensionality than the input, it acts as an incentive for the network to find the main attributes of variations in the dataset (and also explains its use for _dimensionality reduction_)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variants and discussion\n",
    "\n",
    "There are several variants of auto-encoders, such as denoising auto-encoders or variational auto-encoders. Each address some downside of the basic AE model. For instance, the deterministic nature of the basic auto-encoder implies a point-wise mapping of the latent space, meaning that not all the latent positions can be leveraged to produce relevant reconstructions. Because of this reason, there is no way to ensure that the latent space could allow a robust generalization and that any random $\\mathbf{z}$ would generate a meaningful output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cAxMjUCBUa43"
   },
   "source": [
    "<a id=\"implement\"> </a>\n",
    "\n",
    "## Implementation\n",
    "\n",
    "Here, we discuss how we can implement and train a simple auto-encoder network in Pytorch. As discussed earlier, an AE is composed of two parts, an **encoder** and a **decoder**. The goal of the encoder is to \"compress\" the dataset, representing its principal features with a very small code, while the goal of the decoder is to learn how to reproduce the initial input from this code. Hence, we will first need to use some basic imports and definition to setup our problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e1_Y75QXJS6h"
   },
   "source": [
    "### Import Pytorch and other libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "YfIk2es3hJEd"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"bk-root\">\n",
       "        <a href=\"https://bokeh.org\" target=\"_blank\" class=\"bk-logo bk-logo-small bk-logo-notebook\"></a>\n",
       "        <span id=\"1002\">Loading BokehJS ...</span>\n",
       "    </div>\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "(function(root) {\n",
       "  function now() {\n",
       "    return new Date();\n",
       "  }\n",
       "\n",
       "  const force = true;\n",
       "\n",
       "  if (typeof root._bokeh_onload_callbacks === \"undefined\" || force === true) {\n",
       "    root._bokeh_onload_callbacks = [];\n",
       "    root._bokeh_is_loading = undefined;\n",
       "  }\n",
       "\n",
       "const JS_MIME_TYPE = 'application/javascript';\n",
       "  const HTML_MIME_TYPE = 'text/html';\n",
       "  const EXEC_MIME_TYPE = 'application/vnd.bokehjs_exec.v0+json';\n",
       "  const CLASS_NAME = 'output_bokeh rendered_html';\n",
       "\n",
       "  /**\n",
       "   * Render data to the DOM node\n",
       "   */\n",
       "  function render(props, node) {\n",
       "    const script = document.createElement(\"script\");\n",
       "    node.appendChild(script);\n",
       "  }\n",
       "\n",
       "  /**\n",
       "   * Handle when an output is cleared or removed\n",
       "   */\n",
       "  function handleClearOutput(event, handle) {\n",
       "    const cell = handle.cell;\n",
       "\n",
       "    const id = cell.output_area._bokeh_element_id;\n",
       "    const server_id = cell.output_area._bokeh_server_id;\n",
       "    // Clean up Bokeh references\n",
       "    if (id != null && id in Bokeh.index) {\n",
       "      Bokeh.index[id].model.document.clear();\n",
       "      delete Bokeh.index[id];\n",
       "    }\n",
       "\n",
       "    if (server_id !== undefined) {\n",
       "      // Clean up Bokeh references\n",
       "      const cmd_clean = \"from bokeh.io.state import curstate; print(curstate().uuid_to_server['\" + server_id + \"'].get_sessions()[0].document.roots[0]._id)\";\n",
       "      cell.notebook.kernel.execute(cmd_clean, {\n",
       "        iopub: {\n",
       "          output: function(msg) {\n",
       "            const id = msg.content.text.trim();\n",
       "            if (id in Bokeh.index) {\n",
       "              Bokeh.index[id].model.document.clear();\n",
       "              delete Bokeh.index[id];\n",
       "            }\n",
       "          }\n",
       "        }\n",
       "      });\n",
       "      // Destroy server and session\n",
       "      const cmd_destroy = \"import bokeh.io.notebook as ion; ion.destroy_server('\" + server_id + \"')\";\n",
       "      cell.notebook.kernel.execute(cmd_destroy);\n",
       "    }\n",
       "  }\n",
       "\n",
       "  /**\n",
       "   * Handle when a new output is added\n",
       "   */\n",
       "  function handleAddOutput(event, handle) {\n",
       "    const output_area = handle.output_area;\n",
       "    const output = handle.output;\n",
       "\n",
       "    // limit handleAddOutput to display_data with EXEC_MIME_TYPE content only\n",
       "    if ((output.output_type != \"display_data\") || (!Object.prototype.hasOwnProperty.call(output.data, EXEC_MIME_TYPE))) {\n",
       "      return\n",
       "    }\n",
       "\n",
       "    const toinsert = output_area.element.find(\".\" + CLASS_NAME.split(' ')[0]);\n",
       "\n",
       "    if (output.metadata[EXEC_MIME_TYPE][\"id\"] !== undefined) {\n",
       "      toinsert[toinsert.length - 1].firstChild.textContent = output.data[JS_MIME_TYPE];\n",
       "      // store reference to embed id on output_area\n",
       "      output_area._bokeh_element_id = output.metadata[EXEC_MIME_TYPE][\"id\"];\n",
       "    }\n",
       "    if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !== undefined) {\n",
       "      const bk_div = document.createElement(\"div\");\n",
       "      bk_div.innerHTML = output.data[HTML_MIME_TYPE];\n",
       "      const script_attrs = bk_div.children[0].attributes;\n",
       "      for (let i = 0; i < script_attrs.length; i++) {\n",
       "        toinsert[toinsert.length - 1].firstChild.setAttribute(script_attrs[i].name, script_attrs[i].value);\n",
       "        toinsert[toinsert.length - 1].firstChild.textContent = bk_div.children[0].textContent\n",
       "      }\n",
       "      // store reference to server id on output_area\n",
       "      output_area._bokeh_server_id = output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n",
       "    }\n",
       "  }\n",
       "\n",
       "  function register_renderer(events, OutputArea) {\n",
       "\n",
       "    function append_mime(data, metadata, element) {\n",
       "      // create a DOM node to render to\n",
       "      const toinsert = this.create_output_subarea(\n",
       "        metadata,\n",
       "        CLASS_NAME,\n",
       "        EXEC_MIME_TYPE\n",
       "      );\n",
       "      this.keyboard_manager.register_events(toinsert);\n",
       "      // Render to node\n",
       "      const props = {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n",
       "      render(props, toinsert[toinsert.length - 1]);\n",
       "      element.append(toinsert);\n",
       "      return toinsert\n",
       "    }\n",
       "\n",
       "    /* Handle when an output is cleared or removed */\n",
       "    events.on('clear_output.CodeCell', handleClearOutput);\n",
       "    events.on('delete.Cell', handleClearOutput);\n",
       "\n",
       "    /* Handle when a new output is added */\n",
       "    events.on('output_added.OutputArea', handleAddOutput);\n",
       "\n",
       "    /**\n",
       "     * Register the mime type and append_mime function with output_area\n",
       "     */\n",
       "    OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n",
       "      /* Is output safe? */\n",
       "      safe: true,\n",
       "      /* Index of renderer in `output_area.display_order` */\n",
       "      index: 0\n",
       "    });\n",
       "  }\n",
       "\n",
       "  // register the mime type if in Jupyter Notebook environment and previously unregistered\n",
       "  if (root.Jupyter !== undefined) {\n",
       "    const events = require('base/js/events');\n",
       "    const OutputArea = require('notebook/js/outputarea').OutputArea;\n",
       "\n",
       "    if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) == -1) {\n",
       "      register_renderer(events, OutputArea);\n",
       "    }\n",
       "  }\n",
       "  if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n",
       "    root._bokeh_timeout = Date.now() + 5000;\n",
       "    root._bokeh_failed_load = false;\n",
       "  }\n",
       "\n",
       "  const NB_LOAD_WARNING = {'data': {'text/html':\n",
       "     \"<div style='background-color: #fdd'>\\n\"+\n",
       "     \"<p>\\n\"+\n",
       "     \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n",
       "     \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n",
       "     \"</p>\\n\"+\n",
       "     \"<ul>\\n\"+\n",
       "     \"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\\n\"+\n",
       "     \"<li>use INLINE resources instead, as so:</li>\\n\"+\n",
       "     \"</ul>\\n\"+\n",
       "     \"<code>\\n\"+\n",
       "     \"from bokeh.resources import INLINE\\n\"+\n",
       "     \"output_notebook(resources=INLINE)\\n\"+\n",
       "     \"</code>\\n\"+\n",
       "     \"</div>\"}};\n",
       "\n",
       "  function display_loaded() {\n",
       "    const el = document.getElementById(\"1002\");\n",
       "    if (el != null) {\n",
       "      el.textContent = \"BokehJS is loading...\";\n",
       "    }\n",
       "    if (root.Bokeh !== undefined) {\n",
       "      if (el != null) {\n",
       "        el.textContent = \"BokehJS \" + root.Bokeh.version + \" successfully loaded.\";\n",
       "      }\n",
       "    } else if (Date.now() < root._bokeh_timeout) {\n",
       "      setTimeout(display_loaded, 100)\n",
       "    }\n",
       "  }\n",
       "\n",
       "  function run_callbacks() {\n",
       "    try {\n",
       "      root._bokeh_onload_callbacks.forEach(function(callback) {\n",
       "        if (callback != null)\n",
       "          callback();\n",
       "      });\n",
       "    } finally {\n",
       "      delete root._bokeh_onload_callbacks\n",
       "    }\n",
       "    console.debug(\"Bokeh: all callbacks have finished\");\n",
       "  }\n",
       "\n",
       "  function load_libs(css_urls, js_urls, callback) {\n",
       "    if (css_urls == null) css_urls = [];\n",
       "    if (js_urls == null) js_urls = [];\n",
       "\n",
       "    root._bokeh_onload_callbacks.push(callback);\n",
       "    if (root._bokeh_is_loading > 0) {\n",
       "      console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n",
       "      return null;\n",
       "    }\n",
       "    if (js_urls == null || js_urls.length === 0) {\n",
       "      run_callbacks();\n",
       "      return null;\n",
       "    }\n",
       "    console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n",
       "    root._bokeh_is_loading = css_urls.length + js_urls.length;\n",
       "\n",
       "    function on_load() {\n",
       "      root._bokeh_is_loading--;\n",
       "      if (root._bokeh_is_loading === 0) {\n",
       "        console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n",
       "        run_callbacks()\n",
       "      }\n",
       "    }\n",
       "\n",
       "    function on_error(url) {\n",
       "      console.error(\"failed to load \" + url);\n",
       "    }\n",
       "\n",
       "    for (let i = 0; i < css_urls.length; i++) {\n",
       "      const url = css_urls[i];\n",
       "      const element = document.createElement(\"link\");\n",
       "      element.onload = on_load;\n",
       "      element.onerror = on_error.bind(null, url);\n",
       "      element.rel = \"stylesheet\";\n",
       "      element.type = \"text/css\";\n",
       "      element.href = url;\n",
       "      console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n",
       "      document.body.appendChild(element);\n",
       "    }\n",
       "\n",
       "    for (let i = 0; i < js_urls.length; i++) {\n",
       "      const url = js_urls[i];\n",
       "      const element = document.createElement('script');\n",
       "      element.onload = on_load;\n",
       "      element.onerror = on_error.bind(null, url);\n",
       "      element.async = false;\n",
       "      element.src = url;\n",
       "      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
       "      document.head.appendChild(element);\n",
       "    }\n",
       "  };\n",
       "\n",
       "  function inject_raw_css(css) {\n",
       "    const element = document.createElement(\"style\");\n",
       "    element.appendChild(document.createTextNode(css));\n",
       "    document.body.appendChild(element);\n",
       "  }\n",
       "\n",
       "  const js_urls = [\"https://cdn.bokeh.org/bokeh/release/bokeh-2.4.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-gl-2.4.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.4.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.4.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-mathjax-2.4.3.min.js\", \"https://unpkg.com/@holoviz/panel@0.14.3/dist/panel.min.js\"];\n",
       "  const css_urls = [];\n",
       "\n",
       "  const inline_js = [    function(Bokeh) {\n",
       "      Bokeh.set_log_level(\"info\");\n",
       "    },\n",
       "function(Bokeh) {\n",
       "    }\n",
       "  ];\n",
       "\n",
       "  function run_inline_js() {\n",
       "    if (root.Bokeh !== undefined || force === true) {\n",
       "          for (let i = 0; i < inline_js.length; i++) {\n",
       "      inline_js[i].call(root, root.Bokeh);\n",
       "    }\n",
       "if (force === true) {\n",
       "        display_loaded();\n",
       "      }} else if (Date.now() < root._bokeh_timeout) {\n",
       "      setTimeout(run_inline_js, 100);\n",
       "    } else if (!root._bokeh_failed_load) {\n",
       "      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n",
       "      root._bokeh_failed_load = true;\n",
       "    } else if (force !== true) {\n",
       "      const cell = $(document.getElementById(\"1002\")).parents('.cell').data().cell;\n",
       "      cell.output_area.append_execute_result(NB_LOAD_WARNING)\n",
       "    }\n",
       "  }\n",
       "\n",
       "  if (root._bokeh_is_loading === 0) {\n",
       "    console.debug(\"Bokeh: BokehJS loaded, going straight to plotting\");\n",
       "    run_inline_js();\n",
       "  } else {\n",
       "    load_libs(css_urls, js_urls, function() {\n",
       "      console.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n",
       "      run_inline_js();\n",
       "    });\n",
       "  }\n",
       "}(window));"
      ],
      "application/vnd.bokehjs_load.v0+json": "(function(root) {\n  function now() {\n    return new Date();\n  }\n\n  const force = true;\n\n  if (typeof root._bokeh_onload_callbacks === \"undefined\" || force === true) {\n    root._bokeh_onload_callbacks = [];\n    root._bokeh_is_loading = undefined;\n  }\n\n\n  if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n    root._bokeh_timeout = Date.now() + 5000;\n    root._bokeh_failed_load = false;\n  }\n\n  const NB_LOAD_WARNING = {'data': {'text/html':\n     \"<div style='background-color: #fdd'>\\n\"+\n     \"<p>\\n\"+\n     \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n     \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n     \"</p>\\n\"+\n     \"<ul>\\n\"+\n     \"<li>re-rerun `output_notebook()` to attempt to load from CDN again, or</li>\\n\"+\n     \"<li>use INLINE resources instead, as so:</li>\\n\"+\n     \"</ul>\\n\"+\n     \"<code>\\n\"+\n     \"from bokeh.resources import INLINE\\n\"+\n     \"output_notebook(resources=INLINE)\\n\"+\n     \"</code>\\n\"+\n     \"</div>\"}};\n\n  function display_loaded() {\n    const el = document.getElementById(\"1002\");\n    if (el != null) {\n      el.textContent = \"BokehJS is loading...\";\n    }\n    if (root.Bokeh !== undefined) {\n      if (el != null) {\n        el.textContent = \"BokehJS \" + root.Bokeh.version + \" successfully loaded.\";\n      }\n    } else if (Date.now() < root._bokeh_timeout) {\n      setTimeout(display_loaded, 100)\n    }\n  }\n\n  function run_callbacks() {\n    try {\n      root._bokeh_onload_callbacks.forEach(function(callback) {\n        if (callback != null)\n          callback();\n      });\n    } finally {\n      delete root._bokeh_onload_callbacks\n    }\n    console.debug(\"Bokeh: all callbacks have finished\");\n  }\n\n  function load_libs(css_urls, js_urls, callback) {\n    if (css_urls == null) css_urls = [];\n    if (js_urls == null) js_urls = [];\n\n    root._bokeh_onload_callbacks.push(callback);\n    if (root._bokeh_is_loading > 0) {\n      console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n      return null;\n    }\n    if (js_urls == null || js_urls.length === 0) {\n      run_callbacks();\n      return null;\n    }\n    console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n    root._bokeh_is_loading = css_urls.length + js_urls.length;\n\n    function on_load() {\n      root._bokeh_is_loading--;\n      if (root._bokeh_is_loading === 0) {\n        console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n        run_callbacks()\n      }\n    }\n\n    function on_error(url) {\n      console.error(\"failed to load \" + url);\n    }\n\n    for (let i = 0; i < css_urls.length; i++) {\n      const url = css_urls[i];\n      const element = document.createElement(\"link\");\n      element.onload = on_load;\n      element.onerror = on_error.bind(null, url);\n      element.rel = \"stylesheet\";\n      element.type = \"text/css\";\n      element.href = url;\n      console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n      document.body.appendChild(element);\n    }\n\n    for (let i = 0; i < js_urls.length; i++) {\n      const url = js_urls[i];\n      const element = document.createElement('script');\n      element.onload = on_load;\n      element.onerror = on_error.bind(null, url);\n      element.async = false;\n      element.src = url;\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      document.head.appendChild(element);\n    }\n  };\n\n  function inject_raw_css(css) {\n    const element = document.createElement(\"style\");\n    element.appendChild(document.createTextNode(css));\n    document.body.appendChild(element);\n  }\n\n  const js_urls = [\"https://cdn.bokeh.org/bokeh/release/bokeh-2.4.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-gl-2.4.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.4.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.4.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-mathjax-2.4.3.min.js\", \"https://unpkg.com/@holoviz/panel@0.14.3/dist/panel.min.js\"];\n  const css_urls = [];\n\n  const inline_js = [    function(Bokeh) {\n      Bokeh.set_log_level(\"info\");\n    },\nfunction(Bokeh) {\n    }\n  ];\n\n  function run_inline_js() {\n    if (root.Bokeh !== undefined || force === true) {\n          for (let i = 0; i < inline_js.length; i++) {\n      inline_js[i].call(root, root.Bokeh);\n    }\nif (force === true) {\n        display_loaded();\n      }} else if (Date.now() < root._bokeh_timeout) {\n      setTimeout(run_inline_js, 100);\n    } else if (!root._bokeh_failed_load) {\n      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n      root._bokeh_failed_load = true;\n    } else if (force !== true) {\n      const cell = $(document.getElementById(\"1002\")).parents('.cell').data().cell;\n      cell.output_area.append_execute_result(NB_LOAD_WARNING)\n    }\n  }\n\n  if (root._bokeh_is_loading === 0) {\n    console.debug(\"Bokeh: BokehJS loaded, going straight to plotting\");\n    run_inline_js();\n  } else {\n    load_libs(css_urls, js_urls, function() {\n      console.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n      run_inline_js();\n    });\n  }\n}(window));"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "(function(root) {\n",
       "  function now() {\n",
       "    return new Date();\n",
       "  }\n",
       "\n",
       "  var force = true;\n",
       "\n",
       "  if (typeof root._bokeh_onload_callbacks === \"undefined\" || force === true) {\n",
       "    root._bokeh_onload_callbacks = [];\n",
       "    root._bokeh_is_loading = undefined;\n",
       "  }\n",
       "\n",
       "  if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n",
       "    root._bokeh_timeout = Date.now() + 5000;\n",
       "    root._bokeh_failed_load = false;\n",
       "  }\n",
       "\n",
       "  function run_callbacks() {\n",
       "    try {\n",
       "      root._bokeh_onload_callbacks.forEach(function(callback) {\n",
       "        if (callback != null)\n",
       "          callback();\n",
       "      });\n",
       "    } finally {\n",
       "      delete root._bokeh_onload_callbacks\n",
       "    }\n",
       "    console.debug(\"Bokeh: all callbacks have finished\");\n",
       "  }\n",
       "\n",
       "  function load_libs(css_urls, js_urls, js_modules, callback) {\n",
       "    if (css_urls == null) css_urls = [];\n",
       "    if (js_urls == null) js_urls = [];\n",
       "    if (js_modules == null) js_modules = [];\n",
       "\n",
       "    root._bokeh_onload_callbacks.push(callback);\n",
       "    if (root._bokeh_is_loading > 0) {\n",
       "      console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n",
       "      return null;\n",
       "    }\n",
       "    if (js_urls.length === 0 && js_modules.length === 0) {\n",
       "      run_callbacks();\n",
       "      return null;\n",
       "    }\n",
       "    console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n",
       "\n",
       "    function on_load() {\n",
       "      root._bokeh_is_loading--;\n",
       "      if (root._bokeh_is_loading === 0) {\n",
       "        console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n",
       "        run_callbacks()\n",
       "      }\n",
       "    }\n",
       "\n",
       "    function on_error() {\n",
       "      console.error(\"failed to load \" + url);\n",
       "    }\n",
       "\n",
       "    for (var i = 0; i < css_urls.length; i++) {\n",
       "      var url = css_urls[i];\n",
       "      const element = document.createElement(\"link\");\n",
       "      element.onload = on_load;\n",
       "      element.onerror = on_error;\n",
       "      element.rel = \"stylesheet\";\n",
       "      element.type = \"text/css\";\n",
       "      element.href = url;\n",
       "      console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n",
       "      document.body.appendChild(element);\n",
       "    }\n",
       "\n",
       "    var skip = [];\n",
       "    if (window.requirejs) {\n",
       "      window.requirejs.config({'packages': {}, 'paths': {'ace': '//cdnjs.cloudflare.com/ajax/libs/ace/1.4.7', 'gridstack': 'https://cdn.jsdelivr.net/npm/gridstack@4.2.5/dist/gridstack-h5', 'notyf': 'https://cdn.jsdelivr.net/npm/notyf@3/notyf.min'}, 'shim': {'ace/ext-language_tools': {'deps': ['ace/ace']}, 'ace/ext-modelist': {'deps': ['ace/ace']}, 'gridstack': {'exports': 'GridStack'}}});\n",
       "      require([\"ace/ace\"], function(ace) {\n",
       "\twindow.ace = ace\n",
       "\ton_load()\n",
       "      })\n",
       "      require([\"ace/ext-language_tools\"], function() {\n",
       "\ton_load()\n",
       "      })\n",
       "      require([\"ace/ext-modelist\"], function() {\n",
       "\ton_load()\n",
       "      })\n",
       "      require([\"gridstack\"], function(GridStack) {\n",
       "\twindow.GridStack = GridStack\n",
       "\ton_load()\n",
       "      })\n",
       "      require([\"notyf\"], function() {\n",
       "\ton_load()\n",
       "      })\n",
       "      root._bokeh_is_loading = css_urls.length + 5;\n",
       "    } else {\n",
       "      root._bokeh_is_loading = css_urls.length + js_urls.length + js_modules.length;\n",
       "    }    if (((window['ace'] !== undefined) && (!(window['ace'] instanceof HTMLElement))) || window.requirejs) {\n",
       "      var urls = ['https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ace.js', 'https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-language_tools.js', 'https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-modelist.js'];\n",
       "      for (var i = 0; i < urls.length; i++) {\n",
       "        skip.push(urls[i])\n",
       "      }\n",
       "    }    if (((window['GridStack'] !== undefined) && (!(window['GridStack'] instanceof HTMLElement))) || window.requirejs) {\n",
       "      var urls = ['https://cdn.holoviz.org/panel/0.14.3/dist/bundled/gridstack/gridstack@4.2.5/dist/gridstack-h5.js'];\n",
       "      for (var i = 0; i < urls.length; i++) {\n",
       "        skip.push(urls[i])\n",
       "      }\n",
       "    }    if (((window['Notyf'] !== undefined) && (!(window['Notyf'] instanceof HTMLElement))) || window.requirejs) {\n",
       "      var urls = ['https://cdn.holoviz.org/panel/0.14.3/dist/bundled/notificationarea/notyf@3/notyf.min.js'];\n",
       "      for (var i = 0; i < urls.length; i++) {\n",
       "        skip.push(urls[i])\n",
       "      }\n",
       "    }    for (var i = 0; i < js_urls.length; i++) {\n",
       "      var url = js_urls[i];\n",
       "      if (skip.indexOf(url) >= 0) {\n",
       "\tif (!window.requirejs) {\n",
       "\t  on_load();\n",
       "\t}\n",
       "\tcontinue;\n",
       "      }\n",
       "      var element = document.createElement('script');\n",
       "      element.onload = on_load;\n",
       "      element.onerror = on_error;\n",
       "      element.async = false;\n",
       "      element.src = url;\n",
       "      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
       "      document.head.appendChild(element);\n",
       "    }\n",
       "    for (var i = 0; i < js_modules.length; i++) {\n",
       "      var url = js_modules[i];\n",
       "      if (skip.indexOf(url) >= 0) {\n",
       "\tif (!window.requirejs) {\n",
       "\t  on_load();\n",
       "\t}\n",
       "\tcontinue;\n",
       "      }\n",
       "      var element = document.createElement('script');\n",
       "      element.onload = on_load;\n",
       "      element.onerror = on_error;\n",
       "      element.async = false;\n",
       "      element.src = url;\n",
       "      element.type = \"module\";\n",
       "      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
       "      document.head.appendChild(element);\n",
       "    }\n",
       "    if (!js_urls.length && !js_modules.length) {\n",
       "      on_load()\n",
       "    }\n",
       "  };\n",
       "\n",
       "  function inject_raw_css(css) {\n",
       "    const element = document.createElement(\"style\");\n",
       "    element.appendChild(document.createTextNode(css));\n",
       "    document.body.appendChild(element);\n",
       "  }\n",
       "\n",
       "  var js_urls = [\"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ace.js\", \"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-language_tools.js\", \"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-modelist.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-2.4.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-gl-2.4.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.4.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.4.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-mathjax-2.4.3.min.js\", \"https://unpkg.com/@holoviz/panel@0.14.3/dist/panel.min.js\"];\n",
       "  var js_modules = [];\n",
       "  var css_urls = [\"https://cdn.holoviz.org/panel/0.14.3/dist/css/debugger.css\", \"https://cdn.holoviz.org/panel/0.14.3/dist/css/alerts.css\", \"https://cdn.holoviz.org/panel/0.14.3/dist/css/card.css\", \"https://cdn.holoviz.org/panel/0.14.3/dist/css/widgets.css\", \"https://cdn.holoviz.org/panel/0.14.3/dist/css/markdown.css\", \"https://cdn.holoviz.org/panel/0.14.3/dist/css/json.css\", \"https://cdn.holoviz.org/panel/0.14.3/dist/css/loading.css\", \"https://cdn.holoviz.org/panel/0.14.3/dist/css/dataframe.css\"];\n",
       "  var inline_js = [    function(Bokeh) {\n",
       "      inject_raw_css(\"/*\\n ~ CML // Creative Machine Learning ~\\n mml.css : CSS styling information for Panel and Bokeh\\n \\n This file defines the main CSS styling information for the CML course\\n \\n Author               :  Philippe Esling\\n                        <esling@ircam.fr>\\n*/\\n\\nbody {\\n  display: flex;\\n  height: 100vh;\\n  margin: 0px;\\n  overflow-x: hidden;\\n  overflow-y: hidden;\\n}\\n\\n.bk-root .bk, .bk-root .bk:before, .bk-root .bk:after {\\n  font-family: \\\"Josefin Sans\\\";\\n}\\n\\nimg {\\n  max-width: 100%;\\n}\\n\\n#container {\\n  padding:0px;\\n  height:100vh;\\n  width: 100vw;\\n  max-width: 100vw;\\n}\\n\\n#sidebar .mdc-list {\\n  padding-left: 5px;\\n  padding-right: 5px;\\n}\\n\\n.mdc-drawer-app-content {\\n  flex: auto;\\n  position: relative;\\n  overflow: hidden;\\n}\\n\\n.mdc-drawer {\\n  background: #FAFAFA; /* GRAY 50 */\\n}\\n\\n.mdc-drawer-app-content {\\n  margin-left: 0 !important;\\n}\\n\\n.title-bar {\\n  display: contents;\\n  justify-content: center;\\n  align-content: center;\\n  width: 100%;\\n}\\n\\n.mdc-top-app-bar .bk-menu {\\n  color: black\\n}\\n\\n.app-header {\\n  display: contents;\\n  padding-left: 10px;\\n  font-size: 1.25em;\\n}\\n\\nimg.app-logo {\\n  padding-right: 10px;\\n  font-size: 28px;\\n  height: 30px;\\n  max-width: inherit;\\n  padding-top: 12px;\\n  padding-bottom: 6px;\\n}\\n\\n#app-title {\\n  padding-right: 12px;\\n  padding-left: 12px;\\n}\\n\\n.title {\\n  font-family: \\\"Josefin Sans\\\";\\n  color: #fff;\\n  text-decoration: none;\\n  text-decoration-line: none;\\n  text-decoration-style: initial;\\n  text-decoration-color: initial;\\n  font-weight: 400;\\n  font-size: 2em;\\n  line-height: 2em;\\n  white-space: nowrap;\\n}\\n\\n.main-content {\\n  overflow-y: scroll;\\n  overflow-x: auto;\\n}\\n\\n#header {\\n  position: absolute;\\n  z-index: 7;\\n}\\n\\n#header-items {\\n  width: 100%;\\n  margin-left:15px;\\n}\\n\\n.pn-busy-container {\\n  align-items: center;\\n  justify-content: center;\\n  display: flex;\\n}\\n\\n.mdc-drawer__content {\\n  overflow-x: hidden;\\n}\\n.mdc-drawer__content, .main-content {\\n  padding: 12px;\\n}\\n\\n.main-content {\\n  height: calc(100vh - 88px);\\n  max-height: calc(100vh - 88px);\\n  padding-right: 32px;\\n}\\n\\nbutton.mdc-button.mdc-card-button {\\n  color: transparent;\\n  height: 50px;\\n}\\n\\np.bk.mdc-button {\\n  display: none;\\n}\\n\\ndiv.bk.mdc-card {\\n  border-radius: 0px\\n}\\n\\n.mdc-card .bk.card-header {\\n  display: flex;\\n}\\n\\n.bk.mdc-card-title {\\n  font-family: \\\"Josefin Sans\\\";\\n  font-weight: bold;\\n  align-items: center;\\n  display: flex !important;\\n  position: relative !important;\\n}\\n\\n.bk.mdc-card-title:nth-child(2) {\\n  margin-left: -1.4em;\\n}\\n\\n.pn-modal {\\n  overflow-y: scroll;\\n  width: 100%;\\n  display: none;\\n  position: absolute;\\n  top: 0;\\n  left: 0;\\n}\\n\\n.pn-modal-content {\\n  font-family: \\\"Josefin Sans\\\";\\n  background-color: #0e0e0e;\\n  margin: auto;\\n  margin-top: 25px;\\n  margin-bottom: 25px;\\n  padding: 15px 20px 20px 20px;\\n  border: 1px solid #888;\\n  width: 80% !important;\\n}\\n\\n.pn-modal-close {\\n  position: absolute;\\n  right: 25px;\\n  z-index: 100;\\n}\\n\\n.pn-modal-close:hover,\\n.pn-modal-close:focus {\\n  color: #000;\\n  text-decoration: none;\\n  cursor: pointer;\\n}\\n\\n.custom_button_bokeh button.bk.bk-btn.bk-btn-default {\\n    font-size:48pt;\\n    background-color: #05b7ff;\\n    border-color: #05b7ff;\\n}\");\n",
       "    },    function(Bokeh) {\n",
       "      inject_raw_css(\"\\n    .bk.pn-loading.arc:before {\\n      background-image: url(\\\"data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHN0eWxlPSJtYXJnaW46IGF1dG87IGJhY2tncm91bmQ6IG5vbmU7IGRpc3BsYXk6IGJsb2NrOyBzaGFwZS1yZW5kZXJpbmc6IGF1dG87IiB2aWV3Qm94PSIwIDAgMTAwIDEwMCIgcHJlc2VydmVBc3BlY3RSYXRpbz0ieE1pZFlNaWQiPiAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjYzNjM2MzIiBzdHJva2Utd2lkdGg9IjEwIiByPSIzNSIgc3Ryb2tlLWRhc2hhcnJheT0iMTY0LjkzMzYxNDMxMzQ2NDE1IDU2Ljk3Nzg3MTQzNzgyMTM4Ij4gICAgPGFuaW1hdGVUcmFuc2Zvcm0gYXR0cmlidXRlTmFtZT0idHJhbnNmb3JtIiB0eXBlPSJyb3RhdGUiIHJlcGVhdENvdW50PSJpbmRlZmluaXRlIiBkdXI9IjFzIiB2YWx1ZXM9IjAgNTAgNTA7MzYwIDUwIDUwIiBrZXlUaW1lcz0iMDsxIj48L2FuaW1hdGVUcmFuc2Zvcm0+ICA8L2NpcmNsZT48L3N2Zz4=\\\");\\n      background-size: auto calc(min(50%, 400px));\\n    }\\n    \");\n",
       "    },    function(Bokeh) {\n",
       "      Bokeh.set_log_level(\"info\");\n",
       "    },\n",
       "function(Bokeh) {} // ensure no trailing comma for IE\n",
       "  ];\n",
       "\n",
       "  function run_inline_js() {\n",
       "    if ((root.Bokeh !== undefined) || (force === true)) {\n",
       "      for (var i = 0; i < inline_js.length; i++) {\n",
       "        inline_js[i].call(root, root.Bokeh);\n",
       "      }} else if (Date.now() < root._bokeh_timeout) {\n",
       "      setTimeout(run_inline_js, 100);\n",
       "    } else if (!root._bokeh_failed_load) {\n",
       "      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n",
       "      root._bokeh_failed_load = true;\n",
       "    }\n",
       "  }\n",
       "\n",
       "  if (root._bokeh_is_loading === 0) {\n",
       "    console.debug(\"Bokeh: BokehJS loaded, going straight to plotting\");\n",
       "    run_inline_js();\n",
       "  } else {\n",
       "    load_libs(css_urls, js_urls, js_modules, function() {\n",
       "      console.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n",
       "      run_inline_js();\n",
       "    });\n",
       "  }\n",
       "}(window));"
      ],
      "application/vnd.holoviews_load.v0+json": "(function(root) {\n  function now() {\n    return new Date();\n  }\n\n  var force = true;\n\n  if (typeof root._bokeh_onload_callbacks === \"undefined\" || force === true) {\n    root._bokeh_onload_callbacks = [];\n    root._bokeh_is_loading = undefined;\n  }\n\n  if (typeof (root._bokeh_timeout) === \"undefined\" || force === true) {\n    root._bokeh_timeout = Date.now() + 5000;\n    root._bokeh_failed_load = false;\n  }\n\n  function run_callbacks() {\n    try {\n      root._bokeh_onload_callbacks.forEach(function(callback) {\n        if (callback != null)\n          callback();\n      });\n    } finally {\n      delete root._bokeh_onload_callbacks\n    }\n    console.debug(\"Bokeh: all callbacks have finished\");\n  }\n\n  function load_libs(css_urls, js_urls, js_modules, callback) {\n    if (css_urls == null) css_urls = [];\n    if (js_urls == null) js_urls = [];\n    if (js_modules == null) js_modules = [];\n\n    root._bokeh_onload_callbacks.push(callback);\n    if (root._bokeh_is_loading > 0) {\n      console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n      return null;\n    }\n    if (js_urls.length === 0 && js_modules.length === 0) {\n      run_callbacks();\n      return null;\n    }\n    console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n\n    function on_load() {\n      root._bokeh_is_loading--;\n      if (root._bokeh_is_loading === 0) {\n        console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n        run_callbacks()\n      }\n    }\n\n    function on_error() {\n      console.error(\"failed to load \" + url);\n    }\n\n    for (var i = 0; i < css_urls.length; i++) {\n      var url = css_urls[i];\n      const element = document.createElement(\"link\");\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.rel = \"stylesheet\";\n      element.type = \"text/css\";\n      element.href = url;\n      console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n      document.body.appendChild(element);\n    }\n\n    var skip = [];\n    if (window.requirejs) {\n      window.requirejs.config({'packages': {}, 'paths': {'ace': '//cdnjs.cloudflare.com/ajax/libs/ace/1.4.7', 'gridstack': 'https://cdn.jsdelivr.net/npm/gridstack@4.2.5/dist/gridstack-h5', 'notyf': 'https://cdn.jsdelivr.net/npm/notyf@3/notyf.min'}, 'shim': {'ace/ext-language_tools': {'deps': ['ace/ace']}, 'ace/ext-modelist': {'deps': ['ace/ace']}, 'gridstack': {'exports': 'GridStack'}}});\n      require([\"ace/ace\"], function(ace) {\n\twindow.ace = ace\n\ton_load()\n      })\n      require([\"ace/ext-language_tools\"], function() {\n\ton_load()\n      })\n      require([\"ace/ext-modelist\"], function() {\n\ton_load()\n      })\n      require([\"gridstack\"], function(GridStack) {\n\twindow.GridStack = GridStack\n\ton_load()\n      })\n      require([\"notyf\"], function() {\n\ton_load()\n      })\n      root._bokeh_is_loading = css_urls.length + 5;\n    } else {\n      root._bokeh_is_loading = css_urls.length + js_urls.length + js_modules.length;\n    }    if (((window['ace'] !== undefined) && (!(window['ace'] instanceof HTMLElement))) || window.requirejs) {\n      var urls = ['https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ace.js', 'https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-language_tools.js', 'https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-modelist.js'];\n      for (var i = 0; i < urls.length; i++) {\n        skip.push(urls[i])\n      }\n    }    if (((window['GridStack'] !== undefined) && (!(window['GridStack'] instanceof HTMLElement))) || window.requirejs) {\n      var urls = ['https://cdn.holoviz.org/panel/0.14.3/dist/bundled/gridstack/gridstack@4.2.5/dist/gridstack-h5.js'];\n      for (var i = 0; i < urls.length; i++) {\n        skip.push(urls[i])\n      }\n    }    if (((window['Notyf'] !== undefined) && (!(window['Notyf'] instanceof HTMLElement))) || window.requirejs) {\n      var urls = ['https://cdn.holoviz.org/panel/0.14.3/dist/bundled/notificationarea/notyf@3/notyf.min.js'];\n      for (var i = 0; i < urls.length; i++) {\n        skip.push(urls[i])\n      }\n    }    for (var i = 0; i < js_urls.length; i++) {\n      var url = js_urls[i];\n      if (skip.indexOf(url) >= 0) {\n\tif (!window.requirejs) {\n\t  on_load();\n\t}\n\tcontinue;\n      }\n      var element = document.createElement('script');\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.async = false;\n      element.src = url;\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      document.head.appendChild(element);\n    }\n    for (var i = 0; i < js_modules.length; i++) {\n      var url = js_modules[i];\n      if (skip.indexOf(url) >= 0) {\n\tif (!window.requirejs) {\n\t  on_load();\n\t}\n\tcontinue;\n      }\n      var element = document.createElement('script');\n      element.onload = on_load;\n      element.onerror = on_error;\n      element.async = false;\n      element.src = url;\n      element.type = \"module\";\n      console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n      document.head.appendChild(element);\n    }\n    if (!js_urls.length && !js_modules.length) {\n      on_load()\n    }\n  };\n\n  function inject_raw_css(css) {\n    const element = document.createElement(\"style\");\n    element.appendChild(document.createTextNode(css));\n    document.body.appendChild(element);\n  }\n\n  var js_urls = [\"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ace.js\", \"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-language_tools.js\", \"https://cdnjs.cloudflare.com/ajax/libs/ace/1.4.11/ext-modelist.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-2.4.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-gl-2.4.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-2.4.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-2.4.3.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-mathjax-2.4.3.min.js\", \"https://unpkg.com/@holoviz/panel@0.14.3/dist/panel.min.js\"];\n  var js_modules = [];\n  var css_urls = [\"https://cdn.holoviz.org/panel/0.14.3/dist/css/debugger.css\", \"https://cdn.holoviz.org/panel/0.14.3/dist/css/alerts.css\", \"https://cdn.holoviz.org/panel/0.14.3/dist/css/card.css\", \"https://cdn.holoviz.org/panel/0.14.3/dist/css/widgets.css\", \"https://cdn.holoviz.org/panel/0.14.3/dist/css/markdown.css\", \"https://cdn.holoviz.org/panel/0.14.3/dist/css/json.css\", \"https://cdn.holoviz.org/panel/0.14.3/dist/css/loading.css\", \"https://cdn.holoviz.org/panel/0.14.3/dist/css/dataframe.css\"];\n  var inline_js = [    function(Bokeh) {\n      inject_raw_css(\"/*\\n ~ CML // Creative Machine Learning ~\\n mml.css : CSS styling information for Panel and Bokeh\\n \\n This file defines the main CSS styling information for the CML course\\n \\n Author               :  Philippe Esling\\n                        <esling@ircam.fr>\\n*/\\n\\nbody {\\n  display: flex;\\n  height: 100vh;\\n  margin: 0px;\\n  overflow-x: hidden;\\n  overflow-y: hidden;\\n}\\n\\n.bk-root .bk, .bk-root .bk:before, .bk-root .bk:after {\\n  font-family: \\\"Josefin Sans\\\";\\n}\\n\\nimg {\\n  max-width: 100%;\\n}\\n\\n#container {\\n  padding:0px;\\n  height:100vh;\\n  width: 100vw;\\n  max-width: 100vw;\\n}\\n\\n#sidebar .mdc-list {\\n  padding-left: 5px;\\n  padding-right: 5px;\\n}\\n\\n.mdc-drawer-app-content {\\n  flex: auto;\\n  position: relative;\\n  overflow: hidden;\\n}\\n\\n.mdc-drawer {\\n  background: #FAFAFA; /* GRAY 50 */\\n}\\n\\n.mdc-drawer-app-content {\\n  margin-left: 0 !important;\\n}\\n\\n.title-bar {\\n  display: contents;\\n  justify-content: center;\\n  align-content: center;\\n  width: 100%;\\n}\\n\\n.mdc-top-app-bar .bk-menu {\\n  color: black\\n}\\n\\n.app-header {\\n  display: contents;\\n  padding-left: 10px;\\n  font-size: 1.25em;\\n}\\n\\nimg.app-logo {\\n  padding-right: 10px;\\n  font-size: 28px;\\n  height: 30px;\\n  max-width: inherit;\\n  padding-top: 12px;\\n  padding-bottom: 6px;\\n}\\n\\n#app-title {\\n  padding-right: 12px;\\n  padding-left: 12px;\\n}\\n\\n.title {\\n  font-family: \\\"Josefin Sans\\\";\\n  color: #fff;\\n  text-decoration: none;\\n  text-decoration-line: none;\\n  text-decoration-style: initial;\\n  text-decoration-color: initial;\\n  font-weight: 400;\\n  font-size: 2em;\\n  line-height: 2em;\\n  white-space: nowrap;\\n}\\n\\n.main-content {\\n  overflow-y: scroll;\\n  overflow-x: auto;\\n}\\n\\n#header {\\n  position: absolute;\\n  z-index: 7;\\n}\\n\\n#header-items {\\n  width: 100%;\\n  margin-left:15px;\\n}\\n\\n.pn-busy-container {\\n  align-items: center;\\n  justify-content: center;\\n  display: flex;\\n}\\n\\n.mdc-drawer__content {\\n  overflow-x: hidden;\\n}\\n.mdc-drawer__content, .main-content {\\n  padding: 12px;\\n}\\n\\n.main-content {\\n  height: calc(100vh - 88px);\\n  max-height: calc(100vh - 88px);\\n  padding-right: 32px;\\n}\\n\\nbutton.mdc-button.mdc-card-button {\\n  color: transparent;\\n  height: 50px;\\n}\\n\\np.bk.mdc-button {\\n  display: none;\\n}\\n\\ndiv.bk.mdc-card {\\n  border-radius: 0px\\n}\\n\\n.mdc-card .bk.card-header {\\n  display: flex;\\n}\\n\\n.bk.mdc-card-title {\\n  font-family: \\\"Josefin Sans\\\";\\n  font-weight: bold;\\n  align-items: center;\\n  display: flex !important;\\n  position: relative !important;\\n}\\n\\n.bk.mdc-card-title:nth-child(2) {\\n  margin-left: -1.4em;\\n}\\n\\n.pn-modal {\\n  overflow-y: scroll;\\n  width: 100%;\\n  display: none;\\n  position: absolute;\\n  top: 0;\\n  left: 0;\\n}\\n\\n.pn-modal-content {\\n  font-family: \\\"Josefin Sans\\\";\\n  background-color: #0e0e0e;\\n  margin: auto;\\n  margin-top: 25px;\\n  margin-bottom: 25px;\\n  padding: 15px 20px 20px 20px;\\n  border: 1px solid #888;\\n  width: 80% !important;\\n}\\n\\n.pn-modal-close {\\n  position: absolute;\\n  right: 25px;\\n  z-index: 100;\\n}\\n\\n.pn-modal-close:hover,\\n.pn-modal-close:focus {\\n  color: #000;\\n  text-decoration: none;\\n  cursor: pointer;\\n}\\n\\n.custom_button_bokeh button.bk.bk-btn.bk-btn-default {\\n    font-size:48pt;\\n    background-color: #05b7ff;\\n    border-color: #05b7ff;\\n}\");\n    },    function(Bokeh) {\n      inject_raw_css(\"\\n    .bk.pn-loading.arc:before {\\n      background-image: url(\\\"data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHN0eWxlPSJtYXJnaW46IGF1dG87IGJhY2tncm91bmQ6IG5vbmU7IGRpc3BsYXk6IGJsb2NrOyBzaGFwZS1yZW5kZXJpbmc6IGF1dG87IiB2aWV3Qm94PSIwIDAgMTAwIDEwMCIgcHJlc2VydmVBc3BlY3RSYXRpbz0ieE1pZFlNaWQiPiAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjYzNjM2MzIiBzdHJva2Utd2lkdGg9IjEwIiByPSIzNSIgc3Ryb2tlLWRhc2hhcnJheT0iMTY0LjkzMzYxNDMxMzQ2NDE1IDU2Ljk3Nzg3MTQzNzgyMTM4Ij4gICAgPGFuaW1hdGVUcmFuc2Zvcm0gYXR0cmlidXRlTmFtZT0idHJhbnNmb3JtIiB0eXBlPSJyb3RhdGUiIHJlcGVhdENvdW50PSJpbmRlZmluaXRlIiBkdXI9IjFzIiB2YWx1ZXM9IjAgNTAgNTA7MzYwIDUwIDUwIiBrZXlUaW1lcz0iMDsxIj48L2FuaW1hdGVUcmFuc2Zvcm0+ICA8L2NpcmNsZT48L3N2Zz4=\\\");\\n      background-size: auto calc(min(50%, 400px));\\n    }\\n    \");\n    },    function(Bokeh) {\n      Bokeh.set_log_level(\"info\");\n    },\nfunction(Bokeh) {} // ensure no trailing comma for IE\n  ];\n\n  function run_inline_js() {\n    if ((root.Bokeh !== undefined) || (force === true)) {\n      for (var i = 0; i < inline_js.length; i++) {\n        inline_js[i].call(root, root.Bokeh);\n      }} else if (Date.now() < root._bokeh_timeout) {\n      setTimeout(run_inline_js, 100);\n    } else if (!root._bokeh_failed_load) {\n      console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n      root._bokeh_failed_load = true;\n    }\n  }\n\n  if (root._bokeh_is_loading === 0) {\n    console.debug(\"Bokeh: BokehJS loaded, going straight to plotting\");\n    run_inline_js();\n  } else {\n    load_libs(css_urls, js_urls, js_modules, function() {\n      console.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n      run_inline_js();\n    });\n  }\n}(window));"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/javascript": [
       "\n",
       "if ((window.PyViz === undefined) || (window.PyViz instanceof HTMLElement)) {\n",
       "  window.PyViz = {comms: {}, comm_status:{}, kernels:{}, receivers: {}, plot_index: []}\n",
       "}\n",
       "\n",
       "\n",
       "    function JupyterCommManager() {\n",
       "    }\n",
       "\n",
       "    JupyterCommManager.prototype.register_target = function(plot_id, comm_id, msg_handler) {\n",
       "      if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n",
       "        var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n",
       "        comm_manager.register_target(comm_id, function(comm) {\n",
       "          comm.on_msg(msg_handler);\n",
       "        });\n",
       "      } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n",
       "        window.PyViz.kernels[plot_id].registerCommTarget(comm_id, function(comm) {\n",
       "          comm.onMsg = msg_handler;\n",
       "        });\n",
       "      } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n",
       "        google.colab.kernel.comms.registerTarget(comm_id, (comm) => {\n",
       "          var messages = comm.messages[Symbol.asyncIterator]();\n",
       "          function processIteratorResult(result) {\n",
       "            var message = result.value;\n",
       "            console.log(message)\n",
       "            var content = {data: message.data, comm_id};\n",
       "            var buffers = []\n",
       "            for (var buffer of message.buffers || []) {\n",
       "              buffers.push(new DataView(buffer))\n",
       "            }\n",
       "            var metadata = message.metadata || {};\n",
       "            var msg = {content, buffers, metadata}\n",
       "            msg_handler(msg);\n",
       "            return messages.next().then(processIteratorResult);\n",
       "          }\n",
       "          return messages.next().then(processIteratorResult);\n",
       "        })\n",
       "      }\n",
       "    }\n",
       "\n",
       "    JupyterCommManager.prototype.get_client_comm = function(plot_id, comm_id, msg_handler) {\n",
       "      if (comm_id in window.PyViz.comms) {\n",
       "        return window.PyViz.comms[comm_id];\n",
       "      } else if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n",
       "        var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n",
       "        var comm = comm_manager.new_comm(comm_id, {}, {}, {}, comm_id);\n",
       "        if (msg_handler) {\n",
       "          comm.on_msg(msg_handler);\n",
       "        }\n",
       "      } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n",
       "        var comm = window.PyViz.kernels[plot_id].connectToComm(comm_id);\n",
       "        comm.open();\n",
       "        if (msg_handler) {\n",
       "          comm.onMsg = msg_handler;\n",
       "        }\n",
       "      } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n",
       "        var comm_promise = google.colab.kernel.comms.open(comm_id)\n",
       "        comm_promise.then((comm) => {\n",
       "          window.PyViz.comms[comm_id] = comm;\n",
       "          if (msg_handler) {\n",
       "            var messages = comm.messages[Symbol.asyncIterator]();\n",
       "            function processIteratorResult(result) {\n",
       "              var message = result.value;\n",
       "              var content = {data: message.data};\n",
       "              var metadata = message.metadata || {comm_id};\n",
       "              var msg = {content, metadata}\n",
       "              msg_handler(msg);\n",
       "              return messages.next().then(processIteratorResult);\n",
       "            }\n",
       "            return messages.next().then(processIteratorResult);\n",
       "          }\n",
       "        }) \n",
       "        var sendClosure = (data, metadata, buffers, disposeOnDone) => {\n",
       "          return comm_promise.then((comm) => {\n",
       "            comm.send(data, metadata, buffers, disposeOnDone);\n",
       "          });\n",
       "        };\n",
       "        var comm = {\n",
       "          send: sendClosure\n",
       "        };\n",
       "      }\n",
       "      window.PyViz.comms[comm_id] = comm;\n",
       "      return comm;\n",
       "    }\n",
       "    window.PyViz.comm_manager = new JupyterCommManager();\n",
       "    \n",
       "\n",
       "\n",
       "var JS_MIME_TYPE = 'application/javascript';\n",
       "var HTML_MIME_TYPE = 'text/html';\n",
       "var EXEC_MIME_TYPE = 'application/vnd.holoviews_exec.v0+json';\n",
       "var CLASS_NAME = 'output';\n",
       "\n",
       "/**\n",
       " * Render data to the DOM node\n",
       " */\n",
       "function render(props, node) {\n",
       "  var div = document.createElement(\"div\");\n",
       "  var script = document.createElement(\"script\");\n",
       "  node.appendChild(div);\n",
       "  node.appendChild(script);\n",
       "}\n",
       "\n",
       "/**\n",
       " * Handle when a new output is added\n",
       " */\n",
       "function handle_add_output(event, handle) {\n",
       "  var output_area = handle.output_area;\n",
       "  var output = handle.output;\n",
       "  if ((output.data == undefined) || (!output.data.hasOwnProperty(EXEC_MIME_TYPE))) {\n",
       "    return\n",
       "  }\n",
       "  var id = output.metadata[EXEC_MIME_TYPE][\"id\"];\n",
       "  var toinsert = output_area.element.find(\".\" + CLASS_NAME.split(' ')[0]);\n",
       "  if (id !== undefined) {\n",
       "    var nchildren = toinsert.length;\n",
       "    var html_node = toinsert[nchildren-1].children[0];\n",
       "    html_node.innerHTML = output.data[HTML_MIME_TYPE];\n",
       "    var scripts = [];\n",
       "    var nodelist = html_node.querySelectorAll(\"script\");\n",
       "    for (var i in nodelist) {\n",
       "      if (nodelist.hasOwnProperty(i)) {\n",
       "        scripts.push(nodelist[i])\n",
       "      }\n",
       "    }\n",
       "\n",
       "    scripts.forEach( function (oldScript) {\n",
       "      var newScript = document.createElement(\"script\");\n",
       "      var attrs = [];\n",
       "      var nodemap = oldScript.attributes;\n",
       "      for (var j in nodemap) {\n",
       "        if (nodemap.hasOwnProperty(j)) {\n",
       "          attrs.push(nodemap[j])\n",
       "        }\n",
       "      }\n",
       "      attrs.forEach(function(attr) { newScript.setAttribute(attr.name, attr.value) });\n",
       "      newScript.appendChild(document.createTextNode(oldScript.innerHTML));\n",
       "      oldScript.parentNode.replaceChild(newScript, oldScript);\n",
       "    });\n",
       "    if (JS_MIME_TYPE in output.data) {\n",
       "      toinsert[nchildren-1].children[1].textContent = output.data[JS_MIME_TYPE];\n",
       "    }\n",
       "    output_area._hv_plot_id = id;\n",
       "    if ((window.Bokeh !== undefined) && (id in Bokeh.index)) {\n",
       "      window.PyViz.plot_index[id] = Bokeh.index[id];\n",
       "    } else {\n",
       "      window.PyViz.plot_index[id] = null;\n",
       "    }\n",
       "  } else if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !== undefined) {\n",
       "    var bk_div = document.createElement(\"div\");\n",
       "    bk_div.innerHTML = output.data[HTML_MIME_TYPE];\n",
       "    var script_attrs = bk_div.children[0].attributes;\n",
       "    for (var i = 0; i < script_attrs.length; i++) {\n",
       "      toinsert[toinsert.length - 1].childNodes[1].setAttribute(script_attrs[i].name, script_attrs[i].value);\n",
       "    }\n",
       "    // store reference to server id on output_area\n",
       "    output_area._bokeh_server_id = output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n",
       "  }\n",
       "}\n",
       "\n",
       "/**\n",
       " * Handle when an output is cleared or removed\n",
       " */\n",
       "function handle_clear_output(event, handle) {\n",
       "  var id = handle.cell.output_area._hv_plot_id;\n",
       "  var server_id = handle.cell.output_area._bokeh_server_id;\n",
       "  if (((id === undefined) || !(id in PyViz.plot_index)) && (server_id !== undefined)) { return; }\n",
       "  var comm = window.PyViz.comm_manager.get_client_comm(\"hv-extension-comm\", \"hv-extension-comm\", function () {});\n",
       "  if (server_id !== null) {\n",
       "    comm.send({event_type: 'server_delete', 'id': server_id});\n",
       "    return;\n",
       "  } else if (comm !== null) {\n",
       "    comm.send({event_type: 'delete', 'id': id});\n",
       "  }\n",
       "  delete PyViz.plot_index[id];\n",
       "  if ((window.Bokeh !== undefined) & (id in window.Bokeh.index)) {\n",
       "    var doc = window.Bokeh.index[id].model.document\n",
       "    doc.clear();\n",
       "    const i = window.Bokeh.documents.indexOf(doc);\n",
       "    if (i > -1) {\n",
       "      window.Bokeh.documents.splice(i, 1);\n",
       "    }\n",
       "  }\n",
       "}\n",
       "\n",
       "/**\n",
       " * Handle kernel restart event\n",
       " */\n",
       "function handle_kernel_cleanup(event, handle) {\n",
       "  delete PyViz.comms[\"hv-extension-comm\"];\n",
       "  window.PyViz.plot_index = {}\n",
       "}\n",
       "\n",
       "/**\n",
       " * Handle update_display_data messages\n",
       " */\n",
       "function handle_update_output(event, handle) {\n",
       "  handle_clear_output(event, {cell: {output_area: handle.output_area}})\n",
       "  handle_add_output(event, handle)\n",
       "}\n",
       "\n",
       "function register_renderer(events, OutputArea) {\n",
       "  function append_mime(data, metadata, element) {\n",
       "    // create a DOM node to render to\n",
       "    var toinsert = this.create_output_subarea(\n",
       "    metadata,\n",
       "    CLASS_NAME,\n",
       "    EXEC_MIME_TYPE\n",
       "    );\n",
       "    this.keyboard_manager.register_events(toinsert);\n",
       "    // Render to node\n",
       "    var props = {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n",
       "    render(props, toinsert[0]);\n",
       "    element.append(toinsert);\n",
       "    return toinsert\n",
       "  }\n",
       "\n",
       "  events.on('output_added.OutputArea', handle_add_output);\n",
       "  events.on('output_updated.OutputArea', handle_update_output);\n",
       "  events.on('clear_output.CodeCell', handle_clear_output);\n",
       "  events.on('delete.Cell', handle_clear_output);\n",
       "  events.on('kernel_ready.Kernel', handle_kernel_cleanup);\n",
       "\n",
       "  OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n",
       "    safe: true,\n",
       "    index: 0\n",
       "  });\n",
       "}\n",
       "\n",
       "if (window.Jupyter !== undefined) {\n",
       "  try {\n",
       "    var events = require('base/js/events');\n",
       "    var OutputArea = require('notebook/js/outputarea').OutputArea;\n",
       "    if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) == -1) {\n",
       "      register_renderer(events, OutputArea);\n",
       "    }\n",
       "  } catch(err) {\n",
       "  }\n",
       "}\n"
      ],
      "application/vnd.holoviews_load.v0+json": "\nif ((window.PyViz === undefined) || (window.PyViz instanceof HTMLElement)) {\n  window.PyViz = {comms: {}, comm_status:{}, kernels:{}, receivers: {}, plot_index: []}\n}\n\n\n    function JupyterCommManager() {\n    }\n\n    JupyterCommManager.prototype.register_target = function(plot_id, comm_id, msg_handler) {\n      if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n        var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n        comm_manager.register_target(comm_id, function(comm) {\n          comm.on_msg(msg_handler);\n        });\n      } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n        window.PyViz.kernels[plot_id].registerCommTarget(comm_id, function(comm) {\n          comm.onMsg = msg_handler;\n        });\n      } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n        google.colab.kernel.comms.registerTarget(comm_id, (comm) => {\n          var messages = comm.messages[Symbol.asyncIterator]();\n          function processIteratorResult(result) {\n            var message = result.value;\n            console.log(message)\n            var content = {data: message.data, comm_id};\n            var buffers = []\n            for (var buffer of message.buffers || []) {\n              buffers.push(new DataView(buffer))\n            }\n            var metadata = message.metadata || {};\n            var msg = {content, buffers, metadata}\n            msg_handler(msg);\n            return messages.next().then(processIteratorResult);\n          }\n          return messages.next().then(processIteratorResult);\n        })\n      }\n    }\n\n    JupyterCommManager.prototype.get_client_comm = function(plot_id, comm_id, msg_handler) {\n      if (comm_id in window.PyViz.comms) {\n        return window.PyViz.comms[comm_id];\n      } else if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n        var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n        var comm = comm_manager.new_comm(comm_id, {}, {}, {}, comm_id);\n        if (msg_handler) {\n          comm.on_msg(msg_handler);\n        }\n      } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n        var comm = window.PyViz.kernels[plot_id].connectToComm(comm_id);\n        comm.open();\n        if (msg_handler) {\n          comm.onMsg = msg_handler;\n        }\n      } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n        var comm_promise = google.colab.kernel.comms.open(comm_id)\n        comm_promise.then((comm) => {\n          window.PyViz.comms[comm_id] = comm;\n          if (msg_handler) {\n            var messages = comm.messages[Symbol.asyncIterator]();\n            function processIteratorResult(result) {\n              var message = result.value;\n              var content = {data: message.data};\n              var metadata = message.metadata || {comm_id};\n              var msg = {content, metadata}\n              msg_handler(msg);\n              return messages.next().then(processIteratorResult);\n            }\n            return messages.next().then(processIteratorResult);\n          }\n        }) \n        var sendClosure = (data, metadata, buffers, disposeOnDone) => {\n          return comm_promise.then((comm) => {\n            comm.send(data, metadata, buffers, disposeOnDone);\n          });\n        };\n        var comm = {\n          send: sendClosure\n        };\n      }\n      window.PyViz.comms[comm_id] = comm;\n      return comm;\n    }\n    window.PyViz.comm_manager = new JupyterCommManager();\n    \n\n\nvar JS_MIME_TYPE = 'application/javascript';\nvar HTML_MIME_TYPE = 'text/html';\nvar EXEC_MIME_TYPE = 'application/vnd.holoviews_exec.v0+json';\nvar CLASS_NAME = 'output';\n\n/**\n * Render data to the DOM node\n */\nfunction render(props, node) {\n  var div = document.createElement(\"div\");\n  var script = document.createElement(\"script\");\n  node.appendChild(div);\n  node.appendChild(script);\n}\n\n/**\n * Handle when a new output is added\n */\nfunction handle_add_output(event, handle) {\n  var output_area = handle.output_area;\n  var output = handle.output;\n  if ((output.data == undefined) || (!output.data.hasOwnProperty(EXEC_MIME_TYPE))) {\n    return\n  }\n  var id = output.metadata[EXEC_MIME_TYPE][\"id\"];\n  var toinsert = output_area.element.find(\".\" + CLASS_NAME.split(' ')[0]);\n  if (id !== undefined) {\n    var nchildren = toinsert.length;\n    var html_node = toinsert[nchildren-1].children[0];\n    html_node.innerHTML = output.data[HTML_MIME_TYPE];\n    var scripts = [];\n    var nodelist = html_node.querySelectorAll(\"script\");\n    for (var i in nodelist) {\n      if (nodelist.hasOwnProperty(i)) {\n        scripts.push(nodelist[i])\n      }\n    }\n\n    scripts.forEach( function (oldScript) {\n      var newScript = document.createElement(\"script\");\n      var attrs = [];\n      var nodemap = oldScript.attributes;\n      for (var j in nodemap) {\n        if (nodemap.hasOwnProperty(j)) {\n          attrs.push(nodemap[j])\n        }\n      }\n      attrs.forEach(function(attr) { newScript.setAttribute(attr.name, attr.value) });\n      newScript.appendChild(document.createTextNode(oldScript.innerHTML));\n      oldScript.parentNode.replaceChild(newScript, oldScript);\n    });\n    if (JS_MIME_TYPE in output.data) {\n      toinsert[nchildren-1].children[1].textContent = output.data[JS_MIME_TYPE];\n    }\n    output_area._hv_plot_id = id;\n    if ((window.Bokeh !== undefined) && (id in Bokeh.index)) {\n      window.PyViz.plot_index[id] = Bokeh.index[id];\n    } else {\n      window.PyViz.plot_index[id] = null;\n    }\n  } else if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !== undefined) {\n    var bk_div = document.createElement(\"div\");\n    bk_div.innerHTML = output.data[HTML_MIME_TYPE];\n    var script_attrs = bk_div.children[0].attributes;\n    for (var i = 0; i < script_attrs.length; i++) {\n      toinsert[toinsert.length - 1].childNodes[1].setAttribute(script_attrs[i].name, script_attrs[i].value);\n    }\n    // store reference to server id on output_area\n    output_area._bokeh_server_id = output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n  }\n}\n\n/**\n * Handle when an output is cleared or removed\n */\nfunction handle_clear_output(event, handle) {\n  var id = handle.cell.output_area._hv_plot_id;\n  var server_id = handle.cell.output_area._bokeh_server_id;\n  if (((id === undefined) || !(id in PyViz.plot_index)) && (server_id !== undefined)) { return; }\n  var comm = window.PyViz.comm_manager.get_client_comm(\"hv-extension-comm\", \"hv-extension-comm\", function () {});\n  if (server_id !== null) {\n    comm.send({event_type: 'server_delete', 'id': server_id});\n    return;\n  } else if (comm !== null) {\n    comm.send({event_type: 'delete', 'id': id});\n  }\n  delete PyViz.plot_index[id];\n  if ((window.Bokeh !== undefined) & (id in window.Bokeh.index)) {\n    var doc = window.Bokeh.index[id].model.document\n    doc.clear();\n    const i = window.Bokeh.documents.indexOf(doc);\n    if (i > -1) {\n      window.Bokeh.documents.splice(i, 1);\n    }\n  }\n}\n\n/**\n * Handle kernel restart event\n */\nfunction handle_kernel_cleanup(event, handle) {\n  delete PyViz.comms[\"hv-extension-comm\"];\n  window.PyViz.plot_index = {}\n}\n\n/**\n * Handle update_display_data messages\n */\nfunction handle_update_output(event, handle) {\n  handle_clear_output(event, {cell: {output_area: handle.output_area}})\n  handle_add_output(event, handle)\n}\n\nfunction register_renderer(events, OutputArea) {\n  function append_mime(data, metadata, element) {\n    // create a DOM node to render to\n    var toinsert = this.create_output_subarea(\n    metadata,\n    CLASS_NAME,\n    EXEC_MIME_TYPE\n    );\n    this.keyboard_manager.register_events(toinsert);\n    // Render to node\n    var props = {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n    render(props, toinsert[0]);\n    element.append(toinsert);\n    return toinsert\n  }\n\n  events.on('output_added.OutputArea', handle_add_output);\n  events.on('output_updated.OutputArea', handle_update_output);\n  events.on('clear_output.CodeCell', handle_clear_output);\n  events.on('delete.Cell', handle_clear_output);\n  events.on('kernel_ready.Kernel', handle_kernel_cleanup);\n\n  OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n    safe: true,\n    index: 0\n  });\n}\n\nif (window.Jupyter !== undefined) {\n  try {\n    var events = require('base/js/events');\n    var OutputArea = require('notebook/js/outputarea').OutputArea;\n    if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) == -1) {\n      register_renderer(events, OutputArea);\n    }\n  } catch(err) {\n  }\n}\n"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.bk-root, .bk-root .bk:before, .bk-root .bk:after {\n",
       "  font-family: var(--jp-ui-font-size1);\n",
       "  font-size: var(--jp-ui-font-size1);\n",
       "  color: var(--jp-ui-font-color1);\n",
       "}\n",
       "</style>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torchvision\n",
    "from cml.plot import initialize_bokeh\n",
    "from cml.panel import initialize_panel\n",
    "from jupyterthemes.stylefx import set_nb_theme\n",
    "from bokeh.io import show\n",
    "initialize_bokeh()\n",
    "initialize_panel()\n",
    "set_nb_theme(\"onedork\")\n",
    "rng = np.random.RandomState(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iYn4MdZnKCey"
   },
   "source": [
    "### Load the dataset\n",
    "\n",
    "To start with a pragmatic and simple to understand example, we  will try to train the basic AE using the Fashion MNIST dataset. This dataset contains images of size 28x28 pixels, with different pieces of clothing. The following code allows to load (and eventually download) the dataset, by using the `torchvision.datasets` module. We also plot some randomly selected test examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "YZm503-I_tji"
   },
   "outputs": [],
   "source": [
    "import matplotlib\n",
    "dataset_dir = './data'\n",
    "# Going to use 80%/20% split for train/valid\n",
    "valid_ratio = 0.2\n",
    "# Load the dataset for the training/validation sets\n",
    "train_valid_dataset = torchvision.datasets.FashionMNIST(root=dataset_dir, train=True, transform=torchvision.transforms.ToTensor(), download=True)\n",
    "# Split it into training and validation sets\n",
    "nb_train = int((1.0 - valid_ratio) * len(train_valid_dataset))\n",
    "nb_valid =  int(valid_ratio * len(train_valid_dataset))\n",
    "train_dataset, valid_dataset = torch.utils.data.dataset.random_split(train_valid_dataset, [nb_train, nb_valid])\n",
    "# Load the test set\n",
    "test_dataset = torchvision.datasets.FashionMNIST(root=dataset_dir, transform=torchvision.transforms.ToTensor(),train=False)\n",
    "# Prepare \n",
    "num_threads = 4     # Loading the dataset is using 4 CPU threads\n",
    "batch_size  = 128   # Using minibatches of 128 samples\n",
    "train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, num_workers=num_threads)\n",
    "valid_loader = torch.utils.data.DataLoader(dataset=valid_dataset, batch_size=batch_size, shuffle=False, num_workers=num_threads)\n",
    "test_loader = torch.utils.data.DataLoader(dataset=test_dataset,batch_size=batch_size,shuffle=False,num_workers=num_threads)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we are going to check the properties of our different sets and also plot some random examples, in order to better understand what type of data we will be working with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The train set contains 48000 images, in 375 batches\n",
      "The validation set contains 12000 images, in 94 batches\n",
      "The test set contains 10000 images, in 79 batches\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABiYAAAC0CAYAAAAQJpSlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy89olMNAAAACXBIWXMAAA9hAAAPYQGoP6dpAABYYElEQVR4nO3dd3TUVf7/8XeQkgAhgRASQpfQq4AiRRAUUFDpCi4IinUtuOKqq6tiW9eyq1/d/a3uqqigWEFhdXVVQAULCIKidJDeAqTS5f7+8GSWz73vMJcxmbTn4xzP8b65M/PJzJ37acl9xRhjjAAAAAAAAAAAAERBheLeAAAAAAAAAAAAUH5wYwIAAAAAAAAAAEQNNyYAAAAAAAAAAEDUcGMCAAAAAAAAAABEDTcmAAAAAAAAAABA1HBjAgAAAAAAAAAARA03JgAAAAAAAAAAQNRwYwIAAAAAAAAAAEQNNyYAAAAAAAAAAEDUlMkbEzExMYH/KlWqJLVr15Z27drJ+PHj5e2335ajR48W92ainJg7d64MHz5c6tWrJ5UrV5aaNWtKixYtZOTIkfK3v/1NsrKyQn0bN24sMTExJ/X848ePl5iYGJk3b14hbzlKOuY6FAfGHYrayew3S4IXX3xRYmJiZPLkyb/6uSI5DkDhYNwx7kq6vLw8+etf/yp9+vSRlJSU0Djt1q2b3HPPPbJp06bi3kQUkYULF4aOu+6///5Cf/6zzz5bYmJi5KeffvLqH43zz8Kc41B6cO2kfGOuK5/K5I2JfOPGjZNx48bJ6NGjpUePHnL06FF5+eWXZcSIEdKqVStZuHBhcW8iyrj7779f+vbtKzNmzJCEhAS54IILpH///hIXFyczZsyQG2+8UVasWFEs2zZv3jyJiYmR8ePHF8vro/Aw16E4MO5QFEryfhNlF+MOJd0XX3wh6enpMmnSJFm4cKG0bdtWRowYId27d5d169bJAw88IM2bN5ePP/44atv0008/SUxMjJx99tlRe83yaurUqaH/f+WVV4pxSyDCxe2iUpL3xVw7iQ7mupIlWnNdxSJ99mL24osvOrV169bJnXfeKW+88Yb06dNHFixYIB07doz6tqHsW7x4sUyePFkqVaokb7zxhgwZMiTw7zt27JBp06ZJYmLir3qdhx9+WO644w5p2LDhr3oelF7MdSgOjDsUtmjtN4HjMe5Q0i1dulTOOeccOXjwoNx+++1y9913S7Vq1UL/fuzYMXnnnXfktttuky1bthTjlqIoHDlyRF577TUREUlNTZXVq1fL119/LV27di3mLQMKD9dOwFxXfpXpv5jQNG3aVF5//XWZMGGC7N+/X6644ori3iSUUTNmzBBjjFx88cXOjlXkl8n21ltvlZYtW/6q16lbt660bNlSqlat+queB2ULcx2KA+MOv0a09pvA8Rh3KMmMMTJ27Fg5ePCgTJ48Wf785z8HbkqIiFSoUEGGDRsmixcvli5duhTTlqKofPDBB5KRkSE9evSQ3/72tyIS/K1ioCzg2gmY68qvcndjIt9f/vIXqVatmnz77bcyf/78UP34P0nNzs6WW265RZo0aSKVKlWSm2++OdRv79698oc//EFat24tcXFxkpCQIH379pV///vf6ustX75cxowZI6eeeqrExsZKcnKydOzYUW6++WbZvn17oO8XX3whQ4YMkUaNGkmVKlUkNTVVzjjjDLnjjjskNze3SN4PFL7du3eLiEhycnJEj3/uueekffv2EhcXJ6mpqXLNNddIZmam06+gP6/KX3PRGCNPP/20dOjQQapWrSodO3aU8ePHS58+fURE5KWXXgqsF1+e17Yri5jrUBwYd4hEJPvNpUuXym233SadO3eW5ORkqVKlipx66qny29/+VrZt2+b0P34MHjhwQO64447QWEhPT5dHHnlEjDHqay1YsEDOPfdciY+Pl8TERBkwYIB8/fXXBW7b2rVrZfLkydKtWzdJTU2VypUrS/369eWyyy6T1atXe/+MKFqMO5RkH3zwgSxfvlzq168vd9111wn7JiQkSNu2bUPt/fv3ywMPPCBt27YN7Ut79eoV+o1U28mM68mTJ0uTJk1EROTTTz8NnEuw1EnhmjZtmoiIjBkzRsaMGSMiIq+//rocOXJE7X/8uvu+55MFycrKkl69eklMTIzceOONBc5Tx9u/f788/PDDctppp0n16tWlevXqcuaZZ8pLL73k/bq2VatWyfDhwyUpKUmqVasmPXr0kPfff7/A/l9++aUMHjw4NI4bN25c4Pycb+rUqdKzZ0+pUaOGVK1aVdq3by8PP/ywHDx4MNAvJiYm9LP06dMnMPZ9162Hi2snYK4rx3OdKYNExPj8aCNGjDAiYu6///5QbcOGDUZEzBlnnGE6duxoatasaYYMGWKGDRtmJk+ebIwxZtWqVaZBgwZGREzjxo3N4MGDTd++fU3VqlWNiJjHHnss8DrffPONiY2NNSJi2rdvby6++GJzwQUXmNatWxsRMXPnzg31nTVrlqlQoYKJiYkxXbt2NaNGjTLnnXeeadq0qRERs2HDhkJ5j1D07r//fiMipkGDBmbnzp1ej2nUqJEREfP73//eVK5c2fTv398MHTrU1KlTx4iIOeuss8yxY8cCjxk3bpwzjo5/rquvvtpUqlTJnHvuueaSSy4xQ4cONf/617/MgAEDjIiYpk2bmnHjxoX+mzlzZiG9AyhqzHUoDow7FJVI9puXXHKJqVixounUqZMZMmSIGTJkiGncuLEREVO3bl2zdevWQP/8MditWzfTs2dPU6tWLTNs2DAzYMCA0Di66667nNeZPXu2qVixYmj8jho1yrRq1cpUrlzZXH311UZEzL333ht4zO23325iYmJMu3btzAUXXGCGDx9uWrVqZUTE1KhRwyxbtsx5nfx9N6KHcce4K8muv/56IyLmd7/73Uk9Ljs723Tu3NmIiElOTjYjRoww559/vqlSpYoREXPTTTc5jzmZcT1z5kwzfPhwIyImJSUlcC7xr3/961f/3PhFZmamiY2NNZUrVzZ79uwxxhjTvXt3IyJm1qxZ6mMiOZ/s3bu3cxy0Y8cO07FjRyMi5p577gn0L+j8c+fOnaZ9+/ZGRExqaqoZOHCgOf/8801CQoIREXPDDTd4/+xTpkwxImLGjBljEhISTJMmTcyoUaNMr169TExMjImJiTEvvPCC87ipU6eaU045xYiI6dGjhxk1apRp3rx5aKyuWLHCeUz+fBobG2sGDhxoRowYYWrXrh2at/Py8gI/e/5x44ABAwJjf/fu3d4/H4K4dlK+MdeV77muTB6B+l40efDBB42ImNGjR4dq+ScO+R/Mvn37Ao85evSoadeunRER8+ijj5qff/459G9r1qwxTZo0Maeccor5/vvvQ/XLLrvMiIh5/PHHnW1YsWKF2bZtW6jdq1cvIyLmrbfecvouXLjQZGdnh/25UDKsW7fOxMXFGREx8fHxoQP1JUuWmKNHj6qPyZ9cU1NTzcqVK0P13bt3m/T0dCMi5pNPPgk8JtzOtXbt2mb58uXOa82dO9eIiBk3btyv/llRPJjrUBwYdygqkew358yZY3bs2BGo/fzzz+a+++4zImIuv/zywL8dPwZ79+5tsrKyQv+2aNEic8opp5iqVauanJycUD07O9skJycbEQmcGBw7dszcfvvtoeezLxB/+eWXZv369c42v/DCC0ZETJ8+fZx/4wJx9DHuGHclWY8ePYyImKlTp57U42644YbQ5338vm3FihWhizazZ88OPCbScd27d++T+6Hg7bnnnjMiYgYPHhyq/b//9/+MiJiRI0eqj4nkfNK+WLdhwwaTnp5uYmJizFNPPeW8RkHnnwMHDjQiYiZOnGgOHjwYqu/YscN06dLFiIj5z3/+4/Wz51+sExFz2WWXmSNHjoT+bfbs2aF5c8uWLaH6pk2bTFxcnDnllFPMu+++G6r//PPP5uabbzYiYrp06RJ4nbfeesuIiElLSzOrV68O1TMzM03Pnj2NiJhJkyZ5/fyIHNdOyjfmuvI915XJI1DfiybPPPOMERFz3nnnhWrHnzgsWrTIeczMmTONiJjhw4erzzljxgwjEvwtlPPPP9+IiFm6dGnYbcr/jabMzMywfVHyffzxx6Hf/D3+v8TERHPdddcFLpgZ87/JVftNo8cff1w9AQ23c7V/uzgfO9fSj7kOxYFxh6J0svvNE6lXr55JSkoK1PLHYIUKFQInMfkuuOACZ5+af0G3V69eTv/Dhw+b+vXrq/vnE+nRo4eJiYlxxiIXiIsH445xV1K1bNnSiIj54IMPvB+Tm5tr4uLiTIUKFdTfmHzqqaeMiJhzzz3X+zlPNK65MVF08i+ivfnmm6FaRkaGqVSpkomNjVWPZyI5nzz+Yt3y5ctNWlqaqVixopk2bZq6Xdr557fffmtExJx++umBXyzJt2TJEiMi5qKLLvL62fMv1lWvXt3s3bvX+fdLLrnEiIh54IEHQrV77rnHiAR/KSbfwYMHTVpamhERM3/+/FA9/5dWnn32Wecxy5YtMzExMaZ69ermwIEDJ/z58etx7aT8Yq4r33Nduc2YEJHQumH565Idr27dump42H//+18RERk2bJj6nGeddZaIiCxcuDBU69y5s4iIXH/99TJv3jw5evRogduU33fs2LGyaNEiOXbsmM+PghLqnHPOkbVr18qMGTPk2muvlU6dOknFihUlMzNT/vGPf0jHjh1l1apVzuP69+/v1Jo3by4i4qyXHs5FF10U2cajzGCuQ3Fg3CESkew39+zZI1OmTJFJkybJhAkTZPz48TJ+/Hg5cuSI7NmzR/bu3eu8TqNGjaRFixZOXdvXfv755yIiMmrUKKd/pUqVZMSIEQX+PLm5uTJ9+nS5/fbb5aqrrgpt2/bt28UYI+vWrfN7Y1CkGHcoSxYvXiwHDhyQTp06qUGxY8eOFZFf8kvs/WAk4xpFY9OmTfLZZ59JYmKiXHjhhaF6UlKSDBw4UA4ePChvvvlmgY+P5Hzyq6++kl69esm+fftk5syZ8pvf/MZ7e/OP4YYMGSIVKriXmfLXYT/+GM5H//79pWbNmk599OjRIvK/ufL4/9e2u0qVKjJy5MhAvyNHjshXX31V4GPat28v7du3l9zcXFm6dOlJbTdOHtdOyifmul+U57muYtRfsQTJyMgQEZFatWo5/9awYUP1MfkhH7/5zW9OOHjzn1tE5Pe//73Mnz9f5s2bJ3369JHq1atLt27dZNCgQTJ+/HhJSEgI9f3Tn/4k33//vcyePVtmz54tNWvWlJ49e8pFF10kY8aMkdjY2Eh+VBSjypUry9ChQ2Xo0KEiIpKZmSmvvfaa3HnnnbJr1y654YYb5KOPPgo8pn79+s7zxMfHi4jIoUOHTur1CxrLKD+Y61AcGHeI1MnsN6dPny5XX331CYPLc3JynHGo7WdF9H1tfoBco0aN1Mc0btxYrc+ZM0dGjRoVCnQsaNtQMjDuUBIlJSWJiJzw87Tlj52CxkhiYqIkJCRIVlaW7Nu3L/QakY5rFI1XXnlFjDEyYsQIqVKlSuDfxowZI++++65MmzZNrrzySvXxkZxPjh07Vo4ePSqvv/66XHDBBSe1vfnHcHfdddcJg9rtgNVwws2Bx4e8hhv7+fWtW7eKyC834g4fPiy1a9eWatWqFfiYZcuWhR6DosW1k/KHue4X5XmuK9c3Jr799lsREWndurXzbwVdnMj/rZLzzjtPUlJSCnzu2rVrh/6/Ro0aMmfOHFmwYIHMnj1b5s2bJ3PmzJGPPvpIHn74Yfn888+lWbNmIiLSoEED+eabb2TOnDny73//Wz799NPQBZRHH31Uvvzyy9DBI0qnxMREufbaayUtLU0GDx4sc+fOlf3790vVqlVDfbQ7r5HiQhuY61AcGHcoLAXtN3fv3i3jx48XEZEnn3xSBg0aJPXq1ZO4uDgREenevbt8+eWXob/eOV5h7mc1ubm5cvHFF8vevXvlnnvukVGjRkmjRo0kLi5OYmJi5NJLL5Xp06er24aSgXGHkqBjx46yYMECWbJkiYwZM6bQntf+a8aNGzdGPK5RNKZOnSoiIvPmzZOePXsG/u3w4cMiIvLZZ5/Jxo0b1Qtakcw3o0ePlqlTp8o999wjvXr1ktTUVO/H5h/D9ezZU5o2bXrSrx0N2l/xFsVjUHi4dlL2MdcVvtI215XbGxNZWVny4YcfiohInz59vB+XfzfuyiuvlOHDh3s/LiYmRnr27Bn6ou3atUtuvvlmmT59utx1113yxhtvhPpWrFhR+vfvH/qTpI0bN8oVV1whc+bMkUceeUQeffRR79dFydW3b18REfn5558lMzMzsHMFCgtzHYoD4w5Fwd5vvv/++3L48GG59dZbZeLEiU7/9evXF8rr1q1bV0R+GSsarf7555/Lnj17ZMSIEXLfffcV2bah6DHuUJwGDRokf//73+XNN9+URx99VCpWDH/6npaWJiIFj52srCzJzMyUuLi40LIR0RrX8LN48WJZsWKFiIisXbtW1q5dq/Yzxsgrr7wid955Z6G87v333y9paWnyyCOPSN++fWXevHlSp04dr8fmH8MNGTJEJk2aVCjbIxJ+Dswf7/n/v2rVKtm4caO0adPGeUz+bzrXq1dPRH75i6TKlStLRkaG5OXlqb9JbD8GxYNrJ2UTc93/lOe5rtxmTEyaNEny8vLk9NNPl27dunk/rl+/fiIiMnPmzF/1+nXq1JHJkyeLiMjy5ctP2LdRo0Zy++23e/VFyRHuN4ryJ93KlSsHfvs3WipXriwicsL12FH6MdehODDuEImT3W/u27dPRPQ/4f7ss89k586dhbJd+dkmx9/gynf06FF5++23nfqJtm3t2rWyZMmSQtk2/HqMO5Rk5513nrRp00a2bNkiDz300An7Zmdnyw8//CCdO3eWuLg4Wbx4saxZs8bpN23aNBER6dGjR+g3TSMZ15xLFJ38z+jWW28VY4z637x58wJ9C8uf//xnufXWW2XFihXSt29f72XECusYzvbf//5XMjMznfprr70mIhL4Dev8eXP69OlO/8OHD4fWqc/vV6lSJTnzzDMDz3e85cuXy7Jly6R69erSsWPHUJ2xX/i4dlI+Mdf9T3me68rdjYn169fLJZdcIs8//7xUq1ZNnn/++ZN6/PDhw6V169byyiuvyAMPPOCsWWaMkQULFsiCBQtCtWeeeUY2bNjgPNf7778vIr8sLZHviSeekB07dnj1Rcl29913y+9//3s1YHDr1q1yzTXXiMgvAUv5X/hoyr/jqgVIofRjrkNxYNzh1zjZ/WZ+sN20adMkLy8v0Pfaa68ttO0aOXKkJCUlybx58+Sll14K1Y0xcu+998qmTZucx+Rv24wZMwInOpmZmTJhwgQ5cuRIoW0ffh3GHUqymJgYmTZtmsTGxsrkyZPlD3/4Q2DcifwyJmbNmiVdunSRRYsWSbVq1eSKK66QY8eOyfXXXx/ov3r1annwwQdFROSmm24K1SMZ17Vr15ZKlSrJunXr5Oeffy60n7m8+/nnn0MXm/JDTzVnnXWW1KtXT1asWCGLFy8u1G147LHH5JZbbpEffvhBzjnnnEC2V0G6du0q/fr1kwULFsj1118v2dnZTp9ly5bJBx98cFLbkpubK7fcckvgwth//vMfeeONNyQuLk4uv/zyUH3ChAkSFxcnr732mrz33nuh+rFjx+TOO++UrVu3SufOnaVHjx6hf7vxxhtFRGTy5MmBvwzKycmRG264QYwxcs011wSW+OE8uvBx7aT8Ya4LKtdznSmDRMSIiBk3bpwZN26cGTt2rBk8eLBp1aqViYmJMSJimjVrZhYtWuQ8dsOGDUZETO/evQt8/tWrV5smTZoYETF16tQx5557rrn00ktN//79TZ06dYyImCeeeCLUv0OHDkZETOvWrc3w4cPNJZdcEqrFxsaa+fPnh/omJCSYChUqmNNOO81cfPHFZuTIkaZ58+ZGREytWrXM6tWrC/OtQhGaOHFiaCw2b97cDBkyxIwaNcr07NnTVKpUyYiISU9PN1u2bAk9plGjRqagr+XcuXND4/p448aNMyJi5s6dG6if6LnytW/f3oiIOf3008348ePNhAkTzLvvvhvRz4voY65DcWDcoaic7H7z0KFDpk2bNkZETGpqqhk+fLgZNGiQqVq1qunevbvp3r27ERGzYcOG0GuEG4P33nuvEREzZcqUQP2dd94xp5xyihER07VrVzN69GjTunVrU6lSJXPVVVcZETH33ntv4DH9+vUzImISExPNkCFDzJAhQ0xiYqJJT083gwcPjnjfjcLFuGPclQbz5883KSkpRkRM1apVzTnnnGMuvfRSM2jQoFA9NjbWfPzxx8YYY7Kzs03nzp1D+9KRI0eagQMHmtjYWCMi5qabbgo8fyTj2hhjLrzwQiMipk2bNmbs2LFmwoQJ5oUXXojW21Imvf/++6H5KJxbbrnFiIiZOHFiqBbJ+WTv3r3Vzzd/fmzfvr3JyMgI1Qs6/9y5c6c57bTTQnPQ2WefHRqnDRo0cLb1RKZMmWJExPzmN78xCQkJpkmTJmbUqFGmd+/eoePNf/3rX87jXn75ZVOhQgUTExNjevbsaUaPHm1atGhhRMSkpKSYFStWOI+5+uqrjYiYuLg4M2jQIDNy5EiTnJxsRMSceeaZJi8vL9D/m2++MTExMSY2NtYMHjzYTJgwwUyYMCHwHuHkcO2k/GGu+wVz3S9/GlPm5E9o+f9VrFjR1KpVy7Rt29aMGzfOzJgxwxw9elR9rM9FE2OMyczMNA8++KDp1KmTqV69uomNjTWNGzc2AwYMMH//+9/N7t27Q31nzZplrrjiCtOmTRuTmJhoqlatapo3b26uvPJKs3LlysDzvvzyy+bSSy81LVq0MPHx8SY+Pt60bt3a3HLLLYFJGCXf7t27zdSpU82YMWNMu3btTFJSUmgs9ujRwzz66KMmNzc38Jho71zXrFljhgwZYpKSkkyFChXUE1yUXMx1KA6MOxSVSPabe/fuNdddd51p3LixqVKlijn11FPN7bffbvLy8tSTj0gvEBtjzGeffWb69OljqlWrZmrUqGHOOecc88UXX4ROKOz95/79+81dd91lmjVrZqpUqWIaNGhgrr32WpORkfGr9t0oXIw7xl1pkZOTYx5//HHTu3dvk5ycbCpWrGgSExNN165dzb333ms2b94c6J+bm2vuu+8+07p1a1OlShUTHx9vevbsaV599VX1+U92XBvzy8WZsWPHmtTU1NBNNPtcBSdn9OjR3udkixYtCt18OnLkiDGmcC/WGWPMjTfeaETEdOzY0ezZs8cYU/D5pzHGHDhwwDz11FOme/fuJiEhwVSuXNk0aNDA9O7d2zz22GPOOC3I8XPcjz/+aAYPHmxq1qxp4uLiTLdu3czs2bMLfOyCBQvMhRdeaJKSkkylSpVMw4YNzXXXXXfCY72XX37ZdO/ePXTc2aZNG/PQQw+Z/fv3q/1feeUV06lTJxMXFxc6JtbeP/jh2kn5w1z3C+Y6Y2KMCbOYGwAAAAAAAAAAQCEpdxkTAAAAAAAAAACg+HBjAgAAAAAAAAAARA03JgAAAAAAAAAAQNRwYwIAAAAAAAAAAEQNNyYAAAAAAAAAAEDUcGMCAAAAAAAAAABETcVIH3js2DHZtm2bxMfHS0xMTGFuE0oZY4zk5ORIWlqaVKhQdPe6GHM4HuMO0RatMSfCuMP/MNehODDuEG3sY1EcmOtQHBh3iDb2sSgOvuMu4hsT27ZtkwYNGkT6cJRBmzdvlvr16xfZ8zPmoGHcIdqKesyJMO7gYq5DcWDcIdrYx6I4MNehODDuEG3sY1Ecwo27iG9MxMfHR/rQEiEpKcmp9e/f36nNmjUr0M7Lyyu0bTj11FOd2imnnOLU1qxZU2ivWZSKekyU5DF39tlnB9ppaWlOn1dffTVKW3NyrrnmGqf2448/Btqff/55tDbnpJXncVe7du1A+84773T6dOvWzalt27bNqf3000+BtjbvaHe569atG2g3adLE6dO0aVOntnr1aqd2xRVXBNo///yz06ckiMaYKMnjrlKlSoH2kSNHIn6ul19+OdCuWrWq02fevHlOrUaNGoF2YmKi0+e2226LeLtKovI816H4MO4QbeV9Hxspbf85ceLEQNsY4/TZu3evUztw4ECgbe/3RUTefvttp5aVlRV2O0sq5joUB8Zd4ejQoUOg3bVrV6fP1q1bnVrDhg0D7blz5zp9Vq5c+Su3rmRhH4viEG5MRHxjorT/SY52ga1y5cpOrSh/Tm0btBsTpUVRj4lojDn7NbQDeE3FisGvkjaWSqoqVao4NfvnKcnKwriLlD2HxMXFOX2qV6/u1KpVq+bU7Mdq40Kbs2JjYwNt7cRY2watX0l+r49XHHNRSVKY22aPA21samPRHnfa2C9ryvNch+LDuEO0lfd9bKS0n8neV2rnNdo5i/2LIdqNibL2HjLXoTgw7gqHfQ1Nm9e0ecw+xyjN1+J8sY9FcQg3Jgi/BgAAAAAAAAAAUVN6fi26ANqdT3tZCW2JprFjxzo1+7dKRNxlbrS7qPafu4qIJCcnB9qHDx92+mzevNmpaX9Ou2rVqkD74Ycfdvpo21VSl0EpKbTf/j527FigrY2v5557zqllZ2cH2kePHnX6PPHEE07tu+++c2r2Mjvanx1qf9HQvHnzQLtLly5OH632ww8/OLVmzZoF2vYSOyIit9xyi1PbvXt3oK3dGfX9KxQEDR482Km98847gfa+ffucPvaYFhFp3LixUxswYEDE2xZORkaGU6tXr55Ts+dubVkoe8kpRJ82v/n4wx/+ELaPNlYGDhzo1A4ePBhoa8uTtW3b1qktX7487DYAAFBcIv3r7SVLljg1+3jv0KFDTh9teYWcnJxAWzvvuPvuu51a7969ndratWsDbZ/zLwBlS1FfExg3blyg3a9fP6fPhg0bnJqdg1CnTh2nj7ZUcmGKdM4HyhL+YgIAAAAAAAAAAEQNNyYAAAAAAAAAAEDUcGMCAAAAAAAAAABETanPmPCh5TtoVq5c6dTS0tIC7fr16zt9tHwHe63MTZs2OX3WrVvn1BITE52avW6/JlzKOVw+65n+9a9/dWqZmZlOzV7bX1s/VVvDVVvXPzc3N9CuUqWK00fLvti/f3+grWWffPnll05NY7832nulvTd2dgtrJEbGXu9SROTFF190atu3bw+07XX3RfS5Yc+ePU7NHrN2To6IPhbt74OWEVC7dm2nlpeX59TsMautBZqSkuLUdu3a5dRQdHzmzuuvv96p3XvvvU6tZ8+egfZZZ53l9NH2sZ9//nmgreX1aBklO3fudGo++1gAAEqKO+64w6k1bNjQqdnnJ1r+oH3+IOIeO2rnNfY5sojItGnTnNqZZ54ZaJMnAZQ/PtcEUlNTnZqWj6kd39vX1UaOHOn00TLr7HxX7Zzysssuc2pa5qGd26mdb2u4XgLwFxMAAAAAAAAAACCKuDEBAAAAAAAAAACihhsTAAAAAAAAAAAgargxAQAAAAAAAAAAoqbUh1/7hMVo4TRa+JcW7LpkyZJAWwu/PnLkiFOzwzoXLVrk9NGCxLSfRwuT9XkunLzKlSuH7aMFDMfFxQXaWuB6VlaWV80O/B02bJjTRwsOtoOttZBgLYBb+y7Y74P282iBTva4154b4S1btsyrn/3+aiHB2mdQvXp1p3bo0KFAOycnx2sb7DGcmJjo9NECuLVttbdB+66tWrXKqdWsWTPcZqIIjR8/3qk99NBDTk2bX6dPnx5oa/PWk08+6dTscdapUyenz/Lly53alVde6dRyc3MD7aefftrpAwBANPic2/br18+paeeLdtC0dv5QsaJ7OcCuaecP2jGadk7sQztOJBC27OrRo4dTs88ntm3b5vSxg4oRfT7fVa3Pueee69TS09MDbe1a3P79+52afdwuIrJixYpA+7PPPnP6DBo0yKl98803gfa8efOcPs2bN3dqzZo1c2qtW7cOtLVrJdo8bW/7jh07nD5AWcfVbAAAAAAAAAAAEDXcmAAAAAAAAAAAAFHDjQkAAAAAAAAAABA13JgAAAAAAAAAAABRU+rDr33YwV8iIvXq1XNqu3btcmpbt24NtLUgbS108/PPPw+0tfAbLeCnQYMGTs0nhPbo0aNh+yC8atWqBdpacLAWNG4H+dohviJ6EJT2XHXr1g20N23a5PTRPu9atWoF2pUqVXL6aN8F7We0H6sFFcfGxjq1Ro0aBdrr1693+sD15z//OdCuWrWq02fLli1OrUaNGoG2FhSohRNq48D+PLXA8+3btzs1e1u1Ma29ntbP/v5pIXdauPakSZMC7b/85S9OHxSeM888M9DWQjG//PJLp5acnOzU7PlOm1fsMDkRkblz5wbau3fvdvpogZ7Z2dlOzf4eNW3a1Omzbt06pwYAwK/hc8yknS/27NnTqX333XdOrWHDhoG2dn5SuXJlp2Yf99vHZyIia9eudWotW7Z0amlpaYG2FmqsvQ/a+QlKFu380Odzmz9/vlOzQ4Fr167t9NHOiW+66SanZgcaa+fg9nmzJjMz06lp51Xa+2CfR2nXgkojn1D6IUOGODXt/bavq2lj58iRI05NO763zwW1+fCVV15xavYcaT+PiMjzzz/v1OxrhCLuONDOabTz2NGjRwfaL774otNn3759Tg2lg7Z/s79HPt+rkuy9994LtL/66qvQ/x86dEgeffTRsM/BX0wAAAAAAAAAAICo4cYEAAAAAAAAAACIGm5MAAAAAAAAAACAqOHGBAAAAAAAAAAAiJpSH37tExSyf/9+r+c6cOCAU7MDwT777DOnj0/wpxZ+owXCaj+PFmKMopGSkhJoa2FWWgiTHaqlfY5aTQsYjouLC7S1cCWN/Tjf19OCpuxgJq2PFiRmB9kSfu3nySefDLQHDBjg9NECgO0gLC0MTJs/tM/THhtaUJMWym3PY1oonPZc2pxob7+2nfY4F9HD6VB07CBO7TPZuXOnU9PC01evXh1o20F4BT2XPVaaN2/u9Xo5OTlOzR6z9jwmQvg1AKDwaeeCtiZNmjg17fxEC5XOy8sLtFNTU50+2nlsbm5uoL1gwQKnz9GjR51a27ZtnZodLqttJ0HXpZPP+O3evbtTW7FihVPLzs4OtLXjrpo1azq1d955J+w2+NqxY0fYPtrPXL16dae2a9euQLtZs2aRb1gJV6NGjUDbDrwXEVm8eLFTsz9P7RqLNtdp1+weeOCBsNuwZcsWp7Zq1apA++uvv3b6aOes2nbZNe2ceO/evU7NPq/q2rWr0+eDDz5waigdfOZJ7VrJ2LFjnZp2LW/69OmRbZjHNmjb3qhRI6c2cODAQHv8+PEnfA719b16AQAAAAAAAAAAFAJuTAAAAAAAAAAAgKjhxgQAAAAAAAAAAIiaUp8xoa2FZa9TmZ6e7vSx1/0T0dfkT0hICLsN7733XtjH2WvHiejr6GlradvrzWtrzPmu3YUTq1+/ftg+2jr7Nm2t/0OHDjk1bX1CO7tBywjQatpz2bTvi73+rIhItWrVAm1tjURtjbvGjRsH2tqatHDZa5qedtppTh9t/cBRo0YF2lrWgjbPaPORzxyirQFsj0VtjPmy1xrNyMhw+vzzn/90as8//3zEr4mTZ68rra1PrWWiPPvss07NnhftnB8RfT+clZUVaGvZFBptLrPHnU92FQAAReGTTz4JtLV19Tdu3OjU2rRp49TsPAdtDX07T1FE5Lvvvgu0k5OTnT52doSIu167iMi0adMC7YceesjpM2XKFKeGks/neMle+19EpHbt2k7NzpjYs2eP00fLSNHGnH0eruWNaudC9jmMli2gnf9q36FNmzY5tbLK/jy1c0Ht2ohNG0/aZ6dl29k5H9o5sZ2hKeKe/2rnw77X2ex+2s+sXUeyr/3YmR0o3bRxYJ9L9+rVy+nz4osvej3/q6++GmgvX77c6aONYTtz5fzzz3f6aN8j+zqhiMiwYcMC7d27d6vbeiL8xQQAAAAAAAAAAIgabkwAAAAAAAAAAICo4cYEAAAAAAAAAACIGm5MAAAAAAAAAACAqCn14dc+tOAiLTQnLS0t7HNpoTxaiI0dqKQF0GohJFqAcd26dcNuFwrHqaeeGraPFvSak5MTaGufrTbmtPFkB2hpodlaqJQWiG3Ttl0Lv7bH3NatWyN6HArP6NGjw9a2b9/u9NHGnTamfMadNj/ZNS0gW9uGpKQkp3bLLbcE2k888YTTB8XPDmlr0aKF00cLAqxTp45TW7duXaBtB3WK6CFwdpCYNrdpwVv79u1zal26dAm0V69e7fQBAI09H4rox3Z2v65duzp9vvrqq0LbLi3YVduv+/DZ9kGDBjm1v/71r07NnoOPP541xniF6ZYl06dPd2rNmzcPtHfu3On00YKBU1JSnFp6enqgrR3Pa4HY9jFaq1atnD5r1qxxatr+OisrK9C+7777nD72flhE5Prrr3dqKBraPGaL9LuphRfbQdciIo0bNw60tbGknYdo5xj2uUhCQoLTRzuX1mo+tPN+7XyorKpVq1agre1rtM/Avj7mExKsPU7EHcPaeYj2mfgEW0caiK1dI/TZrvK2HyzrfOaVtm3bOjVtHOzatcup2d8b7XqmNk+2a9cu0NaOBbRjSW1/sWDBAqd2sviLCQAAAAAAAAAAEDXcmAAAAAAAAAAAAFHDjQkAAAAAAAAAABA13JgAAAAAAAAAAABRU+rDr7VAHNu8efOc2qRJk5yaT3imFlijBZP4bJcWhKKF5Lz++uthn8s3LAgnZoc3abQxYIfLaQEwviFb9njSQq21mh2c5BuurgXYtGzZMtDWQpW1bU9NTXVqOHnaZ6LV7M/4qaeecvo89NBDTm3z5s1OzZ4vfIKuRdwAMi2QrHr16k5ty5YtTi3SsGv7vSE0rGjZQVjaXJOcnOzUtPBMe0xpgZ5aYJ79/FrQnhZ+XbNmTadmB43Onj3b6QMAvkHXmu7duwfa/fr1c/p07tzZqb366quBth0eXRCfoGttbu3bt69T69SpU6CtBdfm5OQ4NS2AcfHixYH28e9fedx3a8dHy5cvD7S1caedL/74449hn//w4cNOH+28xh4/CxcudPpo40cL6j5w4ECgnZub6/SpX7++U0P0+Hz3tHMALUzYHnMXXHCB0+eTTz4J+1z2vCOiz39aOKtP2KxPYLXW5+DBg05NO//96quvwm5DWWFfQ9P2Edq1C3tMae+tT8i0iHv+qY1p7bns81/f6zXadTZ7LGpjMzEx0anZocNcTylbtP2u7fLLL3dq2vyqXfO197FaQLZ2HGGPa22ca6HZ2ndZe82TxV9MAAAAAAAAAACAqOHGBAAAAAAAAAAAiBpuTAAAAAAAAAAAgKgp9RkT2vpxPuuNL1q0yKm1adOmUF5PxF2b0zcnoFmzZk5Nyyvw2S6cPJ+MCW2tNbvmux6iT8aEtpac9jh7bUVtXTptPURtbVl77VptG7T1i33eP4QX6ff56aefdmp/+tOfnJpPXoU2LnzWSNTWoNXW0/zb3/4W9rm0ca59t5j/ipf2OWkZE126dHFq9udZt25dp4827ux+2njV1rrW1nu1cye0nBSUHdr8Z/NZC9le0/Vk2M+l7a9R8vhmTGhz4hlnnBFor1q1yumjjbuXXnop0H722WedPu+9955Ts7PPRERatGgRaGvnHNoa33v37g20Y2NjnT7a90HLs7KVp/23lrmRlJTk1Oz3Utsnbdu2zalpx1qbNm0K+3rafnf16tWBtnbsVa1aNaemZTvZ5xTaPl07f2jdunWgrWVoIHp89p0iIqNGjQrbRztGtMdJVlaW00ebI7VzVN/zcJvPfl7LhdGOLbU5uKyy54K8vDynj3acbte0z833XNDu5zteffJINNrPU6NGjUBbGxdaxo491sncKb18jxPtHBHfTB1tDrSvyWn7Zo29Xdq2a2P415z/nAh/MQEAAAAAAAAAAKKGGxMAAAAAAAAAACBquDEBAAAAAAAAAACihhsTAAAAAAAAAAAgakp9+HWktDARLRjYDgXRAnJ8Qk60QFjt9bTt2rVrl1ML93qIjB0KpwUUaaFXdnCcFvqUkJDg1HxDE30eZ49DLQxs//79Ts0OahJxgw3j4+OdPlu3bg37XNq4J+QzMj7fcW28agHAWihSdnZ2oK2FE/qMTd/g4Ndffz1sn0jD61C0WrZsGWjboVsiIp988olT69atm1Ozx5QWJqeNOzugNScnx+mTkpLi1N544w2nduaZZwbaTZs2dfosXLjQqSF6fEIMfceOT2ibNv/Zpk6d6tRmzZrl1N58802nxn6wdPLdJ2n9Zs+eHWh37drV6WOHTIu4+8oBAwY4fTp27OjUMjMzndrGjRsDbe04zg7IFnHneG1Mr1271qlp7O9kedrPDx482Klpx0z2+aF2DK7Nidq8Ze8HtfMTbf9pny/UqVPH6aMFpWv7XXtbd+zY4fTRwl7tYwbCr4uXdt1C87vf/S7QXr9+vdfj7HNP7ZzVN+jaPmfSzqG0mv191I4rtO+sNnefd955Tq2ssgNyteBybc6y5zZtLrLHhYg+90TKHj/aGNNoocB2TRtj2rb7vGZRvw8oHL7XZN99992wfbTrOomJiU4t0nMKe07XxrQ232n9CgN/MQEAAAAAAAAAAKKGGxMAAAAAAAAAACBquDEBAAAAAAAAAACihhsTAAAAAAAAAAAgaspk+LVP6IjWRwtPskNBtGAxLRjJpgWHaKGMWtCTHRZE0HXRsUOYfD4PETfk2TckXavZj/V5PRF3rGrjpFKlSk5NCwJ97733Au0OHTo4fb7//nunZmvWrJlTW7lyZdjHwaWNKXvOqlWrltNHCyLcsGGDU7PHi8/ribhzmxaIpIXCvfjii06tS5cuTs2mfR+YE6MrKSkp0LZD70VEnn32WafWr18/p2Z/ntoY0/a7diCYFpinPZc2/9hjtmrVqk4fFA3f77PPd1wLYY/URRdd5NS6d+8eaC9ZssTpk56e7tQmTpzo1Hbt2hVoT58+3Wu77LGqBSZqYYiEbRe/7OzsQHv48OFOn+eff96p2YGX2rirXr26U9OCgu25TRs/2lhct26dU4tUeQq7tvXs2dOpaaHP9rmHPV+I6N9pLdjaDkHXAlS1c4qMjIxAWws+1s5/tbBO+zi0cePGTh/t2PGMM84ItLXvB4qGdr6oXcto2LChU2vdunWgvWrVKqePFuhuj/u4uDinj+/84XPMoJ3n2N8F7RxZ+75oge7NmzcPuw1lhT1etOMQ7TO3g3y3bNni9NH2U1pNO1fwYY8D32s4PtditPlWq9nnMNr8rn0nCb+OjM+11cK83jB06FCnZu/fNm/e7PSpWbOmU9P2xbZIt13bD2vPRfg1AAAAAAAAAAAo9bgxAQAAAAAAAAAAooYbEwAAAAAAAAAAIGq4MQEAAAAAAAAAAKKmVIVfF2YIiRYcYgd6iriBY76h2XYojxYYpQXbaIE7PrT3BifPfv99gs1FRHbu3Bm2jxbU5BtGbdO2yye8ySfgXcQN4Ondu7fX4+zvR6NGjZw+hF9Hxmeue+ihh5yaFkSofXZ2yJw2frQ5yw4z08aYFn7drl07pzZr1qxAWwugRXTZwXQabf+2evVqp2YHG2qP1caPNhbt+VR73L59+5yaFhTnM+dqgdjaz4MTs8MCIw1lTktLc2qPP/64Uxs2bJhT27BhQ6C9detWp89///tfp/bwww8H2llZWU6fWrVqObXzzz/fqXXo0CHQ7tq1q9Pn5ptvdmp2wHdhBn6Xd5EeR2v7Zp/jL+317IB1EZGlS5cG2lpwsDYWR48e7dRuvfXWQNsO5EbR0vYj2r7Snt+aNGni9Nm9e7dTa9CgQdhtsMOwRfTzk2bNmgXaWhCxFoCZkJDg1Ox53w56FXHnZRE3RBlFx56PtOM6zbvvvuvUfvrpp0BbOz7T5j/73NY36Frjc4yo1ezjQe2YsXr16k7NJyDW3s8fPXpUFi9e7DyuNLLfb+190+YZey7Qroto56za+23vd32vEdqP055bq2nHr/bPmJeX5/TRruH4jHVt3GlzKU6eFmTu8/mKuONTO/6bMWOGU7PPUbV9rO8Ytvv5Hpf6PM4nbFtEJCUlJdD2uTZq4y8mAAAAAAAAAABA1HBjAgAAAAAAAAAARA03JgAAAAAAAAAAQNSUqoyJwqStb27nSYiI1KxZM9DW1szT1uOya9q6Ydo6n9r6X/Ya39q62b9mHcbySls73X4ftXUAfXNGfB6nrWlnr68Y6ev5rpesPb/93mjr4mpry9q0tSIRXqR5OgMGDHBq2hrkvtkpPuw5S3s9bV7T1sT2+RkjzRVCZOrXr+/U7DXOtawFbd1sn5wdLe/BJxtHe5zvOLf3qdoa7lrtxx9/9Hr+0q4w196312ytU6eO06dv375OzR6Hbdu2dfpox3WvvvqqU5szZ06gfcUVVzh9tH2zNmfZtDydV155xan9/e9/D7S140Etm6Jhw4aBdvv27Z0+2ndWW7/9zTffDLQXLFjg9CmNIl1H12dNXm0u6tKli1Pr1KmTU7PH/syZM50+1157rVOz56dt27Y5fbTMFW1fPGTIkED75Zdfdvpo7HGnHRNqOX3a9zs1NTXQXrRoUej/f/75Z/n222+9tqk0Ovvss53aDz/84NTs4+uFCxd6PX+kmT12noSIyNq1awNt7XulndsePnzYqaWnp5+wLaLvw7WcOhQN+/PVxtLw4cOdWseOHZ2affynnW9r85M9B2vboB2PaPtre67W5m7t+e3t0l5Py+bR8vxsdm7ewYMHy0zGhH0srx3vazU7d1Lbj2h88l192eNO+8y159aOGWJjYwPtTZs2OX20/aJ9vqI9d6Q/H1w+c43GJ2/BN0vVzvHRMkS07fLJ7In0GqB2bU/LMalRo4ZTu+222wLtSZMmeW3D8fiLCQAAAAAAAAAAEDXcmAAAAAAAAAAAAFHDjQkAAAAAAAAAABA13JgAAAAAAAAAAABRU6pSVHyDYOxgkjZt2jh9mjRp4tT27NkT0TZogWC2+Ph4p6aFbWvPNXDgwEBbC1LEydPCTO1AGS3YTQv9s4Ml7fAjEX3saOE0dvCMHY7jSxtLviE6dtBNRkaG06du3bpOzQ7/0gLPEF6k4df16tVzaj7zmog7b/qEm4u440cLPNfCorTgpOeeey7s62nj2icMHpFJTk52anbw8JQpU5w+rVu3dmp2iKuIG1p48OBBp48WYp2Xlxdoa2NMm6s1y5cvD7TPOussp482p5dVFSpUCMxBkQaqaoYOHRpoX3jhhU6flJQUp2aPkyeffNLpM378eK9taNGiRaCtHQ9eeeWVTs0Oi7bDYUX85+7rr78+0H766aedPmeccYZTa9q0aaA9duxYp4/2ffz666+dmu++obSJdH/gs2/R5iIt6Fqb6+zzlRUrVjh97r77bqf20UcfBdpXX32100eb/1atWuXU7O/WiBEjnD5aeLr9/Np+vlq1ak5Ne7/sbdi9e3fo/48cOVKmwq/t91KbM7Tj65o1awba9vdeRCQhIcGpaYHCNm1sap9Tq1atAm3tmFDbBu18d//+/YG2FuCuhV9nZmYG2j7n/IiMz35e299s2bIl7OO04zptvvUJcNWCrjX2d0E7l/a5rqS9nvbzpKamht0m+3haG/OlgXZsbb9P2vumzSFbt24NtLXzWO391saGXdPGmM98oc0z2utp3xn7utGOHTucPlrIsf2a2tjU5mkUvwceeCDQbtasmdNHGwf2eaU2H2ifuTY2fALcNT7zqXb+q82n9rEp4dcAAAAAAAAAAKBE48YEAAAAAAAAAACIGm5MAAAAAAAAAACAqOHGBAAAAAAAAAAAiJpSFX6tBXn4hNj079/fqe3cudOpaUG9dvCWT1iTiBtiowUFaeFxWiC2T/i1Tyg3AbFBtWvXdmqHDx8OtLVgNy28yQ5y8w0OLswgN3s8aeNLGwNasI4d6KS9D1r49caNG8NuAwqPPc60EDrtM/AJc9PGpvY4e+7RQpK059LCzOzgpNmzZzt9mMeiKykpyanZ3/3XX3/d6XPrrbc6tR9//NGp2fspLbxT239q+2Jbbm6uU+vSpYtTe/fddwPtYcOGOX20ObCsKqzvmBaAft999wXan3zyidNHC2p+9tlnC2WbRNxQYC2Qds6cOU7twQcfDLRHjRrl9NGOxTQNGjQItK+55hqnj/Y+nH/++YH2xIkTnT7ad3bz5s1Ozf6u2ceyvj9LaeBzPKzV0tPTA+3bbrvN6XPXXXc5tb179zq1SEPkL7/88kBbC6x+5513nJp2LGkHE//ud79z+nz55ZdOzT5n0s6XtJ8vLy/PqdnnOceHsJe1IGM75FQLhvYZK9q+7Keffgr7OBG/8xFtu9asWRNoa4Gt2jjQttXeF9jzn4h+/Go/zv4+ioisXLnSqeHXO+OMM5za4MGDnZo2Du0xp507+MzB9jm5iH6MqF2Hscer9j3Q5hu7pj13jRo1nJrPcVNKSkqgbYfClxbaMbnP+6adC2ZkZATavmPF5/hEe5y2DXY/39B1bb61z4Ht64gi+udun2NEOpejaHXs2NGp/fGPfwy0tX2gdq3NrmnznTaGfYLfIw2/1r5X2vddm4cL4zyZv5gAAAAAAAAAAABRw40JAAAAAAAAAAAQNdyYAAAAAAAAAAAAUVOqMiYi1bZtW6emrd0WFxfn1HwyJrQ1wew15qpVqxZuM0VEX3eucePGgXbLli2dPqyxefK0z+TIkSOBtrYmXM2aNZ2a/blp6+z7rpFoj0NtnThtnPisL+ebMWGvebtp0yanj7aO9YYNGwJtbS1HhOe7NqC91r82h2lZDtpamfaY9V2D267Z3yHtuQt6rhYtWjg1FK/k5GSntnXr1kBbWyP7rLPOcmq7d+92avba1tr8qs0j9ljX1sDUxrl2PPDiiy8G2to+XZvvyqru3bsH3vNmzZoF/l3L6dDWau7Ro4dT27dvX6Ct7cu0z9vOn7GfR0Tk22+/dWpafoTt/fffd2raGHjggQcC7ZEjRzp93nzzTaem9bv77rsDbS0L7bvvvnNq27dvP+E2iYgsXbrUqWmfj/1c2pqxZYXP+t9paWlObcaMGYH2N9984/TR5rXCtHjx4kC7VatWTh/7OyoiUqdOHac2a9asQFubN7V1rO1jVZ9jUO1xIuXruNBe516bV7TsBvu91I7ttHXutWMtO2dMy/3QPqdGjRo5NZu9Zr6IfsxpZ5Ro25mVleXU7PwC7b0qrY7/jKOd5+OzXv748eOdPtra6dqYtr/jWp8DBw44NXsfpI0T7RjRJ/9T66NdC7LPYXyzBbT5z+5n7yu096A08MmY0PYj2ntkz0c+56ci+vGKz37eJz9C+z5qNW1b7bGv7Su1OTI1NTXs65Wl+S8SPvkj2vGFdl3Ch3Z9TDvPsGn7WO2aoz2GtZ9Pm7e0ecMnJ0Wbt3wep71/2jbYGROdOnUKvPayZcucx9j4iwkAAAAAAAAAABA13JgAAAAAAAAAAABRw40JAAAAAAAAAAAQNdyYAAAAAAAAAAAAUVMmE8jq168faGuBdnZ4p4hIw4YNndrGjRsDbd8wETuUTAsK0h6nhZzs2LEj0O7Xr5/TRwu/9gkBKs+0z8SmBdH4hMD4BrFqgZR2yKoWgOQTmKWNVd9QZTuobNu2bU6f9u3bOzV7W6Md6FZW+H5OdpiS7xjT5gY7XFHro419O2hK+8y159K2y567fdnvF+Ou8GgBmB988EGgrQVg2iFYIvp+yp7vtIAwLUjMps25Wuiczxizfz4RkXr16oV9XFnRtGnTQOjbeeedF/j3L774wnmM9l7bYX4i7udbt25dp48WUG4HuGrf8eHDhzu1DRs2OLUtW7YE2qtXr3b6aGNu4sSJgfaZZ57p9Bk1apRT044t//GPfwTaHTp0cPpoc+R9990XaNvvi4g+VrXj4FNPPTXQ/vHHHwPtI0eOyNtvv+08LloKc163QwwTExOdPsnJyU7Nnle0MV2YfMaUNkdqY9gOWBdx9+FaqLIWbm+PM22fro3FX3McWhbY40x737Tj+YyMjEBbOzfUwq81dmCxNq9on4ndLz4+3unj+520t7VWrVpOH+34wA7Q1fbfWiB9aXD8e2d/vtrn7RvCbH9uvtctbD169HBqWmivdqxnhzxnZ2c7fbRAY/v7ovXxfR/smhbArX0f7X2FNq9pQe12wLuIe1z073//O9CONJS3uGmhwPZcoL1vWmC1/ThtjGmBxtpz2XzHis885hOaLeJ+/7Rxp83B9r5YC5rXvg+lUaTB9D7zVqTfKe24Z9GiRV6P3b59e6CtHaNp49oeP75jU5u3fMawFp5ujzuf91jE7zju+ONnbcxr+IsJAAAAAAAAAAAQNdyYAAAAAAAAAAAAUcONCQAAAAAAAAAAEDXcmAAAAAAAAAAAAFFTqsKvtaANLezj9NNPD9tHC3zTQmXsgBE7iEvEL8RGez1tu7RAITs45+yzz3b6PP30004NJ+YTIqSFwGifkR22o4U+acE32nPZoTk5OTlOH5+QGy1MSNsuLZjJfqwdFioi0rVrV6cWaYgOIqOFztm0kCRtbNjjUwv10h5n0wLJtKDDvXv3OrVIg0UJvy4cWsBc48aNndqTTz4ZaJ9xxhlOHy0MWZsDMzMzA22fMSbizt9agKg2DrSQbDss+LvvvnP6DBw40KnZ26p910qjqVOnBto1a9YMtLX3Ys+ePU5t3759Tu3rr78OtLUgbW0c2p+39jlqx1kNGjRwanZYuxa23bNnT6f26quvBtraHPbYY485NTvIVsR9v7TjyAkTJjg1+xjU/v6I6Pt0n++eHd6pPU80FeY8bodKa+GE//nPf5zaW2+9FWhfdNFFhbZNzZs3d2p20LyISJMmTQLtP/zhD04fO7jcV+3atZ2adq5ljx/t+6cd70Ua0lhW2PsIbdxpwfQNGzYMtO1wTRF3LhVx5zYR93usncfWrVvXqe3YsSPQ1j5fbX+t7YvtMayd12jh8/ac7hv4XdrY3xPte+MbHmrz/b49/vjjgXa7du2cPkuXLnVqGzdudGr2MYP22WrzjM/xn29ArB3ArX33NLt27Qq0tWNZbfxqn48d1m4fW5bWY0af62Xa8Zh2vGLTrlNon7l2jmpvgzaetHHn8/3Tjkt9zq+1Ptr7YF8P0o7Zykr4tf19LcxrRdo567nnnuvUxowZE2j37t3b6aN9Btpxvx0qrV0H0car/T74jE0RfRzY+0bt+F17Lvu917bdN/jdtnbt2hO+toa/mAAAAAAAAAAAAFHDjQkAAAAAAAAAABA13JgAAAAAAAAAAABRw40JAAAAAAAAAAAQNaUq/No3OOO0004LtO0gIxE9QEsL8rDDjLQwHy20xQ4X0x6n/Tw+ocmtW7d2+uDkaQFw9mepBWpp48SuaYFL2jjRAmXs8Do7kFIk8lBKbdsjDSDzCSgtrcFepYUd5uYTRiTi9/lqY1/7zO3xo40n30AnO2Rdmze1YEUUDjuwUEQkPj7eqdkBflpIrxYOqAV22WPxyJEjTh9t3NlzoDbGtMdpAYX2z71y5Uqnz8UXX+zU7MCz7Oxsp09ZYIdi2m0Rkcsvv9ypDRgwwKldddVVgfYf//hHp48299jHQVp4oBYybQcKirhzjzYmtNC7Dh06BNpz5sxx+vzzn/90aps3b3Zq9vfj7LPPdvrY4bMiIsuXLw+0tdDx3Nxcp7Zp0yanZo/zrKwsp09ZsX79+kD7yiuvdPpo7+W3334baA8aNMjr9RITE51a586dA+3+/ft7PW78+PGBthYqHykt6FDbN9vHFtp3Rjuu0L7L9mOPf72ydsxYp06dQFubV+bOnevU7P1bgwYNnD59+vRxatp8Z48X30By+1xaO2/WQtC1YzT7NbU+dlixiPvz+By7lgVaWLT2ndPCzu3jEu2awejRo8M+7tFHH3X6aMdZ2r7Y3lf6jh37vFw7b9bCqLXzeXu+1fbDS5YscWr2d087D9G2XXtv7ONZex9bWuc7n7leO/72Cb/WHqfNT77nmj597J/Hd57xuUaknfdoc539OO18XhvnZUHbtm2d2m9/+1undv755zs1Lew6Etr5gzb/+JxX+l6L8bl+oo0xbd6wj/u1bbCvGYm4P482t2mvp41F+/hGO4cJp3zs4QEAAAAAAAAAQInAjQkAAAAAAAAAABA13JgAAAAAAAAAAABRU6oyJnw1b9480NbW29XWfIt0bTqtZq9JqGVHaLT1Qe11q7V1rOvXr+/U7PXaEaR9Jj7r6CYlJTk1ez1CbV067bk0tWvXDrS19ew09jj0yUwpqJ/Pz+OTrVJa188sLez399esv+szfnzmSO17peUGaM9vj3Xf8eOzXQhPWwdYm3/sNX7T09OdPto66Nq+WFvP0qaNKXu7tLXZfdaz1R5rr+Nf0DbY63WW1YwJH1OmTPGqQff2228X9yYUuy5dugS+1/fff3/g35cuXeo8RjvO3bhxo1Oz1w3XMj3q1q3r1Ozv/UsvveT0GTp0qFPTnt/OCVi9erXT55prrnFqNm0/H+mxljb/ajX7fdDWXddo22rPm8evba8dK5Rm9vtmZ06IiAwcONCpNWvWLNBeu3at02fRokVe22B/nlqWVKNGjcI+v3Z+qh3HaZ9hu3btwr6elnHTsGHDQPvUU091+pRGsbGxgePWTz75JPDv2hr0a9ascWracUlqamrYPtp4mj59eqC9cOFCp4+WV6HlSzVt2jTQ1tY21+YZO/NGyyLRrndo89EVV1wRaH/44YdOn1dffdWp2bRMR99zYjvXQssxKo20/Y19fUEbd1r2nE2bZ3zX3tdqPuzHRZqxKOK+N9o437t3r1Oz95W+uRql0VtvvRVoa3l0PjmwIu55nrb/8bkmp81RvtdB7OfX+vjsK7Xt1L4P9nVCX9o5sZ1jq8252rm7lj/56aefRrRdx+MvJgAAAAAAAAAAQNRwYwIAAAAAAAAAAEQNNyYAAAAAAAAAAEDUcGMCAAAAAAAAAABETakPv05ISHBqdijItm3bnD4tWrRwaj6hMlqQmxZQY4drayFA2uO058/IyAi0tSCds846y6nZQVYI0oJobNpn5BuoatOCfLRQGztYVuMTrOMb2qP1s7fLDiQT0cOv7BDZshZkWNLYYVza/OFbizQ0zB4HWnCSFqrsEy6mhY1pYxGFQwts9QmbbNy4sVPLy8tzatrnaQcL+ga72nOUHWQmos9tWriYT5BYx44dnZodBLl9+/awzwNA98033wTa06ZNC7STkpKcx2iBhX369HFq9hwyd+5cp482P9m0fZkdVCyiB8c+8cQTgfZPP/0U9vU0kQZda+bPn+/UtPc5Ozs70Nbmd412DGjvw48P9dXm59LMPvezgyZFRL7++munZh9HpaSkOH208z5tDNv7yh07djh9tP3naaedFmjbgcYi+nGcdg5jhzlv2LDB6aMFA9vHB/Y4LK1uuummwDmh/d7a51Ii+nHWvn37nJp9HOIbkn7OOecE2sOGDXP6tG3b1qmlp6c7NTuMWpuztM/SHqstW7Z0+nz00UdObejQoU7N51xB23/Y26WFX2vfF23fsHjx4rDbUBpp84w9d2tjWJt7bD7hwgX186GdY/jsd7THaezn0q79aOyfx/f1SoP7778/cF10+PDhgX/XvmPa/KDNI/b7po0L+5qsiHt+ePDgwbB9CmLvB7X5QXt+e86oXr261+tp11k+/vjjQHvlypVOH+1YtU6dOoG29llo256amurV72SVnVEPAAAAAAAAAABKPG5MAAAAAAAAAACAqOHGBAAAAAAAAAAAiBpuTAAAAAAAAAAAgKgp9eHXp59+ulOzw0q0EBItlEfjE4ijhbHYgUq+4dfac9k1rU/37t2dGuHXJ6aF2viECWuBWvZnqX1G2hjQavbY1PpoQT65ublOzedxdiidiLv92s+s1ewQWd+ARAQVZqiXVtPGfqRB5fZY0bbdDsIT8Qu+0x7HmCo6dpiziPuZaOrWrevUtFBGLRjLnnO1eUXbh9u0EENtTtTmU5/AMS3Q0z7++OSTT8I+DwA/L7/8cnFvQplnB44XVENk7HNIbT+l7T/tfbG2/1mxYoVT0/af9j48Pj7e6aMF9x4fSi6iB9dqxwfatnbo0MGp+Tyuffv2gbYWFF4arVmzJnAs8uGHHwb+vV69es5jEhISnFrNmjWdWnJycqDdqVMnp492jmqPQ+34Wzu+184d7CBz7bPVtt2u3XjjjU6fZ555xqn50Maqzzm+dh1IO7Zs2LChU/viiy9OZhNLDZ/zAu1Y2w5m932cNl61a2g+tDFsP39hBk/XqlXLqWnvg31uG+l1gJLonnvuCbQXLlwYaF9zzTXOY1q0aOHU7GtMIu485Rs8bX/GWki5Nrdpn4v9WO25fL4zL730klN74YUXnNpnn30W9rmuu+46p3bJJZc4tYyMjEBbG/vavkALyc7Kygq7XeHwFxMAAAAAAAAAACBquDEBAAAAAAAAAACihhsTAAAAAAAAAAAgakp9xkTv3r2dWmZmZqCtrZelrdWorWFnryXmu4a7vb6YvY629twF1ezn19Y869Kli1PDifl83trnpq3haq+rpo0vbc05bZ19ew09rY+2hl7VqlUDbW0tW23dd+1nzMnJCbS1NWm19eXsbbWfB358187U1k20aXOK9vw+Y197nD0/2eNQRM+08FGY63wivMcff9ypPfnkk2Efp31O2hx14MABp2avu+mTs6TRxrm2brb2ndHmRdsFF1zg9VwAAIi4azNrx8TavvLHH38MtBs3buz00TLjmjRp4tTsc5a9e/c6fbTjeftcuk6dOk4fbf107bl27twZaKekpDh9tPdm06ZNgbaWiVcazZw5M9B+++23A+1zzz3XeUxaWppT066B2Mc92meknR+uXbs20NaO97XzAp912OvXr+/02bhxo1Pr169foP3DDz84fTTadtnfK22Ndy3PwN52rY/2XdCOg8tq7pj2nti0ec0nt9XnPFNEP6+0n197Lp9t8MmhKGi7bNr6/Bo7h0W7ZqRdfyqN3nrrrRO2C9KzZ0+nZs8Zffv2dfq0adPGqWkZN4Xlq6++cmr//Oc/ndqUKVOKbBu0fBstB0er2XyuGYkUzvjkqg8AAAAAAAAAAIgabkwAAAAAAAAAAICo4cYEAAAAAAAAAACIGm5MAAAAAAAAAACAqCn14dctWrRwajt27Ai0tSBiLawp0oBWLQDEDgbSAnK0x2nsUCef4B6EpwVh2bTAFy30yg4g00K9tKC677//3qnNnj070E5PT3f6rFixwqnZwdPaNmjhs3bgkvaazZo1c/rk5eU5Nfv98gnIQuTsecw3NExjz0faPKOFcdnjOjc3N+x2FrRddliaFqKIorN9+/aIHqeFR2vBYnv27HFqWli6TZu37LGoBcxp848WYKeNWZsWGAoAQEHsYEntuCc+Pt6pJSUlBdpa6LO2f9P2Zfa+UXs9Lai2efPmgXbt2rW9HteoUSOnZu8/ExMTnT5agLHdTzsGLQvs4+0PP/zQ63E+4ala+LX92YqI1K1bN9BOTU11+mjno9q1E7umhVg/88wz7sZGSBuHNp/zF62mhbJrx60ff/yxU9MeWxb4vG/atQWNPT9p84A2xrRjee2xtkjPk33CtkVEDh48GGinpKSEfW4R9zxKu9ZkP3dpZb/f2mepmT9/ftjavffeG9E2adcEfcedXdOueUSqWrVqTk27/mZbtmyZU9P2/fa8qJ1va9erte+R73f+RPiLCQAAAAAAAAAAEDXcmAAAAAAAAAAAAFHDjQkAAAAAAAAAABA13JgAAAAAAAAAAABRU6rCr+0QMRE9/CsjIyPQbtWqldNHC8jJzMx0ar6BLDY7PEQLZtJqhw4dcmp26IgWCKuFW9lBVpEGm5ZV27Ztc2qnn356oK2FD3366adO7aabbgq0FyxY4PTRxqq2Dfbn6xtEs379+kC7fv36Tp/k5GSn1qRJk7D9nn32WafPVVdd5dTsQB4tABfhaaFCmh07dgTa2tygzWHa80caNG2Hf2lhb1p4kzaG7aAp3+BuFA7t/dYCwewAtnfeecfpo80PGns/qIW7aXOgvb/Wwvg02vdBC++2accM9nNFerwAACh77PMuOwRaRA/KtGt2QKyIvt/SwljtQErtXETbf9r7WC3Y0jcw1D6Xso9dC9ou+7jUJ+QYQdqY++qrr4phS4rXnj17nFqLFi2KYUtKP+1agh0I7hv8bQfca3Od9r3Xzlfs+U+bi7TzX/vYXeujza3a89tzqfbzaLT5z1a9enWv5yrpSuK5UqTXQIqaT9C1r9zcXK9aceKqDwAAAAAAAAAAiBpuTAAAAAAAAAAAgKjhxgQAAAAAAAAAAIgabkwAAAAAAAAAAICoKVXh15deeqlTS0pKcmp2yJYWblSvXj2npoV/1ahRI9DWwna0x9WuXfuEzyMi0rJlS6eWmprq1OxAbDtgqKBar169Au3XX3/d6VOeLVy40KkNHz480G7atKnT591333Vqu3btCrS7devm9HnmmWecWvPmzZ1agwYNAu2EhASnjxYq3ahRo0BbG/dr1qxxavPmzXNqDz30kFOz3XjjjU7NHqtaMBTC8wn1EnHfby2I0A4WExGpVatW5BsXAS1USgv6smu+48c3LBwnpgWS+QSCzZgxw6ndeuutXs9v0wIKtf2bHaiuBYhqQdpakNh3330XdrsI3QQAnAx7n9e1a1enzzfffOPUqlSpEmjbQdQiIs2aNXNqFSu6p/UZGRmBtrYv00JV7fMfLcRVO+bUjlWXLVsWaGvbrp3P2/3Wr1/v9AEQXVr4tT1nacffGnt+OnDggNNHu2anhbrb587aXKSde/qcm2jX+uyfWcSdg7/44ouwzy3izq/afKidCwFlCX8xAQAAAAAAAAAAooYbEwAAAAAAAAAAIGq4MQEAAAAAAAAAAKKmVGVMzJw506nt3LnTqcXHxwfa33//vdNHW2NTW3s6PT090G7fvr3TJysry6k9++yzgba2zp227py2Xmdubm6gvWTJEqdPw4YNndq6deucGv5n1apVTm3IkCGBtpYx4fO+2msmioiMGDHCa7vsTIm6des6ferXrx/2NX/44Qenj7YOe6QGDBjg1Ozvo8/69HD5rme/devWQLt///5OH23tYK1m55bY86iI33rC2rqf2nOlpKQ4tWnTpgXau3fvdvpoyJgoOj5rr27fvt2pLVq0yOu57LGhrZGtfR/sfd5PP/3k9ThtrW4AAArb//3f/wXa2prkdj6ciHue9/zzzxfuhhWze+65x6mlpaU5tY8//jjQfvPNN4tsmwD4efrpp52afeyuZSVq7OsS//jHP5w+HTt2dGpadqudS+ibJ2Gfx2rX7Pbt2+fUNm/e7NTsnDzfrI25c+cG2naOngjXVFD28RcTAAAAAAAAAAAgargxAQAAAAAAAAAAooYbEwAAAAAAAAAAIGoizpgojjW9tTXftPXWDh06FGhr26qtPa09l7023P79+8P20V5T23Ztvf+YmJiw2+qznQW9ZlEq6jFRHGPOZ331wuYzdrTxa/cr6vdLe2+K4zMqi+Mu0tfUxoVW0+Yee17R5hltu+xxpz2uqOesaH9G0Xi90p6boe0rDxw4ELafNi60/aL9OO257WOBgmqlRVmc61DyMe4QbWV1H6vtf7R9V1lfS9z3ffBdn72wMNehOJSFcWe/RqSv6XOeKaKf2/q8ppZ9YT+X7+sV5nUQ+3HacxfmNamyuo9FyRZuTER8YyInJyfSh0Zs7dq1XjUUj5ycHCe4ubCfP9o2bNgQ9dfMzs4+YVtEZM2aNdHanALZwcvFpSyOu0gtXry4uDehXCjqMZf/GqXZ+PHji3sTyhzmOhQHxh2irazuY+0w7PLqkUceKe5NUDHXoTiUhXHnEyDtQ7v4vnTp0oieqzTJyso6YbuwldV9LEq2cOMuxkR4O+vYsWOybds2iY+PV3+bEeWHMUZycnIkLS1NvRNdWBhzOB7jDtEWrTEnwrjD/zDXoTgw7hBt7GNRHJjrUBwYd4g29rEoDr7jLuIbEwAAAAAAAAAAACeL8GsAAAAAAAAAABA13JgAAAAAAAAAAABRw40JAAAAAAAAAAAQNdyYAAAAAAAAAAAAUcONCQAAAAAAAAAAEDXcmAAAAAAAAAAAAFHDjQkAAAAAAAAAABA13JgAAAAAAAAAAABRw40JAAAAAAAAAAAQNdyYAAAAAAAAAAAAUcONCQAAAAAAAAAAEDXcmAAAAAAAAAAAAFHz/wGd3BdmU0IPrwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 2000x500 with 10 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"The train set contains {} images, in {} batches\".format(len(train_loader.dataset), len(train_loader)))\n",
    "print(\"The validation set contains {} images, in {} batches\".format(len(valid_loader.dataset), len(valid_loader)))\n",
    "print(\"The test set contains {} images, in {} batches\".format(len(test_loader.dataset), len(test_loader)))\n",
    "nsamples = 10\n",
    "classes_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal','Shirt', 'Sneaker', 'Bag', 'Ankle boot']\n",
    "imgs_test, labels = next(iter(train_loader))\n",
    "fig = plt.figure(figsize=(20,5))\n",
    "for i in range(nsamples):\n",
    "    ax = plt.subplot(1,nsamples, i+1)\n",
    "    plt.imshow(imgs_test[i, 0, :, :], vmin=0, vmax=1.0, cmap=matplotlib.cm.gray)\n",
    "    ax.set_title(\"{}\".format(classes_names[labels[i]]), fontsize=15)\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"architecture\"></a>\n",
    "\n",
    "### Define the deep autoencoding architecture\n",
    "\n",
    "First, we will define the deep autoencoder architecture as a guideline to what we are aiming to train by creating a custom nn.Module class. The autoencoder will have multiple hidden layers in both the encoder and decoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DeepAutoencoder(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(DeepAutoencoder, self).__init__()\n",
    "        # Encoder\n",
    "        self.encoder = nn.Sequential(\n",
    "            nn.Linear(28 * 28, 128),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(128, 128),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(128, 64),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(64, 64),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(64, 32),\n",
    "            nn.ReLU()\n",
    "        )\n",
    "        # Decoder\n",
    "        self.decoder = nn.Sequential(\n",
    "            nn.Linear(32, 64),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(64, 64),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(64, 128),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(128, 128),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(128, 28 * 28),\n",
    "            nn.Sigmoid()\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.encoder(x)\n",
    "        x = self.decoder(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"pretraining\"></a>\n",
    "### Layer-wise pretraining\n",
    "\n",
    "In this section, we will perform layer-wise pretraining of the deep autoencoder. We will train each layer individually as a shallow autoencoder, with its own encoder and decoder.\n",
    "\n",
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #013220; border-color: #03C03C\">\n",
    "\n",
    "> ### Question 1.1 - Implementing the **layer-wise pretraining**\n",
    "\n",
    "> 1. Complete the `pretrain_layer` function that performs the pretraining of a given layer\n",
    ">    * Define the architecture of a single autoencoder\n",
    ">    * Create a criterion and optimizer\n",
    ">    * **Obtain the encoded features from previously trained layers**\n",
    "> 2. Update the training loop to learn the weights of both layers.\n",
    "\n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = \"cpu\"\n",
    "# Function for layer-wise pretraining\n",
    "def pretrain_layer(pretrained_layers, layer_id, layer_in, layer_out, train_loader, num_epochs=50):\n",
    "    \n",
    "    ######################\n",
    "    # YOUR CODE GOES HERE\n",
    "    ######################\n",
    "    # Define the shallow autoencoder architecture\n",
    "    shallow_autoencoder = ...\n",
    "    # Our training criterion and optimizer\n",
    "    criterion = ...\n",
    "    optimizer = ...\n",
    "    \n",
    "    # Training loop\n",
    "    for epoch in range(num_epochs):\n",
    "        for batch_features, _ in train_loader:\n",
    "            \n",
    "            ######################\n",
    "            # YOUR CODE GOES HERE\n",
    "            ######################\n",
    "            # Eventually pass the batch through our previous layers\n",
    "            batch_encoded = ...\n",
    "            # Training loop for a given layer\n",
    "            ...\n",
    "        # Print the corresponding advances\n",
    "        print(f\"Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}\")\n",
    "\n",
    "    # Return the trained shallow autoencoder\n",
    "    return shallow_autoencoder\n",
    "\n",
    "# Pretrain each layer of the autoencoder\n",
    "pretrained_layers = []\n",
    "input_sizes = [28 * 28, 128, 128, 64, 64]\n",
    "output_sizes = [128, 128, 64, 64, 32]\n",
    "for i in range(len(input_sizes)):\n",
    "    print(f\"Pretraining layer {i+1}\")\n",
    "    pretrained_layer = pretrain_layer(pretrained_layers, i, input_sizes[i], output_sizes[i], train_loader)\n",
    "    pretrained_layers.append(pretrained_layer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"finetuning\"></a>\n",
    "### Fine-tuning the Autoencoder\n",
    "\n",
    "After layer-wise pretraining, we will fine-tune the entire deep autoencoder using the pretrained layers' weights. Note that what we have right now is a set of pretrained layers contained in the `pretrained_layers` array. However, we still need to connect these layers together by **transferring their weights to our full deep autoencoder architecture**.\n",
    "\n",
    "If you check the documentation and code for the `nn.Linear` layer, you can see that the actual weights are stored inside two Tensors respectively named\n",
    "- `layer.weight.data`\n",
    "- `layer.bias.data`\n",
    "\n",
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #013220; border-color: #03C03C\">\n",
    "\n",
    "> ### Question 1.2 - Finetuning the autoencoder\n",
    "\n",
    "> 1. Transfer the weights from the pretrained layers to the full architecture\n",
    "> 2. Fill in the training criterion and optimizer\n",
    "> 2. Finish the training loop to finetune your deep AE\n",
    "  \n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the deep autoencoder with pretrained weights\n",
    "autoencoder = DeepAutoencoder().to(device)\n",
    "# Load pretrained weights into the deep autoencoder\n",
    "for i, layer in enumerate(autoencoder.encoder):\n",
    "    if isinstance(layer, nn.Linear):\n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "        layer.weight.data = ...\n",
    "        layer.bias.data = ...\n",
    "\n",
    "for i, layer in enumerate(autoencoder.decoder):\n",
    "    if isinstance(layer, nn.Linear):\n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "\n",
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################\n",
    "# Fine-tune the deep autoencoder\n",
    "criterion = ...\n",
    "optimizer = ...\n",
    "num_epochs = 100\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "    for batch_features, _ in train_loader:\n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "        ...\n",
    "\n",
    "    print(f\"Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"visualization\"></a>\n",
    "### Visualizing the Results\n",
    "\n",
    "First, we can visualize the results by comparing input images and their corresponding reconstructed images produced by the deep autoencoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "# Function to display images\n",
    "def display_images(images, title):\n",
    "    fig, axes = plt.subplots(nrows=1, ncols=10, figsize=(20, 2))\n",
    "    for i, ax in enumerate(axes):\n",
    "        ax.imshow(images[i].reshape(28, 28), cmap='gray')\n",
    "        ax.axis('off')\n",
    "    plt.suptitle(title, fontsize=16)\n",
    "    plt.show()\n",
    "\n",
    "# Display the input and reconstructed images\n",
    "for batch_features, _ in test_loader:\n",
    "    batch_features = batch_features.view(-1, 28 * 28).to(device)\n",
    "    outputs = autoencoder(batch_features)\n",
    "    input_images = batch_features.cpu().detach().numpy()\n",
    "    reconstructed_images = outputs.cpu().detach().numpy()\n",
    "    display_images(input_images, 'Input Images')\n",
    "    display_images(reconstructed_images, 'Reconstructed Images')\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"finetuning\"></a>\n",
    "### Using the encoder for supervised classification\n",
    "\n",
    "The following question is entirely open, without any specific code guideline to follow. This is aimed at exercising your own problem-solving skills in order to define the code architecture by yourself. Now that we have a finetuned auto-encoder, we would like to use the _encoding_ part for _supervised classification_. As seen in class, this amounts to a form of _transfer learning_. Hence, we need to retrieve the encoder and add extra layers that we will train in a supervised manner (in order to recognize the class of different fashion items).\n",
    "\n",
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #013220; border-color: #03C03C\">\n",
    "\n",
    "> ### Question 1.3 - Transfer learning for classification\n",
    "\n",
    "> 1. Define the architecture of your target classifier\n",
    "> 2. Transfer the weights from our finetuned encoder\n",
    "> 3. Define the complete training of this classifier\n",
    "> 4. Evaluate your solution\n",
    "  \n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep autoencoders using JAX\n",
    "\n",
    "The following is an **optional exercise** in which you can try to implement the same mechanisms in JAX. To speed up the implementation, we propose to rely on the Flax library, which allows an easy definition of models.\n",
    "\n",
    "### Setup and Dataset Preparation\n",
    "\n",
    "First, we need to import the necessary JAX libraries. Note that we will reuse the same dataset as previously, but you will need to _cast the batches_ into a JAX-compliant format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries\n",
    "import jax\n",
    "import jax.numpy as jnp\n",
    "from jax import grad, jit, vmap\n",
    "import optax\n",
    "import flax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"architecture\"></a>\n",
    "### Autoencoder training\n",
    "\n",
    "Next, we will define the deep autoencoder architecture using the Flax library in JAX.\n",
    "\n",
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #192841; border-color: #779ecb\">\n",
    "\n",
    "> ### Exercice (optional)\n",
    ">   1. Define your architecture using Flax.\n",
    ">   2. Perform the layerwise pretraining\n",
    ">   3. Transfer the weights to a deep architecture\n",
    ">   4. Finetune and evaluate your model\n",
    "\n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ellipsis"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import flax.linen as nn\n",
    "\n",
    "# Encoder\n",
    "class Encoder(nn.Module):\n",
    "    \n",
    "    @nn.compact\n",
    "    def __call__(self, x):\n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "        ...\n",
    "        \n",
    "\n",
    "# Encoder\n",
    "class Decoder(nn.Module):\n",
    "    \n",
    "    @nn.compact\n",
    "    def __call__(self, x):\n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "        ...\n",
    "        \n",
    "\n",
    "# Autoencoder\n",
    "class Autoencoder(nn.Module):\n",
    "    \n",
    "    @nn.compact\n",
    "    def __call__(self, x):\n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE\n",
    "        ######################\n",
    "        ...\n",
    "\n",
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################\n",
    "# Layerwise pretraining\n",
    "...\n",
    "\n",
    "# Model finetuning\n",
    "...\n",
    "\n",
    "# Model evaluation\n",
    "...\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VEdCXSwCoKok"
   },
   "source": [
    "# Generative auto-encoder (modern version)\n",
    "\n",
    "As seen in the course, the previous training version of AEs was used for several years, but several advances in **model regularization** now allows to bypass the need for layer-wise training and directly train deep version of the AEs. These advances are _batch normalization_, _ReLU_ activations and _Dropout_. Furthermore, by definition, the AE can already be seen as a form of _generative model_ (although it lacks a probabilistic definition). \n",
    "\n",
    "Therefore, we will define in the following a simple AE seen as a generative model, to directly learn how to compress and reconstruct images from simple small vectors. Hence, we will need an `encoder`, which compresses the images into a small latent vector, and a `decoder`, that reconstructs the original image from this code. Here, we will start very basic and define the encoder and decoder as simple `Linear` layers. Note that we also use `BatchNorm`, `ReLU` and `Dropout` to regularize our training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AE(nn.Module):\n",
    "    def __init__(self, encoding_dim):\n",
    "        super(AE, self).__init__()\n",
    "        self.latent_dim = encoding_dim\n",
    "        self.hidden_dim = 256\n",
    "        self.encoder = nn.Sequential(\n",
    "          nn.Linear(28 * 28, self.hidden_dim),\n",
    "          nn.BatchNorm1d(self.hidden_dim), nn.ReLU(),\n",
    "          nn.Dropout(0.3),\n",
    "          nn.Linear(self.hidden_dim, self.hidden_dim), \n",
    "          nn.BatchNorm1d(self.hidden_dim), nn.ReLU(),\n",
    "          nn.Dropout(0.3),\n",
    "          nn.Linear(self.hidden_dim, self.latent_dim), nn.Sigmoid()\n",
    "        )\n",
    "        self.decoder = nn.Sequential(\n",
    "          nn.Linear(self.latent_dim, self.hidden_dim), \n",
    "          nn.BatchNorm1d(self.hidden_dim), nn.ReLU(),\n",
    "          nn.Dropout(0.3),\n",
    "          nn.Linear(self.hidden_dim, self.hidden_dim), \n",
    "          nn.BatchNorm1d(self.hidden_dim), nn.ReLU(),\n",
    "          nn.Dropout(0.3),\n",
    "          nn.Linear(self.hidden_dim, 28 * 28), nn.Sigmoid()\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        encoded = self.encoder(x.reshape(-1, 28 * 28))\n",
    "        decoded = self.decoder(encoded)\n",
    "        return decoded.reshape(-1, 1, 28, 28)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see that the model depends on a given `encoding_dim` variable, which defines the size of the latent code. Therefore, we can instantiate our model arbitrarliy with `64` dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "0MUxidpyChjX"
   },
   "outputs": [],
   "source": [
    "latent_dim = 64   \n",
    "model = AE(latent_dim) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only remaining part that we did not discuss yet is what type of _loss_ (defined as $\\mathcal{L}$) we can use to train our model. First, we will simply rely on the _Mean Squared Error_ (MSE) loss, which is defined as\n",
    "\\begin{equation}\n",
    "    \\mathcal{L}_{MSE}(\\hat{\\mathbf{x}}, \\mathbf{x}) = \\mid \\hat{\\mathbf{x}}, \\mathbf{x} \\mid^{2}\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "9I1JlqEIDCI4"
   },
   "outputs": [],
   "source": [
    "# Loss function that we will use\n",
    "loss_fn = torch.nn.MSELoss(reduction='sum')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7oJSeMTroABs"
   },
   "source": [
    "<a id=\"application\"> </a>\n",
    "\n",
    "### Training the model\n",
    "\n",
    "Train the model using `x_train` as both the input and the target. The `encoder` will learn to compress the dataset from 784 dimensions to the latent space, and the `decoder` will learn to reconstruct the original images.\n",
    "."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "h1RI9OfHDBsK"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(3343.1089, grad_fn=<MseLossBackward>)\n",
      "tensor(3005.9805, grad_fn=<MseLossBackward>)\n",
      "tensor(2812.6462, grad_fn=<MseLossBackward>)\n",
      "tensor(2727.8533, grad_fn=<MseLossBackward>)\n",
      "tensor(2595.9111, grad_fn=<MseLossBackward>)\n",
      "tensor(2553.7651, grad_fn=<MseLossBackward>)\n",
      "tensor(2544.0669, grad_fn=<MseLossBackward>)\n",
      "tensor(2468.2820, grad_fn=<MseLossBackward>)\n",
      "tensor(2400.9646, grad_fn=<MseLossBackward>)\n",
      "tensor(2364.0288, grad_fn=<MseLossBackward>)\n",
      "tensor(2379.8469, grad_fn=<MseLossBackward>)\n",
      "tensor(2298.6030, grad_fn=<MseLossBackward>)\n",
      "tensor(2324.0264, grad_fn=<MseLossBackward>)\n",
      "tensor(2216.2561, grad_fn=<MseLossBackward>)\n",
      "tensor(2306.3574, grad_fn=<MseLossBackward>)\n",
      "tensor(2223.1157, grad_fn=<MseLossBackward>)\n",
      "tensor(2159.7192, grad_fn=<MseLossBackward>)\n",
      "tensor(2185.4272, grad_fn=<MseLossBackward>)\n",
      "tensor(2213.2729, grad_fn=<MseLossBackward>)\n",
      "tensor(2150.3906, grad_fn=<MseLossBackward>)\n"
     ]
    }
   ],
   "source": [
    "# Learning rate\n",
    "learning_rate = 1e-4\n",
    "# Optimizer to fit the weights of the network\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)\n",
    "for t in range(20):\n",
    "    full_loss = torch.Tensor([0])\n",
    "    # Forward pass: compute predicted y by passing x to the model.\n",
    "    for i, (x, _) in enumerate(train_loader):\n",
    "        y_pred = model(x)\n",
    "        # Compute the loss.\n",
    "        loss = loss_fn(y_pred, x)\n",
    "        # Before the backward pass, zero all of the network gradients\n",
    "        optimizer.zero_grad()\n",
    "        # Backward pass: compute gradient of the loss with respect to parameters\n",
    "        loss.backward()\n",
    "        # Calling the step function to update the parameters\n",
    "        optimizer.step()\n",
    "    print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wAM1QBhtoC-n"
   },
   "source": [
    "Now that the model is trained, we can test it by encoding and decoding images from the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "Pbr5WCj7FQUi"
   },
   "outputs": [],
   "source": [
    "encoded_imgs = model.encoder(imgs_test.reshape(-1,28*28))\n",
    "decoded_imgs = model.decoder(encoded_imgs).reshape(-1,28,28).detach().numpy()\n",
    "\n",
    "#decoded_imgs = model(imgs_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By plotting the images, we can see that the model is able to perform an adequate (yet somewhat blurry) reconstruction of the input images. The interesting point is that this reconstruction comes from a code of only `64` dimensions, whereas the original images have `784` dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "s4LlDOS6FUA1"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABHEAAAD0CAYAAAAc2ro5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzsnXm8XFWV/VeYBEIgAyEjZCRzCCQhBMOUSJAxCAGUQUCwUUSbD4MtzigqandjN3a3NipiY8ugYMvwU5BZQCSBQEhCwpB5ggQSgkEmU78/6r3DOivvHOpV6g23sr6fTz7seufUrVtnn3Purctee3colUowxhhjjDHGGGOMMe2bbdr6BIwxxhhjjDHGGGPM++OHOMYYY4wxxhhjjDEFwA9xjDHGGGOMMcYYYwqAH+IYY4wxxhhjjDHGFAA/xDHGGGOMMcYYY4wpAH6IY4wxxhhjjDHGGFMAivwQ58MAhlTxvp0AnFvjczkRwBkAetb4uFsD9mPxsQ+Lj31YH9iPxcc+rA/sx+JjH9YH9mPxsQ+boMgPcXYCsH0V7+sAoFONz2UmgDcBjKrxcbcG7MfiYx8WH/uwPrAfi499WB/Yj8XHPqwP7MfiYx82wbaXX355W59DRwCjAbwN4G8Nf9sBQJeGv3UAMAzAqwBKDe27AVgBYCWATXK8bVAe2I4ANgDohvJglwB8AMAuAGY0HJtp7DccwI4N79Xz3K/h+Nr2GoBdG857YSVfug6xH4uPfVh87MP6wH4sPvZhfWA/Fh/7sD6wH4uPfVhD2vohzlQAp6M8QEcCGApgFsoOvgRlp/4Dyk4b39AGAGcC+AiAdxEP4C4A/hXA6wDGAPgEgGMBPIHyoE8AcASAkwD8gd7XGcDPARyEsqPHABgB4OmG9kENn/lXAOMa2mbLd+mLduDQNsJ+LD72YfGxD+sD+7H42If1gf1YfOzD+sB+LD72Ya0plUpt9W/bUqn0ffnbZ0ql0qAG+8hSqfTDhn5NvX9iqVQ6XP52aalU2oNen1Yqlc5o4r1faeJvl5VKpb70+mtNnG/fUqnUuVQqXV7h+WwN/+zH4v+zD4v/zz6sj3/2Y/H/2Yf18c9+LP4/+7A+/tmPxf9nH7bAv+3a8PlRLwDPyd8eRTks6sWG1/8L4O/NOOYuAF6m188CGNmM968lm0O2pqP8NG4hymFXezbjmPWO/Vh87MPiYx/WB/Zj8bEP6wP7sfjYh/WB/Vh87MMWoC0TG69EOZSKORTAU1twzKdQDp1q5PQtOBazL4AvAfgpgB9jc33c1oz9WHzsw+JjH9YH9mPxsQ/rA/ux+NiH9YH9WHzswxagLXPilFBOanQ+gO4ol+xaCuARACejrH/bD8BYAF1RfsIGlJ+OfQrAYQ1tYwC8BWA5gPkAJgM4HsDRKDu4A8patm1RdsqHGt47AGUd3uMABgK4EOVERo83HPN8lJ8aLgcwEWWN3H4N5zkGwMaGz2vqfN4GsKwWg1QA7MfiYx8WH/uwPrAfi499WB/Yj8XHPqwP7MfiYx+2AB1KpdL792pZtgXQH+XM02/W+NhjUU4+dFsNjrUXyhNwTQ2OVY/Yj8XHPiw+9mF9YD8WH/uwPrAfi499WB/Yj8XHPqwh7eEhTq3ZDeWnbZ1QfrJ3CcoZpk2xsB+Lj31YfOzD+sB+LD72YX1gPxYf+7A+sB+Lz1btw3p8iNMN5RCoN1GuDf9W256OqRL7sfjYh8XHPqwP7MfiYx/WB/Zj8bEP6wP7sfhs1T6sx4c4xhhjjDHGGGOMMXVHW1anMsYYY4wxxhhjjDEVsl1zOq9Zs6a0ZMmSljqXiE6dOkWvd9ttt2C/++67TdoA8Pbbbwebo4z+/ve49Pz222/fZD8A2HbbbYPdsWPHYK9ZE+c3euut1ona6tevH7p3796hFsdqTR926BCfcqVRXwMGDAj2jjvuGGz1oR6fWbhwYbDfeeedij63pRk/fvxalLOybzGt6UfzHkVdiybGa7H4eC3WB16LxcdrsT6ox7XIv/U2bdoUtelvikrg34cAsNNOOwX7r39t+1QsXov1QaVrsVkPcZYsWYL999+/6pNqDpMnT45eH3XUUcFeu3ZtsF955ZWo3/Lly4PND1k2bIjLvPfo0SPY+nBh1113Dfb48eOD/eMf/zjqxw8KWpIZM2age/ea7Kut6sPttounlz5wS3H99dcHe+jQocFWH+6www7B1s349NNPD/bKlSuTn7XNNu8Fo+k8qLXUsFQq1Ww3bE0/mvco6lo0MUVai/ywmvcroLqb0FrA56QP0/VGuaXwWqwPirQWTdN4LdYH9bgWe/fuHew33ngjalu/fn2zj9elS5fo9ejRo4P90EMPNft4tcZrsT6odC1aTmWMMcYYY4wxxhhTAJoViVMLRowYEWyOsgDiyIe//e1vURtLmVatWhXs559/Pup34YUXBvvUU08N9uOPPx714yiRr3zlK8lz7Nq1a7AHDhwY9Rs1alSw582bF7W98MIL2BrQ/zPM/xe20sibI444Inrds2fPYLOfxo0bF/XjeaBROp///OeDfdFFFyU/O/d/jTlskvs5GbgxpqXI7am5yJtp06YF+5vf/GbU9uqrrwabI1R1/+PI03POOSdqu/POO4PNe6Duh3z+rRWVY4wxZuuBI2xOOeWUqO2YY44JNv/OVOXG8OHDg/2BD3ygos/l3x1AnPLh5ZdfDvb8+fOjfueff37yGMZUiyNxjDHGGGOMMcYYYwqAH+IYY4wxxhhjjDHGFAA/xDHGGGOMMcYYY4wpAK2SE2fs2LHBHjlyZLC1ZDfnwdFSbTNnzgx2//79g826SAD43//932A/9thjwX7mmWeifqzb33nnnaO2IUOGBPuWW24J9tKlS5P9DjjggKiNz6s9ZCyvJVyNJJfzQDOkn3feecFmneoee+wR9VuxYkWw161bF2wuFQjElchWr14dtY0ZMybYv/71r4Otmtjrrrsu2DxfgDj/RK6KlTFFQysKNaJzm/vVet5PmjQpes1r+NJLL43avvCFLwSbdez77LNP1G/WrFnB5mtGe6fSPXXQoEHR64985CPB5j1PNfcTJ04MdufOnYP95JNPRv1+//vfB/vcc8+N2o499thg33bbbU2+5/3OvyXnkzHGmK2DCy64INicAxOIKxjzb8nXXnst6seVa4888shgP/fcc1E//k2iVXc53xzn0dRcn5dddlmwOXerMVuCI3GMMcYYY4wxxhhjCoAf4hhjjDHGGGOMMcYUgBaRU3Xs2DF6zWHeLEnSkP4333wz2LvuumvUxnKWZcuWBbtTp05RP5ZGcT+V9nTp0iXYGqLO5chZfsPlxoE4TG/jxo1RW48ePYLN8q/FixdH/YoSXp4qG6s+/M53vhNslj0AsRyKS4JzWT4g9uFee+0VbJWz8Rzh8QZiv7399tvBVvndN77xjWBrCOWXv/zlJs83VwLYmCJQ6V5TaT8uz9m3b9+ojfcIltNeccUVUT+WTL377rtR27/9278F+9prrw22rr1Ky4S2B1J7KpcsBYAf/OAHwR4wYEDUxhLk559/Ptivv/561I+lyz179mzy70B8zdTzYNnrWWedFezLL7886veJT3wi2PPmzYvaONxcfWxibr/99uh1nz59gn3//fcHe7fddov6TZ8+Pdinnnpq1PaHP/yh2eeh1/j2fJ9ijNk6+N73vhds3ec4FcIOO+wQbJVf8+9Avf9n+HeD/l7h6zh/rl5b+TpuTK1wJI4xxhhjjDHGGGNMAfBDHGOMMcYYY4wxxpgC4Ic4xhhjjDHGGGOMMQWgRXLiHHzwwdFrznXDGn7NMbPnnnsGm8tHA7F+nnOcvPPOO1E/Lv/GWkUtC8c5A+bOnRu1rV+/vslz4vwsQKx51Jw4fIwJEyYEW3PiFEVfnsr7wnksgLh0OJf5A2K9aC4XEM8XzjukJcZTeW/0+KyJZb8A8fzhnAMAcNVVVwX7k5/8ZLCdA8cUnUpzcXHZTV4f3bp1i/rl9uSddtop2Lznc/4VIM6dpmvsiSeeaPL8xo4dG73mUqDtndQ+cv3110ev+Zo5e/bsqO2tt94KNue6GThwYNSPNf2c30vzqXD+tvnz50dtvBe/8cYbTZ4fEJdS1TLlPDdSOYGKSrXf5+yzzw42l57dd999o368XjjvkF77+LPvuOOOqO2Xv/xlsH/84x8H+7HHHkuen+4PRcnjZ4ypX3LXMb5W8W9H/f3FOXHGjx8fbP1NyHBJcSDOw8d7o/5e1M82phY4EscYY4wxxhhjjDGmAPghjjHGGGOMMcYYY0wBqJmcisPrNdSew7w5JFjlMatWrQo2h2sDcbnTlGQKiMO1OYxO+zF6HiyhWrduXbA1bHnYsGHBfuqpp6I2/jw+p1GjRkX95syZkzyv9gqPz5AhQ6I2ljNoiVouHc5jwpIAYPOSpo1o2D6HcnPpWj0Gy7NY0qXnqCUBOdSSSxjecMMNTZ6f2XLU9/w6J1Fg6cERRxwRtXFpR5X6bK2kymKqhOOEE04I9i233NLk+4FYpqgySpauchlPlbhyCWuVp6r0qpH9998/er333nsHm8swA5vvM+0JlT8xPH5aQp33PW7TfY5DudlXOs48fmwDcYg5S1T1s3r06BHsiy++OGrjsrD1IKHi8ddxYPh7s2QKiMdy0aJFwVY5G1/vZsyYEezOnTtH/Xr16hVslhwDwHHHHRdsLhOvEgH22y9+8YvkeRhjTFuzZMmS6DXLtnlf7t69e9SPr4sPP/xwsPXehPvp70Xum5MZG9MSOBLHGGOMMcYYY4wxpgD4IY4xxhhjjDHGGGNMAaiZnGrKlCnB1gohLGeZPHlysGfOnBn1YzmVhghz6HVOGsVtLN/RUOdcSDCHwe2yyy7BZmkVAMyaNSvYHMIMxJnTOaR5+PDhyX5Lly5NnlN7Yp999gk2jw8Q+1qlMZwxnkPI2QZiGRz7SY/HYYy5UHZ+X8eOHZP9dF5x2CRXbTEth67L1DpVf0ybNi3YLOcAgIsuuijYXJGF1x4QzxOda/Ug/WBScqpdd9016sd74b333hvsQYMGRf14LF966aWorVL5hUo6KoH3FCDeEw4//PCo7c4772z28VuLY445JtgqDe3UqVOw9RrErFy5MtgvvPBC1MZ7NvtH/a2yYIarjPF60MogLFs78MADk8erB1LXnYceeih6PXjw4GA/88wzURvL5XiMVY7Mvho9enSw+ZoL5KsxstSR16Xeb1133XXB1jl32223BduVqowxbY3uPbwv870J/7YA4r2Sf7fm7kN1z+c0G9xP7y+NaQkciWOMMcYYY4wxxhhTAPwQxxhjjDHGGGOMMaYA+CGOMcYYY4wxxhhjTAGoWU6cJ554ItgHHXRQ1MZlZfv06RNsLauaK8/G2kXOe5DTYVer0eb38XloaTnWg2teFz7H3XffPdirV6+O+mn+iCLA30fHhPM35DShuVLPnK+D80Oobj+XH4K1r5xLQEuu9u7dO3n8XGlCszmaRyaXzyjVL8fZZ5+dfM+NN94YbC5VDcT5ck4//fRg/+hHP4r6VVrOvB5Ifb+99toren3ttdcGm9cD7+nVonOC172eH/ub/dulS5eo34IFC4Kta1bzd7UnuFS6fnceJ81hw9ePXIlxHlvOYaOlVHnf1GsanwcfQ/OMcW6jUaNGoZ7QvGnsq6OOOirYBx98cNSPy9fq3sXXnVxertdeey3Yqbx9QJwnKvdZ7Gs+NgDMmzcv2D//+c+jtm7dujV5/Nz+b4wxLYXe8/H+yNe4jRs3Rv04f1tu7+XrqebV4esuXx/0+mlMS+BIHGOMMcYYY4wxxpgC4Ic4xhhjjDHGGGOMMQWgZnKqhQsXBvvll1+O2jikmkOttQQsy29UdsRhcLlybwy3NUcewX1ZzsOl5IC4bOiYMWOitvvvvz/Yjz76aLCXLFlS8Xm0V1iCpCV+ufSzyqm4tCrLINQ3HK7Ici0NhcyF/nOYJJdZ5XKuQFr2BsQlB7t27QqzOTx+Gmaaojlh9lwe/NRTTw32nDlzon4PPvhgsMePHx+18VxgedDWTGrf1DUwd+7cio5XaanhXL/cXs4ccMABwdY9mfcBDYkeNmxYRcdvC1hmzKXCgVgCo2PEEmS2VXa18847N/m5Klvm9awyHZbA8v6tey/vt/pd+vXrF+wiXgtz9xFf+9rXgv3iiy9GbTxPVebHPuVwfJUcsw8XLVoUbL4eA7HfcmuMj6+SOPbhiBEjorYTTjgh2L/97W+DrdK8nGTalOG1o3sZw5INAJgyZUqwf//731f0WdXK3fgcdf+pdM82ptbwmujbt2/UxnLQVAlwIC2hz8n/Fb6/5HNSqTfLUDWtg8mTk7qxTyvd09Q3fA399Kc/Hezbb7896nf33XcHW+97eJ7lfhex9E/vyxYvXvx+p74ZjsQxxhhjjDHGGGOMKQB+iGOMMcYYY4wxxhhTAPwQxxhjjDHGGGOMMaYA1CwnDqN5Uh577LFgr127NtiaZ4T1bDvttFPUxrqynO6Nc9jk+rH2PKd/ZL2j5gjgvBy33npr8nyZWpRabmtYC6jnzD5VnSqXoWe0bCuPUa6cPOfL0bYdd9wx2Kx1Vd0+6xO1xDiXUdYcE6ZMLfLgnH/++cHWfBFTp04NNs8tLYV9yimnBHvChAlR22c/+9lg83rOnaPOyaKszRQ5HXGvXr2CzSW6gTjnQaV5b3JUmy+H1zrnWNP8a6xJ15wcPXr0aN7JtiK5PZVz+XDuOSDe51IlwBWe2wMHDozaeK/kYwPxePJeqfrydevWBVt143xNKEpOHB4vzYnD48e5mubPnx/14zHSPYjnNsP3MkB8T8RaevUTn6OeL69nPobmm+M9Qf305S9/OdicE8c5cJpG/cg+yOXB4ZyLP/nJT6K2Sy+9tKLPrnTP5jmofsydY72RG4cUvLa/853vRG133XVXsG+77baorTl5Os3m5OYz3+fn8uQxubZc6fBKc69OmzYt2D//+c+T/bYmeMz1npvHspq8N5MnT45eX3PNNcHWaxrfNx500EHB1ntGzomj+2Kl+UE5X+7TTz8dtZ122mnJ96VwJI4xxhhjjDHGGGNMAfBDHGOMMcYYY4wxxpgC0CJyqhwvvPBCsK+++uqojaUTKpfgEORcKTgOyeKQJg1pzYXOpULyuUQcAMyaNavJz2oORZRpcKi+hhZyCXkum6vvYx9q+UwOT+Q29Vml0jSW4WiJcS6prOXe+BxTJXq3do499thga/hiz549g82lhlWaxjLFsWPHRm3s40cffTTYr732WtRv+PDhwT7ppJMqOvdalCosChqqymuMyytqaWQmNyYtPV4skWP5iJa45e/55ptvtug51RIeP732sQRJS4KzvCoXNs6hv9ymIcG83+qcSUlb9drXqVOn5DH22WefYD/yyCMoArkQ+S9+8YvB5rFkSRmQvwbxvQmPsfqG70V4zHWe836q9z38Pva1ytdZVvLSSy9FbePGjQs2y1/XrFkT9auF/LI9U+n9R26PYvvKK6+M+rH8QtfRZZddFuwrrrgi2LNnz06eR46cdIg/62c/+1nUpj4vAjqWDI8D+/ewww6L+rGM8uijjw62yl15v7vkkkuithUrVgR76dKlwdby0yxZfP7556O2elxXlcL747333hu1DR06NNj821H3w9RvyUqlVUC8j+buIZ988snkMbcWcvfcuk+m2HfffaPXBx54YLD52jRy5Mio35133hnsfv36RW2cAobTv/Dx9LOfeuqpqC31DOCmm26KXqckzdXiSBxjjDHGGGOMMcaYAuCHOMYYY4wxxhhjjDEFoNXlVDlyVRU4hC0XdsVt/B4NbeOwLg0l5XD9XLUTrcJVCfUQ/sih1iqnWr9+fbA1bJypVLrCY65VUDiMXCt+cNgkS280nI/D2XLyC5V8mTJTpkxJtrEshOUEXPULiH21bNmyqI3Dhzl8Vj93w4YNFZ7xe+TmHUvBgM2rIBWNnJyK11VOTlUt7N9UxYGmXjNcoYkrHCq5KhJagac9kQux5e+hofapimp6jeQ9O7fn8f6ox+A9kOWRus/z+eoxRowYgfZOrkqGcuaZZwZ70aJFwdbKmxxqrRUSU/107FK+VokAj79Ksvjehq+ZOfldjk9/+tPBZllPkcnJKnL7V4r+/ftHr7/xjW8Em8P6c5Xi9P6GK6GxlIQl/gDwrW99K9gPPfRQ8hz5PPg9AHDGGWcEe++9947azj333OQx2xJeE7qec9Ix/j5cKeb73/9+1O+6664LtkrMmEGDBgV7zJgxUdt+++0X7A996EPJY3B1Td0TWJLK10itBMv3Yiz/AuqjSpbuganffpVK6HNyKh2vVEVkPcbq1auTxyw6Oek1X0ua8/v3H/7hH4J9zDHHBJvXFBD/Rli8eHGwtRocV/sbP3581MbnyHutVn7k/ZU/CwD+/Oc/B5vXtlaq5PsETdHSsWNHNBdH4hhjjDHGGGOMMcYUAD/EMcYYY4wxxhhjjCkAfohjjDHGGGOMMcYYUwDaVU4cLk1abcnuSknpy4F0Xp2cRj1HpaUoi0Lnzp2DrfkQOOeB6rg5hxCXJtXx4XHmXA6aD0KPz7DfuJ9q/Vm7qVrX1BzhsuRAPkdHW1BNaddc+XYdF9Zes3/mzZsX9eM21gPz34HY31dffXXUdscddwT7wQcfDLbqXVkbPnHixKiNS+5+8IMfDLZq4/m1+vgvf/kLgHjuF4lcHoDDDz882JzjA4hz5LAvVHOfI5VHIqdPVzhvD5fZfuONN6J+nPeGryfA5mXp25pdd9012DwWWkac9y/9vqzZ5j1Ktdy8T3Pesl69ekX9eG3q3r7bbru97/GAeO3o3qH5QdojuZw4H/nIR6I2zj/DZcU1fxsfQ/e/SvPDpXLp6N9zeVv4+pcrZ857suaD47LSp5xySrA1J061a701UB+nfNCc8+ScC9/+9reDzWMExOubS7trDi8uk/zMM89EbewDvq8aPXp01O/6668Pts4TzuXB1wfNL8L54HS/aCka50vu3lxJ3avreyZMmBDsm2++OWrje0y+V8jda+but/j6OX369OQxvvjFLwb7oosuito4h6CuU/Ybzx89Xz5GPeTAUfjaBKS/Y6V5+HLrvto8Wbq+6wkd70pzqvE++dGPfjRqa7znBuK8N/fdd1/Uj/M9cc6uffbZJ+qne0mqjfdJzXurv3GYSZMmBfull14K9vLly6N+s2fPDrbmKatmbToSxxhjjDHGGGOMMaYA+CGOMcYYY4wxxhhjTAFoFTlVpfIODi2tVE6lIaI52UAKDbPiUDAt91oNbR06XGtYnqIh91wyTUP6OcScQ1w1fJfJtXFYuoY48vzhcHAtqcvnoSGo/D4Oc+vdu3fUr73JqXKk5GM5KZmyYMGCYHNo+IABA6J+HOL63//938GeM2dOM874PaZNmxbssWPHRm0817i8HxCXY+VwSJXbcCj1kiVLorZnn30WQL4MfTU0zttq94jc3srrQ/fFI444Itjsw5kzZ0b9Ro4cGWwuq7pq1aqo37//+78H+84774zaqglZVvgcOTxVw3ZZ7sbh8EAcst4eYGkCj4WGXXNJSvUj+5j3L+2Xk0cyvG/qtS8lM9Y1kQvr51LG7ZXcvceXv/zl6DVL9Hhv1fHn8crJtXJ+SpW+1nXE79PvwtdT7qdzjo+v12CWxrJ8h8PaAWDp0qVoS3LjXGnYupai/frXvx5s/b5cIpxLxV588cVRvx/84AfBvuaaa4K9YsWKqN8ee+wRbL2/+c1vfhPsnEynUviarv7mPXXPPffc4s+qhMY5nVsDCvfdd999g33jjTdG/XjP/NGPfhS1fe9732vy2Cop5H2tFvf3N9xwQ7C/853vRG08X1jSbN5DJaqVru9UKXIlJw2t9LdMvf0OZDT9wOTJk4PN9+p9+/aN+vGYnHXWWVEbp1bg64ze87HEnH+P6jWNfyOwPAsAevToEWyWuet1nOcV99M23kcee+yxqB/P1SFDhkRt1fyWdCSOMcYYY4wxxhhjTAHwQxxjjDHGGGOMMcaYAtAqcqpKw8g4zKjSSlC5jNOVouFxHB7Okgv9LH6t4Xu5tiKSki5xGJrCYddAXDmGQ9HU1xyWzqFyLLcD4szh6hsOf+Xjq8SL2/j89Jh8TirDaStSUpzceqtUptizZ89gH3zwwVHbySefHOxvfvObwdbqJB/+8IeDzfOE/577XAA44YQTgj18+PBgq3SL0SpmnOWe54WGZfK81uPffffdAGofErulx6v2/bwnde3aNdhawemee+4JNst6dHz++Z//OdgcSgsA//Ef/xHsxYsXJ8+J1/c555wTtfHa5HWpc5NDdx944IGoTSsNtDVczYbDdrUCGkv7dK9k2UauggpXA+T3aCUs3ud0r+BxZ0mtVqzhPVbPqT1Xd0vtpzwvVV7DVYNy1UdSlaWAyu8PUiH9ev/CftLw/pR8QPvxOWmbSqgb+dSnPhW9ZulZW0gJcuOq1cNOOumkYLPUVKuRzZ07N9i6PlJ729lnnx29nj9/frDPO++8YKskgcdMJcJ8vjlY+s0VU4C4EuHPfvazYC9cuDDqx99z0KBBFX3ultI4p3OVgPSej2VlPF4qjf7Sl74U7FyVxZRUVT87JYfU17n7NPY9X2eBze+JKj1+ql97qxRXC3IV/3L7QC0qDFcqySp6dSqWdwLAiSeeGGyu+AbE/mDpkv524j2TK8wC8d7L61T3Mb6n5/0pJ2lWCTj/ZshV6hs1alSw9f6S0z3wbwu9H+Z7Lj3HatK3OBLHGGOMMcYYY4wxpgD4IY4xxhhjjDHGGGNMAfBDHGOMMcYYY4wxxpgC0Co5cSqF9f65ktGsL1etakrLndM05uDP5VwC+llKPehMGS4tyXo/1Thyrptbb701auP3VVqOlcdYc+Lw+1TjyDpJ1qKqFn727NnB1vJ3nI+HtYrtTduq+RZyeW+49Ob+++8fbM0XwuVT77333qiNy3SffvrpwdbS4UceeWSwWRd64YUXRv1mzZoV7I9+9KNR29577x1szkcwY8aMqB/vA5pbgOcozwv9zpyvQ3OPrFu3rsm/twWpvUz3HF4fqhnnOfLd73432J95hqdoAAAgAElEQVT//OejfrwmWDesuv2VK1cGW/fFc889t8l+1157bdTvyiuvDPa4ceOiNtZVc84VLjcOAGvWrAn2j3/8Y7RneB/huag5EHIlu1PlTXUubNiwIdicX0j31NwxeD7xmlJt+LHHHtvk8ZRhw4YFm/OEtDc++clPJts4PwzvH5qvg8dSr1W8NnNlsPkYufuLVC43fZ0rSa8llRm+FnLeO74WAJuXY0+dYy3p2LEjJk6cCGDzctG8BnSNsQ/YP4250Bp59dVXg63XGc4/w2tW9+v/+7//C/ZnP/vZYHOOCSDOCcHXSCDef7ns9IMPPhj14/mpa33KlCnB5lxoufxIeoy2RO89J02aFGzO63PfffdF/X79618Hm8vCA7GveE3ofOV1pLn1mGryXWlOv9zxK/2dUeneUVRy9+Tsu9zvwFwZ8Vwbk7s/1N8eRWDbbbdNnvf//M//BFvvtVK5mjifJgBccsklwe7Xr1/Uxrms+F7hM5/5TNSPf9NwzhrN27hixYpg6x7Hef14j9d59eyzzwb71FNPjdq6devW5PnqtZX3UN1PG39nNAdH4hhjjDHGGGOMMcYUAD/EMcYYY4wxxhhjjCkA7UpOxSGKWu65UmlUtbIphsPvOORYy1NvTXCJ2pzEjEPKNASY5TvsJw1t4/A7DmfTMFNGwxg5hC3nQ5boqJyKz4O/s0pT2orG88vJp1gGB8Sh8CzhWL9+fdTvhhtuSB6Tywt/7nOfC/ZTTz0V9eOQZg5R11KFXEZ87dq1yc9leRuHxgOxlEFDIHl+8Vhx2XMgX3K3UV5Qi/2FSZU1zpEK7dV5yWvg0EMPjdq4tOOqVauCffHFF0f9DjrooGAfffTRweZy7wDw8ssvB1vLlPN57LbbbsHm0uNAXB7yP//zP6M2DunlfUTD1fl76Zxub7Aco0+fPsHW8GWWKaqEgPc2nve58HJeAyx/BWI5Wk4SxHNNy2LyulL/8J4zevToYLcHOVXj99VrCYdNq4yQrye5MvG8xl544YWorVH+A8TSCR1/ljjxOeb8pG08X7gfXweBeN3rOuJjcvi37gk8pzmUvSXZcccdQyg7X6eA+BqkEhUe265duwb7gAMOSH4W73lA7BMeWw2Z5/O45pprgv3DH/4w6scld1lOCsT3Yzw/+XrcHBYsWBBsXbP8+plnnonaxowZA6D2MqvG8ctdF/U6w6XhWWarx2CpON+vAul7TB2TlMRJP4v3Yd0TeO2wfJgl5EAsS7/zzjujNu6rsg0mte6Bzfe0IqLXIP79WE159Vxp+1zfXGlzXrNFYaeddsLIkSMB5H/35GRkfN/++9//Pmr7wx/+EGy9F2ncW4B4Xf7Lv/xL1I/XFe+tH//4x6N+V199dbD1dwb7Krc//OM//mOwhwwZErXx9Y/3B51jvBZ5bIDNUzxUgiNxjDHGGGOMMcYYYwqAH+IYY4wxxhhjjDHGFAA/xDHGGGOMMcYYY4wpAO0qJ04u/wnrsNnOlTCtlFz5QC3pWin1VsaP9eWsN9WxY20k66yBuAQkaxBVf5rKO6K+TmnQgbQmUXP48DkefPDBURvr5Hk+qra5Ldhuu+2CxpbL6gHAsmXLgn3WWWdFbazj5xKjXM4UAH76058Ge+zYsVEbl+XkErM33XRT1O8nP/lJsLkM8fTp06N+udK8nKOD86loaVDOv6NaZNbJ8h7DJQyBeD6pxr/xPHL5h6phS/cJfr9q4nlcNc/KPffcU9HxH3744Sbtww8/POrHeRk05xHnduDxZ38C8XzMwaUoVbN82223VXSM9gDnxMmVneRcU5rbiK9PvF/pXGA9ey6/XE77z228VnRusU84FwwQ76OsPW8PpHT9EyZMCPZzzz0XtfFYcl4GzV3E+6SWUuU9j8c4l+umUvQ9fL6cg0BzSvD86d+/f9TGeacYzRHD5bO/+MUvZs+rVrzyyiu47rrrACD8t5YMHjw42N///vejNi51+8c//jHYmruFS9hyviEt6/7SSy8Fe+bMmVGb5lWqBs45cemllwZbc1PwmtVzbMwfpKXst4TOnTuHPHl6Lk8//XSweU0B8bjy+zR/DV/7DjnkkKjtiSeeCDavCc2vxPlNeB/TdcT5lfS78H7D9xWa94bz/33729+O2viYfDzdx5cuXRpszj0HAP/1X/+FosPjDNQ+H141uRBz+ZGKwjvvvIOVK1cCAMaPHx+1vf7668HO5Yvl8t0jRoyI2nie6j0Lt/HY6dzmaxrvmZp/h+9RuR+A8B0V/u0AxD7VPY/vDTi3jea94fPX/bQaHIljjDHGGGOMMcYYUwD8EMcYY4wxxhhjjDGmALSKnCpVgk1LrnHIVC4MOHVsYPMQ5Grgz+JQTP0slg1oKHG9waWq//a3vwVbx5tLt3EIJxBLYLikrIaqcrgZh5nmQvbUN+w3Pp7Kqfj4/L2AWI7Cbb17906eR2ux4447hvKSXKIWQCix2hTsRw6H/POf/xz1Y4mEhgj/5je/Cfb/+3//L9gq6+ISwnw8LVXIYYg6n7iEJkvfrr/++qgflxSdOnVq1MZhlDNmzAi2hlnzOarUpz2R2k91bnPJ1Ztvvrmm56ByLH6t8/G4444LNpcH5/NTcuXS2VbplpbAZWpdHn5L4TnGsiMOPwZiv+audzxmOrdTx9DrLB9Pw4VZusU+0LKYXDpXQ8hzMp32wlFHHRW9ZkmnXiN4/nGp3vvvvz/qx1JdPp7C16OcfLhS1L+pNaCSOL42cBlsALjyyiuD/eSTTwZbQ8+POeaYYLeWnKpjx45h/xk6dGjUlioLDcTrj+cv20AsYzrxxBO36Fy3BJbR8BrT9cbXbpUrzJs3L9if+cxngq3rnvcEDf9vlCHwfNlSNm3aFCQIKnfitamyCpadsoRB/c73m0cffXTUxmWJeb3l9kJOL6Dweej481rkta7l5Fleq/diPG/5eHr9ZImrnu/cuXOT59+e4bmu35fnRqUpN3hP0v0pt1+l2nR+6v1ZEXjrrbdCqgK9brGETSXgvOZ47Wg6Au6nvwN5T+Fj6PxN3ffotfrGG29Ec9H9lM9DfyPw71P+7ajfi/up7Frl2pXgSBxjjDHGGGOMMcaYAuCHOMYYY4wxxhhjjDEFoFXkVKlwM60solmcKzmGkqquoSF1HIKlYY6p6lcaLrw1yak47IvD5zWMbsWKFcHWMeFwQh7zSqtOqdSG/aGh5hwGx+F32o9D27jyARBXb+CKHNVWLKslGzduDKHsmpmfQxa1ggP7kUP+NTSQw7UffPDBqI0lcyx30tBDrvTAcieFqzuoVI0rXHGVoxwaAq/ruxGuDgTEoZK6jzSOaXuoOpc6h1/+8pfRa65kdOGFF0ZtL774YrA5fPt3v/td1G/27NnNPj+WTAHABRdcEOzDDjss+T4O1c9VSeKQWQ2VT1XOAdqH7xheY1wtQfcolrqoHIb3TrZV9sDH5DFTGQ2PkYY+8+vc3stVYvr06RO1cYizVmtpL2hFP94/9Lv27ds32F/96leDraHc06ZNC7buTzzXeQ+qtLpJc2SCfEz+Lvq9WJL73e9+N2pjeSpXa1L5NF9rNKRcJXi1YuPGjXjssccAIPy3KbTCJEv7+J5gyJAhUT+urqJjlpIN6D0SHz91bQLiOaRrMSUvze2bCq/9XHUpnjM5KWCt2LBhA+666y4Am8t2+f5A9zidY43ouuH70Ny9Zy41A4+zSnlSx9MxTlXc0X48D/Q7pySueo/KewTfvwGb70dFIVfZp9JrPY8L+yD3/ty84Pfpvqz36UVDJZO56lS85/HepWPAx9B7OR4/tnX8eR3x5+o64r1crz/82byORo4cGfXj+0v9LvzZfAz9XvycQ+X/WsmwEhyJY4wxxhhjjDHGGFMA/BDHGGOMMcYYY4wxpgD4IY4xxhhjjDHGGGNMAWiVnDgpVJecykUDpDWKqntLaYKbUzIu1U91ye25DHGtSZXzUz1wroxnSi+q2nLWE6by6ACxnlJ15zwPuE3zLrFuf+bMmVHb2LFjmzxepSULW5IOHTqEua8+4O+bK6nOPs1p7nO5oNavXx9szYHEmmXWzGrOFJ4LX/rSl6K2V155BU2hulj2j2qlOS9Qz549k8dgnWxL5WyoBtX6Dxw4MNjst1NOOSXqd+CBBwb75JNPjto4BwSX+p40aVLUj/X4vN/94Ac/iPqxrv4Tn/hE1PaNb3wj2FwGVmEdfy7PR64MdlEZMGBAsPU7VVMiVfcEXn+cCypXtjqnPU/p0IF4Tur1OJcDpL3AuaSAOIeE3rPw+F133XXB1jXAPtS5ncqHoXtyat7nfKhzKeUbzWmQW3+8f6dycgDxfqF7vpZgb234ugXk9yXTdujcXrZsWRudiWlvaP5EhudNLp8K05zcYpUcI1favt7Q3wj6uhHNq9OacN7BSlm1alULnEltcSSOMcYYY4wxxhhjTAHwQxxjjDHGGGOMMcaYAtCmsegsbQDypRI5TI1Dk3MlFXOSHX6t4eqp8oFaNrReQvkrYeeddw52Lny7sez1+1FpWGNOJsNo6CIfn32oIZNc5n7+/PlRG7+Pj8cSrLZi06ZNQRrWvXv3qI0lVFzyHYjnOssqcvJFLQXMoZI8fip9Yt+xNI1LWgPAv/7rvyJFpbIBRsP6eW7kwv9Zapebn63N0KFDo9epMoT9+vWLXvM4X3HFFVEb+5T34T333DPqx2M+efLkYJ9zzjlRPy5FrvuiSq8qQcefz4P34WpKMrYXOLw6VSociCU8us/xa5Y4aeg2H5P3BC6ZDMThw7nrIu/F2o/LEOt84vNtT2tsu+22C3tZThKs16AJEyY02Y/XAxDv0cuXL4/a+PN431XJd0puqOPIr3Nl4nlNqTRBSxSnzveqq64KNpceB+LrhLa1tZzKGFNsVKLJ8B7Fct9aSKZy5ORUOfmXMdXSfu6ijDHGGGOMMcYYY0wSP8QxxhhjjDHGGGOMKQB+iGOMMcYYY4wxxhhTANo0qYuWP2YNtear4DwLqZwp+jpXmjVXYpw/O5dPRc+/nuHx57FTPT5r3bUcK+vx+Xh6DB5n9gXnX9G2jRs3Jtty+n7O9bNy5cqojTWtnD9Ay7G2FY1z/frrr4/+PmXKlGBPnDgxauPS2ewDLQnIvuKS4noMHhcukwwAw4cPD/bvfve7YP/85z/Xr9LkOel5MDlts67LQYMGBZtLHOq84M9uD2XkG9Hz5Hwn7Dddb5zzSHOkpL6rzgPe/+65555gP/LII1G/P/zhD8HWfDkM+y23B+fg/aIIJatT8LjzOL/88stRv65duwZb5wKPBefE0XHhec/v0b2X9zwtU57KT6Vr5aGHHgr2sGHDkKJa/7cEAwYMwC9/+UsA8XgDcf6o22+/PWpL5YB79tlnk5+l48rjx2OiPuQxT+XRaep16rN5rWveM81vluKSSy4J9rRp06K2Hj16BPvYY4+N2rameydjTO3h+03Nl8jXtWp+E+oeyvcteu+Z2m/1nPi3hjG1wpE4xhhjjDHGGGOMMQXAD3GMMcYYY4wxxhhjCkCbyqk0xH/Dhg3BzpUf5fA1lV+kylDnQuBUssEhdxwSp/1YVsOh7EAc7lwLCUFbk5JQ6feZNWtWsAcPHhy1cfh2rvw7jx2HIGp4Ob/W8U+FRmrZv1yJcS4JuHDhwmC3p9K4QFxyGQDuvPPOJm0AGDJkSLBZJqXlqVmKoxIbDrVneYeO33e/+91g50pB81xIyaeawwMPPBC95u+2du3aYOdkSsuWLdvi83g/evfuja9//esANi9bzyWKdewWLVoUbPa9jp3urwyX3WQZlkoF+ZhLliwJ9le/+tWo33333RfsmTNnJj+30v0vV2Kcz1fnbZHg8dxvv/2CrVKW9evXJ4+RkhXqnsqlvlkOqWsgF/LN+y1fZ7t06RL1Y/+zjAuI91+VjbUlK1euDGtR5zZLgb70pS9VdLychLfSUrN6DF6zfN3S8Wff5KS/PA/42ACwevXqis6R0bLhfK255ZZbojaVPxtjTHPo3r17sN94442obUvv0fW6ysfL/ZbM3cvm7seMqZb29WvUGGOMMcYYY4wxxjSJH+IYY4wxxhhjjDHGFIAWkVNVGm6WqzCjVadYNpWqkqHH5GNoeJ0en+EKDhxCrlIc/iwNR+a+/Nl6vkWBJTUcvq1hjHPnzg328ccfH7Vx9SeeE+oLHn/2da4ClZKSxOWy2HN4ORD7kMPSc6Hy7Z3nnnuuSfvhhx9ui9MBUJ3EMLd+9XjXXXdds4/fGqxZswY//elPAcRVXgDgzDPPDLaG5fbp0yfYL730UrCXL18e9WNJBEt3gFjKwmtb5aksV5o+fXqwly5dGvX7p3/6JzSX3P6fk9Xxumd5XNFg+SHLjlQeyfuNVnuaM2dOsHl/1GpDPE58/OZcF7mN5VlPP/101O+3v/1tsFWaxLLBXAWn1mbjxo14/PHHAQDHHXfcFh9PrzM8Rnod0/uKRnQesExq3rx5wVbJGt+LqMyY1xwff/To0VE/lcamjsF77XnnnZd8jzHG1JL+/fsHWyX/qSpUuUq4ud+mfDxtS/3O1H2dUxQYUysciWOMMcYYY4wxxhhTAPwQxxhjjDHGGGOMMaYA+CGOMcYYY4wxxhhjTAGoWU6cXH4DpmvXrsHWHCSsa9ScCJyroZoyxDmNo8Iax0rzqWhOnFRJ5Vy+oPZMqhypalGZnj17Rq9Z05/KGQTE458rz15pW07PmstvwzlGOLeA5iowphreeecdrFixAgBw8cUXV/w+zosydOjQYKvmeq+99gr22LFjozbuy3ur5tVZsGBBsLlk/FNPPVXx+VZDrsT4oYceGuyDDz446nfaaae16HnVkldffTXYvI/qHnXOOecE+9prr43a2P8bNmwItuZC4dLhfA3WfjlNP+dy4XxLJ510UtSPrw+a/4X92Dj3twb23Xfftj6FmlGUexZjTP0yYMCAYK9bty5qS/1u099w/FuSr7v6u4Cvkx07dozaOF8mX8c1X6i+z5ha4EgcY4wxxhhjjDHGmALghzjGGGOMMcYYY4wxBaBmcqpUGXGF23r16hW1cRlUDafncDYOidMy06lyb3pO3KafxbAEiMPm9HWlcrKiwqWG+btqKdtu3boF+5FHHonajjjiiGCvWbMm2Bq2z2OekrYB+dLtLHniuTlo0KCoX2NZ2aYYOHBgsBctWhRslYkZ05pw+d9cKeD2Tk6WoeuZOeSQQ1ridFqdlMRJ4T3w7LPPTvbj66nKd/haxXs2S7CAuBT9qlWrojZ9nYLDxvX6zHt7zsfGGGNMilGjRlXUj68z+jvt5ZdfDvayZcuC3alTp6jf8OHDg/36669HbXx9zl3Hq0kDYsz74UgcY4wxxhhjjDHGmALghzjGGGOMMcYYY4wxBcAPcYwxxhhjjDHGGGMKQM1y4jC5XAec9+aqq66K2rjsLWvnAWCPPfYIdufOnYOtunrWJ7IGUfuxVl/Pl9+XOh4APProo8FOlRRv6rOLCOeO4XLFixcvjvrxuM6ZMydqmz59esucXAvBOSA4p4R+Z2OMaS633357sDmfjerxK4X3q0rz17QEXO6Vc4kBwPr164PN+XeMMcaYSjnxxBODrTngVq9eHew//elPweYS4ABw6aWXBptzYL7yyitRv6uvvjrYP/nJT5Ln1L9//2Afd9xxUVu95001bYMjcYwxxhhjjDHGGGMKgB/iGGOMMcYYY4wxxhSADjnpUxOsAeAY6NanH4DuNTqWfdh22I/Fxz6sD+zH4mMf1gf2Y/GxD+sD+7H42If1QUV+bO5DHGOMMcYYY4wxxhjTBlhOZYwxxhhjjDHGGFMA/BDHGGOMMcYYY4wxpgD4IY4xxhhjjDHGGGNMAfBDHGOMMcYYY4wxxpgC4Ic4xhhjjDHGGGOMMQXAD3GMMcYYY4wxxhhjCoAf4hhjjDHGGGOMMcYUAD/EMcYYY4wxxhhjjCkAfohjjDHGGGOMMcYYUwD8EKf98jEAw9v6JMwWYz8WH/uwPrAfi499WB/Yj8XHPqwP7MfiYx/WB83249bwEOfcgn7mMwCOqcFx6gX7sfjYh/WB/Vh87MP6wH4sPvZhfWA/Fh/7sD7YavxYzUOcnQHs3mDvBGCwtO8IYCyA3on3DwCwL4AdAGwrbX0BjAGwPf1tOwCdG/42BkCXJo7ZBcABAHrKMXcFMLThfHcHsAu1dQDQreF80fA9dmqwd6G+2zS8d0fEdAIwHsAe8pld5TP1fR8AsF/Dd1W2BTAOwF4AljTRXkvsxzJF9qN9WKbIPgTsx0aK7Ef7sEyRfQjYj40U2Y/2YZki+xCwHxspsh/twzJF9iFgPzbS7vy47eWXX97c90wG8B0ArwL4JMpfuhuAhQAOBHAegPUAJgKYAuAROtFvN/TfHsA5AA4GcF9D++UoT4DtAfwDgEUAXgEwCMBPUB649QCOB/A2gOUN7zsUwKkA1gCYBOCzAH7b0HYygA+hPICjUB7c+Q1tvQCcBeAwAB9B2ZGHA3gYwDQAZwC4G0AfAJ8AMAzAzIb3HtNwHq8DOAjAvwH4GcoT9MyGvzV+5jYAlja8bzyACxq+x/4AjgTwEJ1P47iOBzC14Xs+ipbBfiy+H+3D4vsQsB+B4vvRPiy+DwH7ESi+H+3D4vsQsB+B4vvRPiy+DwH7EWivfiyVStX8+0qpVDqnib9/T14fXiqVDmmwzyqVSiOobZtSqXRSg31CqVQ6iNq2LZVK36fXV5VKpR3o9dfIPrNUKn2Y2kfLOVye+R6dS6XSfQ3/beo7Nto9S6XSpxvs7Uql0rek70cr/MzvyuuDSqXSEQ32N0ul0k7UdkKpVLq0Sv/Yj1uPH+3D4vvQfqwPP9qHxfeh/VgffrQPi+9D+7E+/GgfFt+H9mM79eN2FT/t2Zz/kdfboBwSdLn8vfGJW38Av6C/bwLwmwZ7FMpPoxr5O4A36PUGlJ9O8Xv5PKYC+EeUw6/moKwrq5TbUH5CVil9ATwnf7upwvc2NT5/avhvBwB/o7//GeWnkS2N/fgeRfWjffgeRfUhYD8yRfWjffgeRfUhYD8yRfWjffgeRfUhYD8yRfWjffgeRfUhYD8y7cKPW/IQR9kE4EkAV6DsjEY6NPx3MYB9AMymtgEoh089gXJ4VKPjt8fmmrIUJ6AcRvXHhtdXNPO8U5TI3oPsFQBGSN8uDf3fb1I8gXJo2Tv0t8bx+TvKIWd/bXh9WDPOtZbYj8X3o31YfB8C9mM9+NE+LL4PAfuxHvxoHxbfh4D9WA9+tA+L70PAfmxzP1aTE+cfUdaQDUbZAW/iPZ3aCwAuA9ADZX3XWQBea2h/puG9wwEMQdkJ2wJ4FsDzKOvYDkD5KdSZAP6j4b0HoayjWwdgAYCjUNatPdDQ/k8oD26vhvN6G+WnWY30RVnPNwBlPdu2KE+sSQBOR3nQBqM80f5C79sfwN4N7z2w4d/vALyFsgbubADdUda3HYqyxq3RSQc2vHc0gBMBrERZ57cAwBcbznVsw/hsRFk7NwfAt1DWGU5FOVHSVABPAXgZtcd+LL4f7cPi+xCwH+vBj/Zh8X0I2I/14Ef7sPg+BOzHevCjfVh8HwL2Y7v1Y4dSqfT+vZrPnihH+Sxqom13lLNHL0H85A4Nf+/a0NacE9sRQD+Ukxy92kR7d5SzSjf1mTkGAViN8oArOzR85mqUEx0pe6L8HZY30da34f2LsPn3HIzyxH2lGefZUtiPxfejfVh8HwL2I1B8P9qHxfchYD8CxfejfVh8HwL2I1B8P9qHxfchYD8CbeDHlnqIY4wxxhhjjDHGGGNqyDZtfQLGGGOMMcYYY4wx5v3xQxxjjDHGGGOMMcaYAuCHOMYYY4wxxhhjjDEFoFklxtesWVNasmRJzT68Q4cO0ettt9022Ntvv33U1qVLl2DvsMMOwd60aVPU76233gr23/72Xvl1zf3Dx9h5552jNv7sd999N9h//etfo34bN76X9+idd96J2vjztjTvUP/+/bH77rt3eP+e78/atWtLixcvrsWhasaOO8ZV5fr06RPsbbZ57znj3/8e56bi+aPzYNmyZcFmH+bQ+VjrfFHjx49fi3KyrS2m1mvRVEa/fv3QvXv3mqzFIvtQ10quLbUXtvR6y1Hva5Gvpfq60nHW/Vb32LZma1uLuTXHFC3PYb2vxa2BrW0t1itFXYvbbRf/lOXfDUxub8y16fU09Vl8jcxdP1vyWuq1WB9Uuhab9RBnyZIl2H///Zts4xuMSm82dOF17tw52D179ozapk+fHuz+/fsHe8OGDVG/F198MdjPPvtssN9+++2oX9++fYM9duzYqK1Xr17BfvXV95Je/+lPf4r6zZgxI9grVqyI2vihDm8Ounh5rHQTaWz7y1/+gt133x21YPHixTjggAOa/DwmdS45W9HvysfkjW/w4MFRvyuvvDLY/IBt/fr1UT9+EPfGG29EbRdeeGGT79Nz4s1Z5yM//OENWS8Qlf4w3bRpU812w9xabO/oBTF3QWtvP0hmzJiB7t1rco/TLn2o8ze1vnUN8NpR//Ley+soNw/U77W+6SmVSm2+FlN7v445j7WOS2pP3WWXXaJ+u+22W7DZH7mb03Xr1kWv+X+KpM5Bz6Mlb1aLuhYrXWPV9qv0x0LOT62577aHtWi2jKKuRRNT1LWoc+8DH/hAsHP/s5df83VR72923XXXJo+nn8VBBPrblIMA9PdKLfFafH/a8n8gVkqla7FZD3GA97587sd76ikoAHTt2jXYH/vYx6K2cePGBTsXOcO2PoA544wzgs0PhTSKZqeddgq2PoB5+umng71mzZpgT5s2Lep3+OGHB/uFF16I2m677bZgL1/+XrUxfZjEr3VM9UlurdEx5hv63MMO9q9+nxx8jG7dugX7Qx/6UNTviCOOCDb7mjdIID8H+YHbLbfcEuzXXnst6sdjrMfn75n6/kD8oEVLL5oAACAASURBVOn11+Oqc5U+0NyaaOl5baon9/CWb1b0gSfvAxr5xsdIrSk9Jn+W9uVIyTfffDPqxzdHzdmbWoPcXpDbe/m1HoOvY8OGDQv2Zz/72agf71H8PyY04pVvVu+9996o7Z577gn2qlWrgq37Zi0e3OT+50YRaE6kGvs+9z9IUv8jIfewJzd2qXWpn2WMMW1Bbj/k33df+cpXorYpU6YEe9asWclj8MMOVVMwPXr0SLatXbs22J06dQo2X2eB+H8sL126NGqrpXKj3qnFA5hKr4tF8IVz4hhjjDHGGGOMMcYUAD/EMcYYY4wxxhhjjCkAfohjjDHGGGOMMcYYUwCanRMnpRHjfAbap3fv3sG+4IILmnwPEOeV4epCQKy7HzFiRLA12e3DDz8c7AEDBgRbcyxwjpwFCxZEbatXrw4258vRY7Amc9CgQVHbaaedFuxf/epXweZcAkCct0FzCeRyC20Jjf7R4/Pn59o450UuydfAgQOjtqOPPjrYU6dODbbOA85nwz7k5JxA7EPWpQLA5z73uWCfddZZwb7jjjuifjfccEOwOf8REOfb4DmdS4CcSzJpTC1oSc2urvtU8tNcgnatNsd9eX/Qc+d8aXp8TkTPa1117Fx5L7dO24KcljuX8Jnz3nBSfwCYPHlysDkPgCbC52swJz3Wc+LrLO+9AKIEh08++WSw//jHP0b9OAec5iyqhiIkIVRy56hrjL8fz1nOTwSk88PpvOb8R5oXinO2pa5vxmytpHKw5BLKV0sqqbzzUTWNXgcOPPDAYPNvCyC+l+AEw5r/lPdALoqj9x+c1F9/h+y5557B5oI8WgjgqKOOCvYvfvGL5PFNHl+rYhyJY4wxxhhjjDHGGFMA/BDHGGOMMcYYY4wxpgBUXWI8V4pWwwS5ZDT30xJsXP6ZS7Xp5+VCiTkMjsu4aRlULa3KbNy4MdgdO3ZMvmfdunXBXrIkLunO4etc+vWVV16J+nH4dO67tAa5sqipEDYOHwSAT3/608EeP3581MYhjhxSrtI57sdjzGGRQBwqriHqPM4sHzj11FOjfieccEKwH3300ajthz/8YbB5LqmEg+e7SjhyZQuNqYSWkJOkwsZ1j2NpFMs5eF/UNoXlHUOHDg02r3MglkeqHGjevHnBZmmVni+HJat0tT3DPuCypwBw5plnBnvixIlRG/uB90CVMfHYMro/8TWY914gDiNn6Va/fv2ifvPnzw/2LbfcErWxnGdrgn2j4fgsy95jjz2C/eEPfzjqx2ts0aJFwVY/DRkyJNgqAWfpG0sP2e9APH/aWoZoTFvAe3LuXjxXBjtXOjqXvsBsjt5b87VE2xj2id5z8PtS0nEgljT36tUrauP7m9z5skwuJ1u3nG5zWrPsd9HkWt45jDHGGGOMMcYYYwqAH+IYY4wxxhhjjDHGFAA/xDHGGGOMMcYYY4wpAFWXGNe8N6wh1NKknDeF8xRoPhU+huoCWV/IJWY1F0OPHj2CzTlUNJ8Ka8VVD86f9cYbbyTPiY+hpc5Zwzd8+PBgP/vss1E/fR/T0jpZ1f5xfgn1L2vkWR967rnnRv0OOuigYGvOo1Q5df2enL+IczaoxjSX14jh89BSfpx76YMf/GDU1qdPn2Bzrp+cz1RL6xLjm5PLvVQpOd1qpbmdcuurPfmtpTW6vNY5fxQQ5+vgXC2ac2XcuHHB5j0TiPdJzomje/LAgQODrfmpBg8eHGzOj6O5zlifrntYa+Sn0rlXaZlazj3EOXAAYOrUqcHW3DacFyBXvp1zhLHvdJ5z/hP1Mc8N/txBgwZF/bjkqp7vb3/722DntP/tSZdejR5fr028ds4444yojcerW7duwd5vv/2ifpzPge+B9JqWy3E1YsSIYM+ePTvYmpduwYIFwZ47d27UtrXmNdpaqCY3Yj1S6T1AbkwqbWO7JXLg1QN6vVi5cmWw1Vd8X8DXVj0G/37kXHvqA/6tqnnGUvl4dF/m+xa9tpo8rbkGWjP/Ti1wJI4xxhhjjDHGGGNMAfBDHGOMMcYYY4wxxpgC0Gw5VUqCwCFrHJYGxOFmHJ6kYfeMhtpzaDyHO+kx5syZ0+Tn6nmvWLEi2LvuumvUxnIZDoljGREQh+drSXSGw4+1HB1/Lw33bylJR0oSx6H0WlaUx5zLoE6aNCnqx6W4WY4ExKH1HE7IJd2B2B9cQlfHn2VwKqvj8+dx1e/Mx+AQdSCWwbFc5P7774/65aRh+nlbE7nSm0ylskFezxoWy8fX46XaNFQyNWea6ltP5MqgchgxrzHdM9kfGirMfXNljWfOnBlslXc88cQTweZ9QMtb8/6q+0VrhDDrPOe9IbcGevfuHWyWpOoxdJ9jqWiuxDhfJ1lyrD7gNZAr28rnoeuNy8Nrmey77ror2PUgy0ntLSyRAoDPfe5zwdYx4bXD91EqN2d4jNXXHO6v9xCTJ08ONpciX7NmTdSPZe833XRT1HbvvfcmP9u0DG0psanna1+tqXas2pN8uz3B46n3fLn7Nb7nYImqykv59x3/1mBbP0uPwX1zKSl4b1d/e41tTup+qa2kVbX67Nzvomr2AUfiGGOMMcYYY4wxxhQAP8QxxhhjjDHGGGOMKQDNllOl4LAgldFwaC6Hu3NIMJAPG+fqGpr1O3UM7qeSHQ5te/nll6M2Donj882Fw2sYFIepc1idhunxeWillcb31aKaD9MY9t2c0C0+h169egVbM7Vz6KJWJmE4NFLnAY95LpySxy4njWFbj8Hv0/B+ft/IkSODfd999yGFHr+lK4wVBV4D6itu0znJkhiu8KK+YumHrheusMRtWj2NJScqM6n1GmxrUlW6tDoV9+P9VMef+6kUk6UavCeohINDonUv5OsBH19lt1z9Svf1XFW5WqFzm8dW5zbPKa6+xXuoomPG+y+vAZXtMiwb5T1Uzyn3Xfh9eq3mfiqt5u9WrZyqpddic47Pffnafvzxx0f9WMbE+xgQS5J4LNU3fG3hMVZpI68j/S48X/baa69g77bbblE/PsfVq1dHbU8++WSTbZYEtB4pWbD6IDeXi1aFpTXY0nHIyS9cdaq2pK59QDzWfN+i9zep67PKRHlf1s/i/ZaP15x7/yLOhfa4f1R7b5CSRbeE7C23J1SDf2EaY4wxxhhjjDHGFAA/xDHGGGOMMcYYY4wpAH6IY4wxxhhjjDHGGFMAqs6Jo3o/1rprPgPW8XMpVc2dwO/T8mz8mjWOqinj3DdaYpbJ5Unh3Bhdu3YNtpaWY1S/zjk1OG+DflauJF3jOdZab9h4PPVhpdrq/v37B1u/T5cuXYKt/l2yZEmwOR9JThfIPlSdqh4/xdtvvx1szeGTK0nMOTT4O+v56hhsTeR8x9phnms6zrp2mIEDBwaby0mvWLEi6tepU6dg58q6P/fcc8FWvWuuVHw9lwDldaR5rHic2W/aj9em5jrhPZnXItuKjjf35RxHmhOH9+uJEydGbb/5zW+Sn7elNK6D5uQ94Dk2YsSIYOv8Zc29rhUe65SuW19zXgD9LParrlM+Rio/CxDnhtF8OVw2m68H7UVTDzTvXPi78zVi6tSpUT++P9K8eDyWPM/1usJzndeU5mjgtahtPH/4+Lx/AvFc2n///aM2Lk3O13G97zO1Q/cVXle5fTR3jPZQwrceyJUMzuUN8fg3Dx0X3m9y112+l9C1wvuj/jZI9dPfIbyn8vVUz4mPUUQfFyEvZKVrMecbRn9j1uJ3AH+23n+5xLgxxhhjjDHGGGNMneKHOMYYY4wxxhhjjDEFoGo5VS4MiMuBA3HILZeT1jBsLnmppVRZQsUlNTX8iOUx3KYh/nweWnaOZTQcUq79cqU8eQz4eBr+zyGBGs7XGO7cUmFsudCtXOgnl43V0EIefy1byj7MSZC4LVf2Lycf4LDJXJnyXMlblsRxydVc6GaufHY9wmOhIYksFWQZRY8ePZLH4PL1ADBp0qRgc5n3xx57LOq3atWqYGvI7NNPPx3sdevWBZtlk/q+Ioa7VkuuPDi/5pLdut641LDuK7zH8fF4/1RUusr7MM8fnUs5KUlrhANXK8XZe++9g60SJJbfsIxG2/j7qg94XHhfVh/w/q1t7BPeR/V8GW0bMGBAsB9//PFgt6f1lpsnOu/5ms2SuNz1TUvdc18eB70erVy5Mti8jnSMFy5cGGy+hgHAvvvuG2yWHuq9B1+3WCINxOXSeW/Vfbc9+bQI5ML/VUbJ656vaZVKzIH0/V9OZl/vpORPuT2B106urLSOI48/+y3XrxbUi1Q8Jz1lf/E1TeF7E95H1Qf8Wu/p+Ri8TnXNbs1pF6ohV+o71Y99qLJlvodU3/B9Cf92f/HFF6N+fE3OzZHcb1P+bL3P1et1JTgSxxhjjDHGGGOMMaYA+CGOMcYYY4wxxhhjTAHwQxxjjDHGGGOMMcaYAlB1ThzV97F+nsv4ArEm7Jlnngm2aq25n2q0OYcN55VRvdmiRYuCzdpF1TGyZk11aamSjVpCk7XtWnKXx4CPp3ld+HVraSYbv3su701OB805cfR7c14j1twDse6Xx1xzV7D+keeVahxzZcr5/Hm+aM4VPkf1Ox+fc7poDgI+x9w8Kyo5rT77Tv3IuTf23HPPYA8fPjzqx6XDhw4dGrVxnonevXsHm3MZAcCNN94Y7MWLF0dtL730UrBZ06o6W56TOv+bk2ugaOTWPY9Jbq/isaxU+59Dj8H7/3777Rds3WO4VPKBBx4Ytd18880VffaW0JwS4wx/D84dBsQ54Li8OhD7JJcrjs+LfafacM4Tpddg3vc4z4Duefw+vT7ss88+wW4Nf1RD7rqoexznSuNS3Dqu/D5dA8uWLQs2j+see+wR9WOtfm69cT5BbeP7Et5rtZw8X2u17aCDDgr2DTfcEGzN15TLeVUP18VawPeven/D10kt8875Ga+55ppga76lXL6+FDrHeT2rj+s5X05ujvJeOGjQoGAPHjw46sf3nrqvcxvvAfobITfGqdwbuXxweu2utER9e4P3F772AfEey+Os10/e2zgfifqe++lvTn7Nvsr9hi0Sjd9P51Quj1M16JjzfQWPsc5fvmYOGzYs2AcccEDUj9epXp/5ffz74aabbor6/elPfwq2rhs+R54v+pvz5JNPDvacOXOitttuuw3NxZE4xhhjjDHGGGOMMQXAD3GMMcYYY4wxxhhjCkCz5FQdOnQIIU4aUsYhgFpGO8UjjzwSveZjTpw4MWpjmRSHznHZW4WlABrWreFUKThkSsOK586dG2z9zlxyl8dGQ8ZyUhwNX6sVKTlVpWGbHParoeF8zip54TA4/q4qU0uF6uv4sHRCzyP1XTTEm8MrNYyVw+r4u+QkCPUYXpyTF3BYMcvsAGD06NHB5hJ+/HdFpWo8FziEXOcMh6refvvtURv7lWUIOp94fafKSLaUDKBaGU6tP1vXbEoao/s/77W6xnht8mep5Id9r/OA5UZcJplLywPA/Pnzg81yPgAYNWoUWprm+JH3SpahaklUHs9cqc2UhFTbeGybE0rPx+BzVDkyH1/Pg79nLkS6PcE+1XsAvk+ZMGFCsDl0G4ivM1o6nCVZvFZ0D+L5rLIWhvdCnX98zeTvomuF5ybvu0A8Hh/84AeDzaWugfy9WT1eJystT83rheU3l1xySdSP5XQ673jtPPDAA8F+4oknon6p8vV6XnxOfK0GgDPOOCPYP/rRj6I2vs9tT1Ra6jsnjWFbr4ssb1NJL8MycpaD6/FZyqMSi6eeeirYeo/ao0ePYPP3GjJkSNSP71Fnz54dtbHcvEjkJGKpfVTXAO9ZudLPfE1TmRQfk6/Vuu45PUORaLzn0HuPWu/hOQkb32/w9RIALrzwwmDz/YX+JuTfrcuXL4/a+JnCCSecEGyVR37ta19LHoPnCK/1Qw89NOo3ffr0YOu8veOOO9BcHIljjDHGGGOMMcYYUwD8EMcYY4wxxhhjjDGmAFSt19FQqpRkQV9ryC3DEonnn38+auPM7hwayDYQh4fnQio5HFmPwefLIV4ansWh4ho6zMfkkCkNGeO2XLWclkDPJReqnwotzVVSUVkFjwmHgebCtTmkLhe+pxIn9hV/Lw1L5hA4DYvlz2Nf5MIp64VU1QMgDrVnCdWpp54a9eOQXg4XnjdvXtSPQyW1ugZXWuGqRDrm/D5dpzwP2d8sLQDiLPLq4wULFgDYXOJQD7B/tcpDnz59gs3jqPJU9odWWeG21LrUfrqeeZ/hfmvXro368dziOQdsLsFrCZojUeXvmJOo5o7HY5jb51LkPisnDePQcK26we/TY/Aay0nwcrT0fqv7HfuGK+kB8Z7H0gm9dvOeqWuMr4s8J1QylZIlapVPHksdK/4uvMfp+ebmD3/e1KlTg633bLw2c/cWRaVSKY5KWy644IJgc1UU3a9ylW54Pn3mM58JNlcLA2Jpjs4nnmvjxo0L9vnnnx/14yovLFcAgE996lMA2l+1sdwexGOn48p9WTKlVTMfe+yxYPMY6/EmTZoUbN3jxowZE2yWX6pchOeFSp/Yh3x90/O49dZbg60SOL2GFgW+F9Ox5fsF/p2mKSpUutxI7h5fr5mp33d6DK4uWCRS6Tea+/6mjsE+1PvG8ePHB/vwww8P9iGHHBL142sLz2WVO/Xv3z/YWjGKP5ulz1yJEQDuvvvuYL/44otRG68r/p2h11J+RqH3ublnFikciWOMMcYYY4wxxhhTAPwQxxhjjDHGGGOMMaYA+CGOMcYYY4wxxhhjTAGoWQ1r1qXl9O3cprloOK+Fata4LCdr1jhnBrC5rq4RLeXF+mDNjcHnwdpK1a+xtk11kpWWm2QNnOoFW6LsaocOHcJn5nTqufPMlUvlvDKqteWxzM0XzpHD56HzJafDZj0qv09LNLKOWMsas36Wv79qHHOaz5ZkS7WqTR2rKVTby2uTc0Kwvh+INcCcC4v15EA8L3r27Bm1cTlpnmt6TpxnQkukcqlWzv0wfPjwqB/vA/weAPjhD38IoDrNao5qcglUOt9yx06ViefcCEC8B3E5b82hwSWKtWw1r3Vu0/2H9dG61/K+8swzzwRb9wTWuOt6bo21qfODP1M/n8+V53Mu75LmIKs0rwxf01JlafX4+l34vHiP1nxwe++9d7DVP7yuqs0v1dL5N3KlTg888MCoja+FnCsmlxtJx4TXFeeb0/Pg1zz+ejw+J23jMec5ob7guZQrtzty5Mhg69g8/vjjwW5OKfu2JpcPjts0h95RRx0V7NNPPz3Zj+8r5s+fH+xFixZF/XjMON8SEOeiYx9cddVVUT/eR3W/5fPgeZfLW8E5XvgY7S0nDvtNfxNwfhsu9wsAzz77bLAXL14c7Ouvvz7qx/skj4/eG951113B1hLjnFeO1/OoUaOifryedZ3yuC9dujTYnLtDX/N+UyRy+UT1Ws+5g3L3bHz/z/uc7pvsH70G8/H5d6X6Kvc7sD3n1azkd0al95p6LeEcWxdddFHUNnHixGDzterpp5+O+vE84Lw3Oic41xTvd0B8L8a/5TWHEp8/78FA+tqq58u/i3htA5vny6sER+IYY4wxxhhjjDHGFAA/xDHGGGOMMcYYY4wpAM2SU5VKpYokPrlQsVxJbQ4tyklsuJ+GqqYkOxpSmStPxyFTLNdSqQeHXam8I/U9NZyMj6Fj2/i+lgq1yx1X2/i8eezU1zwOWlKP36dlaRkeEz4PLQfIbSqT4lBkDqvT0sg8X1gSAsTfmX2oUo+2klNt6WflwsZ5vWnoIUucWFqlIfPsL5bA6Dzn96k0g9cfh8iOGDEi+Vlnn3121MZrP1eynkPbtSxsYyi0rt8tpZbzpdqQdl6XGoLK48ph++prnj+6r/N65r1Ww4053FVL2XLbSSedFGwNbX7ggQeCrSWP9bvVksbv35wS4yzf0+sHw3ubhvfyePIx9HN5bvD+pWHj7Dv1D79mn2rpYp4LObkb2yrLzdHSe6yucZZBHHHEEVEbh1Tz2Ok58hjpnE1JjfQYvCfz+Ou9DUsE9HrHvuf1wGV4gdg3fDx9HzN48ODoNc/vtiivm5Oj5fryeffp0yfqN2HChGB//OMfj9pYRsj+ZlkOEPv70UcfDbauAV7rOuapssZ6bU2VztVz5Ouszk+WCer9dmP4f7XSyPejUp8B8Xgdf/zxwT7xxBOjfuzT5557Lmr785//HOyVK1cGW9dAag/S/Y59rcfgfZLHnOWuADBjxoxgs0QRiOcW37/onszH1/MoCupv3qdVxsb7V04ayu9LSeSA2Fe6xrgvzwu9Vhd13BvJ7UF6/8Lfndebluw+9thjk8e49dZbg83zuV+/flG/0aNHB5tTKeg6Yl/rZ/G84NQP+ruSy4q/+uqrURvLppYtWxZsvT4/+eSTwR4zZkzUlip5n8OROMYYY4wxxhhjjDEFwA9xjDHGGGOMMcYYYwpAs6tTNYaLqfyCX6s8icPgOORYw+P4fSqr4FAufp+GRfFr/izNyM5hSyqP4dBSDgvLZffXELtUuKX2a4kKVJWiPsxVq0pVUtGQQQ7zZgkNEI8z+zAXrs3H01BIDrHT6mAp+ZeGSa5YsSLYWpGIfchh6Sq7ao4sra1JZYrXNcAh5VopjqUtGsbKDBo0KNhLliwJtq5Fnk8ays37AEsXNOyQK2Np2Cd/Nw5vXr58edSP16KG/zf6P7dGqiEVLs7zJlfxKPf3nFyOx5xDUHWNcYg/V0jRPZ7HSyupcHgq+03H8vDDDw/2IYccErVxyCzL+7SiC4fD6zzjtV5rqlnnHO7L602lLRzSq5ILDtVl3+m+zPOe15j6O1fxI3Xd1bBi3mP1+pyr8lcpLV0JR8+L5aNakYLHn+ez7h+5vTYl29VQ7pTkTH3Ivtbx5+sYt+l65vmsa4wrgLC8Zr/99ov68T6gc6QlaZwfKgNmX6k0ns+VKxbptY99xzIjIN5f2NYqnexj9qnuhywfVskdS6NYRqPrnn382muvJY/B46GyK5YvsGQMeG8M9HO3lEYf5iSd2jZw4MBgn3baacHW9fbggw8Ge/bs2VEb7728z+SkPIzez+dkp3xvw/NFJV5cMYvld3oM3i9078hVSay171oK9QGvnZx/+PvqPsd7Pa+/3O8JHS/2ea5CZFHGmenQoUNYZ1p9luW9Oqd4P+E9Q8eVq7fxfSIQX6v4szUVR6qascqY+H25SsRMrjKjnu8TTzwRbJZV5mR1WrmqGhyJY4wxxhhjjDHGGFMA/BDHGGOMMcYYY4wxpgD4IY4xxhhjjDHGGGNMAWi2SK9Rc5Yqh517D5DXFrK+WvXfrDFjjavq11l3mitnznpaLQuYKkmt5Tr5u6jWjzWC/Nl6HjmtaktQKpXCuTZHo8ljzpq+XE4i1fayJjFVrlbPi/M35HT72sbHZM2kwnlRhg8fHrWxpp7Pl3OIAHEejtakcV01JxcKfw/WjOpc4H6qKeeSgax91TwDnJOksRQpAOyzzz5RP9b+q26c9bSseed8O0C+zDufB2vFdd3zHFq/fn3U1ti3pXLiNCevEr/O5QdhH+o8YJ/yPqn7bqq0pu677Ccty8trjPNJHXbYYcnz1RLyuc9meA3ff//9yePXmkr2bvUV5wfifU731FQJaiB9Dcpdq/h4urZ5neq+zHOB9wvVhvP1U7XnPL8494aOTVvkEmv8fnq9mDJlSrC1jLaOUSN6PeJ5r2uMj8F7kt5jca4+Pp7uVXz+up9yG68H1f5zbhDN78Ml1xn19cSJE4PNeVuAzXN21Irtt98+XId+9rOfRW2cd0rh8WR/qK+47DT7A4jX1fPPPx9sXc/sL76e6LxjH2gOOO7LuXnUVzyfeG0DcZ6JoUOHBltzSbBfNR9g4z6t+0hLkcqvCcRzivON3XvvvVG/pUuXBju1frVN9yNem7wv5vLX5a7jPHc0l2TufPlenNv0PBjda2t9T9NS5H47Kbyf8V6sexSPGftA90MeZx0/ngts65rIzaf2yjbbbBP2Gp1TvLdoG5fYzv2G5ns5zWHG+xofX4/BOdt4zHU/5dc6l1I5rjRv5sKFC4P91FNPRW08f3iv5d8tQFymXK+D1eQJdCSOMcYYY4wxxhhjTAHwQxxjjDHGGGOMMcaYAlCzmmc5eQyHjqXCEIE4dHX16tXJ47P8QiUR/Fn8Hi1jzSFZKiVJhd/lvlel4XEa0p+TfLVEKdUOHTqE75uT4SgcRsvhtjquzOLFi6PXHLKbkvXoefD4q5/YhxommSrbqqHNXNpRwx/583ieaWhzTsrTkuVwG9dPrgynwt+RfarnzeHIWlqQpTP8/VT2wSVNuXw7HxuIZVgqQ+BQRC7vq+VSU+U6gbj0ba4sJX92StLSUmGwzTluak41J4RaQ+sb0XD8VNl1LUPL643LyQPx2uRSvjpP586dG+xf/epXURuXGOf5qN+DQ1w1FFZLd9eS1NirJIbhdcB7Ta78pV4jeJ6yf3RseZ/msGKWcQFpGTAQr29u0/BmnmvqH947uXxzc0pttnQouu53LJ/VfYF9w/clKnHiMGkNmeZjsG9UytO9e/dg85jz34HY17ly5uwnvabx/rpgwYKobcyYMU2eu57Hxz72sWDffPPNUVtOIrglbLfdduFao9+J/aOh+zye3Kb+5hB/bWO/DhkypMm/A/Ha4dLrek7cT/3IvuN5ovsN79N6n8XnxX7Uezpebyo7yF1zqoUlHCqJ4Gu0ziGW0rKkl+89gPi76j09+5fv+XQ/Zd/kyp7z+Kh/+/fvH+y99tor2Dx3gFgGqPsKSxv5fPV7caoKlXBwCfP2RqVl3nMyNv6+Oi68Pnhu6X0oj5/+5uQ1x9dTnZ+6HxWB7bbbLnx3leayP3Qv5HHgfUfv23lNqG/4+DxH165dG/Xj3yN8T5WTluq+wvOF7xO5bDgAPPDAA8Hm64meP3//F154IerHUjOdS3w9qBRH4hhjS/dLzQAAF7tJREFUjDHGGGOMMcYUAD/EMcYYY4wxxhhjjCkAfohjjDHGGGOMMcYYUwCalROnQ4cOQXuo+UkqLVPH2jPVwLGuLpd/hjWOqulPHU+1wnw81sEqnFdBS1SyxlF1+qkSd7kcCTndba0olUrhuLn8PArnsuDvqvlN+Bis/QPi/AjsQ9VJcinyXC6BnO957HguqS6cP1s1rKyb5DK6Ol9Yj9ta5Rq32WabMB6an0RfM/vvv3+wOReN6nU5Fw37A4j1waw7Va0q55Jg/ahqm1nTr5/FeTO4bOujjz4a9WM/ah4Ofs1zSHW8vL5nzZoVtTXOtVr7t5rj8d7L60335Nx6TuWy4nKmQKwP5uOrn3iNcf4aANh3332DzXNHx5jz4OjanjdvXrC5NDmXMQbifEiaq6Ulc6mkjp3T9PP64L1H9zkua6xzlv3D1xnVWvP1KJczi/dlzc3Deye35XKJaT4K/mzef5pDS+QZ22abbcIeyPk0gDiXm+YE4XnP303XXq4MMe+9/N1Ut8+fzeOv+32qdDEQ50Hj9aE+5POv9L5Hr608vzXPx+OPP46W4J133sGqVasAAJdddlnUxtdwXa88hrxHcd4SIL726347aNCgJs9J93jOLcE5vHTO8DVT/cNlankv13wn7Du9p+bz5/HQ9cVzTcvLP/LIIwCAL3zhC6gVO++8c8hj99WvfjVq41yZfE0A4usCX/PHjRsX9WN/6P7E48XXEl2zfAweE91beb7wegDiOcf3OYMHD4768Xc55phjoja+HvBn67rna4PeH5133nlor+T2et4PdS2mcmLquucx42PoOuLxVB/z+3ic9dx1fywC7777brivnzx5ctTG1ye9LvI6zeXl4rmoOWz4dwHvcXp/xG2cW49zLOpn6+8dvt+cOXNmsDnXJhDPA51LfF78nXW+8PfUfUXncSU4EscYY4wxxhhjjDGmAPghjjHGGGOMMcYYY0wBaFbsDktxqi2Jy+FDGhrOob4aDs6lyPgYHPoKxCFYHEaXk1ho6T+WhfC5a3k/DifTsCgOXU19fyAOodXwu5YqT93ou5zETM+Tvyt/Nw2jY1mLjheHleVKqaWkUCq7yklRUpIBDeebP39+8nw5jJhD9nTe5kKRc9KFWjF+/PjodU76xTKpI488MthazpZlcjruPGYcdqoh3/w+luloqC+PkYZ8c3m+v/zlL8HWMtY8XzXkm8eHQ9m1ZGJjGH5TtNRabPzuubmsn819UyHx2qZzltcBv0/HJFVCViUc/FnHHXdc1MZz8O677w72H//4x6gfyy81rJSPweercgceGx1T3atqSWp+5PYGvt5xeG9uLuh6TpUVV/+wdIb3NZ0X7O/cXsbnrnKbnNSHr6187c5dF1uDDh06hPmh8gteK3qePK4sf3nwwQejfkcddVSwWToBxNeWlORYyZWE5pB+ltsB8X1Vrpw8+5D3eyAuu8r3Tiy5BuK9XCVqLVGaGijPm8br38KFC6M2nr96P8j3I7wHLlq0KOrHYfe5ksH8/fSaxq/ZP7re2D+58H/2le5xfHxdU7z2+bN1PvFYpfawWt7nvPnmm0EK8dxzz0Vtffv2DTZLwwFg2LBhweY1oGOXS6XA5Yp5j9OS07wn8LjmUiKo/IslySyrGzVqVNSP79n0HoX3UP5svc9lSa7OR5bmtTdyvzP5upMr7c7zXO9R2a98DJ0zvN/qXpmS0epanDBhQrD1+pC7T0j1aw02bdoUrhl8/w3E60PhMWFZk+5BPEbaxvOUx5+lWkAsy+d+Ks/ifTJ3b8PnkStJr88e/n9759Nr59TG4XWkY00jTar6VmgRgggDMRLxDczN+xlMTMx8AoOOxIAYkAiqJWjVAPW3rSpKSyLVpIyp8472cq1fu5a9j717PO11jdY+6znrWc/6c69n79y/++ae4/epvI7tZwiKUYiQHnriiIiIiIiIiIhMAH/EERERERERERGZAIuHQu4wkuLQxYnuU+lmxCwQ6TK1ffv2WqZcImVXPXlSuk3SXS7dwfl/dCXNbB18znRJpJs15SMpE0h3LTJz8VqVlGOjmXY45uku/8MPP9RyusdxLH/99ddaTpd+ujym9IZwLaULMOeGrnIZ2ZzulOxTKa17HK9LN8JRdqrR/P4btmzZUsc++zMaF87dF198UcuZKYhtpGsp9yYlUylLpFs67cPu3bub69g+s8GV0rpi8r6577k3cz9zDVEWmNIhruVe1rVVubPmHh/JTnkt90qubdq8tF0cI45lriXugZHcjJI1urWX0sovDh8+XMspiaO9SLdnyqa4fii3K2Us0dmIq+q89NYF/57X0FZwLeZ+G0mL2CafPe0O55FznHuA+yMlBFxfPTf0UlqbkOcu+8s1uMgZt4o9+Ndff9UzO8+0XgaYUtp1+tVXX9UyM9uU0o5/ylOZAYPtpZyKtpzjlW77dNFOW0j4f2ljaC9Sanvs2LFaZkagHBveO7MArUrauL6+Xtd+nue0B5Qcl9I+L/dA2k2+v2UdP/ckr6W0zz7KljiSnLH/fDfOc5GkXem9m2RWPz4LJYOl/L3Wlvme8+eff1Z521NPPdXU8R0y1xClVnwXSZn8CO4xSuxGZzBtYZ4xo4xHnDe28frrrzfX0V7nuu29J4xk0TmHo3N9M5j3LKCtzDOBn3sZqErph0wYfTfKtUDyHYxQRjs6w+aVVl0NaE97e7+Uy8e1JyfNM6I3T1nHcu5n3muU3an3nTDh+1bOE8/utLUp0Z2RGa5oBzKsyEhC3UNPHBERERERERGRCeCPOCIiIiIiIiIiE8AfcUREREREREREJsBCMXHW1taqrix1+tTupZ6Qes+RFpqaxFEsFGrMUpfWS4Oaej7qnlMfRz0pddWj9HSj9OAsp8ZuNG69//m3zO65SLvUtDNt5Z49e5rrGGdlpJNkGxkbibpD/k+uHeoHcx1wrjifeS/O4cmTJ5u6Rx99tJYZFyD136O4F6vUtM7GN+O3cO9k/AX2nfFEUn9P3Wmm6uMzsf18dsbDoI44dZ+jOWB6W16XbXCtZVpdPgvrUr/MNXPo0KGmbjZuy9yLa2tr1faM1lTODfvJWBt33313c92OHTtqOWOMMe4HxzLjGnHP8b4ZN+Thhx+u5UybuH///lrm/Oae5RhkLCfuP65Hpu0spZRTp07V8kgnv2xmtnu033MeGaNipA3nOPHcyvuxnClmaQN76YmzbpQaeaQ9Z39TJ872aRMWmZtV2NS1tbU6Fpm2vpd+tJQ2fgj3R9rkXsyVUV0+J2NX8F0h55pjnvuoFycr4zyw/Xwv4bsU13TuZ6ZQTtu0yr3Ya5tzl/PIcaKtGaWizftwLEbvfGQUO6cX3zHbZBy5fK55x3nePdV7R91ofMUes/7kmqL9yGc9d+5cLc/73Btdh5zr0fvfCKb9HsW4IhmnrLd+Rv3IcVv23P1b5o0Jw3TwCd/tuJ/zna8XdyrXXS8eXCmt7WDfc5zzu8fUyHXCZ813hZ79G31PHq1ZXpfvR/zMuck+jWwHGe2HeftIco2NzqGN2CM9cUREREREREREJoA/4oiIiIiIiIiITICFU4zPXIYWcWnuuV6nPIkuvelSSNnGyLWNbnB0V08XKbpWjeQd7GM+M12h0i19XrfTkWxiVS7Hs3YXcQ3jszJ1aKatZIrxnENey7pMYcoxont5rqNeytVS+ilwc730UsSW0pdTjdJNJquUU83cBVM+dPvtt9fyfffd19Rx3Jny8O23326uo4t+uvzxfnTdH7m3UpaY7Z05c6aWmb62lHaO6R6Za4vjnOuJbv1M/Uu5USmtDC3X9Sr24vr6etetk/ZklHqRz5Z9pPyCrtultPuKz517INfPjJ07dzafabsOHjzY1J0+fbqWKT8ZuXWnTaZdeeSRR67Ydiljm79KCUdvn/fkTqW065TrfOR+m2uBdXzefFbauVHa21EaTn7mmhml4bztttuaOt6P+3mzU6mW8vf4nThxovk790CuWUpSjx49Wsvff/99c92+ffv+8b6ltOOQMlauEb4DpS3k3OR7CdvndaMzLdfBkSNHapkpxjNd6v/+979azr2Y/dpserZhEUnmyEW/R28+/ulePYnCKmzcqtu/Er335430ZRT6YaOpnjciQRpJ7Ud94ufRffluO7rX1ZrDZTOSLyb8fke7OXrn4Hk0sod5X56ntMV5r127dnX7+19mHts4kntuVG7YYxSupdeHvG6j/VjGe8qy95+eOCIiIiIiIiIiE8AfcUREREREREREJoA/4oiIiIiIiIiITICFY+LMGOn2E9ZRd5ipNpm6MuNaUJc9ip1AfSLjcKTemG3ks/DebD/T6vJeqRtnDIJRrIJRKs/N1K6m9q8XoyO1o998800tM15HKf0UnIyxU0qr92eMi9TRj+K2sH3WMVV6KW1KZfa9lHZ++SxcV6WMtbmrmsM//vij9uONN95o6tjvjFPDVNA//vhjLWdchTvvvLOWGVullHa+OCfbt29vrmOqaY5ZxpxgXe77eWMVjFJtc/4feOCBWs79PLIJM/uxzPm84YYb6lrP+3Ff5V5kqkpel9rg3bt313LuHcZNYryZtGP8vz179tRypmE+e/ZsLWdMHK4D9nGROAO9mEcZw4f9SDbDns6bLpXzn7aMcZzSfnGv82zNZ+3FUxnp+7MNxhkg+VxM837//fc3dbw392zGO8v0oGRV8alm7ea7Au1T9uuTTz6p5Y8++qiWc79xzY5iAXLuR2niaWtz3/O9Kvcz78X2R+nkM90u99gHH3xQy3fddVdzHe1rxq7K97apsJG1N4oXsYx7rdqubeZ76CIxiUY2o9fGvHMzijEzb//mTa+8yPeAeefmv5ZGfF5Gz8eYhjm2fJ/dunVrt71eOve8judTns9so3dGlnL59915mEr8oqvZz3n34j/93zLu/V9ATxwRERERERERkQngjzgiIiIiIiIiIhNgITnV+vp6dcsbuRWlizZdeul6RilLKa1MI92rR6nbSM+lfJTedeSGyWc5f/58ty5dpJkedNT+SGq1irSra2trtd8jF8vsC92f6TKYrudMrZpj0msj57OXVjrXBNtIl++e/O7ixYvNddu2bavlzz77rKnj/NJlnf3L9pNVpc5dX1+vY3Pu3LnuPbPu3XffveJ1uWc5dyMXVM5pjgM/j9I8bsRFMd3xuTdfeOGFpu61116rZdqYTCNO2USmbZ+N9TLdKbds2VKlUQ8++GBTt3fv3lqmZKOUdq44rpRblNKm8859SukqJS6UamUd90rKRV555ZVapkyvlHa+R27jozSoTF/83Xff1TIlXtlmrrO0H6tgkfXx22+/XfHvOS5cl2m/2AbnLs8crhOW0/2bc5B1Kc3pXTeSMffO8UXs5Cps6vr6erUpv/zyS1PHzyn3PHDgQC1z3afsjfso57cnnUybTPkT28hzlv+XcoG89kr3TdKu0PYeO3aslnPNMcV4ph/fSDrua4VrId3z1eK/KhXbyP+NpGHzphhfdp+mRNooysXnlYvl9wRC+5XvNyOpcu//8lzku+e8Mr6NSjGvJ673MdETR0RERERERERkAvgjjoiIiIiIiIjIBFhadiqSrrJ0daNLe8+1t5TWdb+UNhsDXeLSRZ4SAro0ZwYHZk7IfvBev//+ey2n2xbHYCSF6mVKyrqrxeyeo3un6zxdqjnG6V7OTEbMcFTK5XMwI9cS3f3ZjxtvvLHbXmbJIpzfXC+U9KUbPSU1HKt8js3ITlVK3yV9I+ttlIlmtLaXkTlhI4xcWtNllp85xzlvo3FbRXaHS5cu1TWWmb2eeOKJWn7ssceaOtqkw4cP1/I999zTXMesU5Q5ltJmW2P2hsygwCxWbP/ZZ59trjt+/Hgtj1yWR3IRjnGuU9ryXbt21fK9997bXEfp2ZkzZ7r9WBUj9+es4/P3skyVMl6zPXlSSgX5mWdfZmijbUu5F20sbWpmIeS4p62kPec62YxzsMerr77afKZ86PPPP2/qTp48Wcscr5R7PvPMM7Wc5xgzvXEcUjpHSRKlc9yjpbQStpRHcr1w3mhTSinl6NGjtZwyY54NrKPMMdvIjI7XOqP1fL27/8vluCauTO97VCltNrx8P0sZ74z8nsBzl23kdTx38169LMX5DskzePTuQ1wX8k/oiSMiIiIiIiIiMgH8EUdEREREREREZAL4I46IiIiIiIiIyARYOCbOTLuXmr55U45TJ5htUDedmn7qDqnzPnv2bHMdNf7U3Gfcm0wTTXr/l/2lRj3r2N+RrnPEPLFrFoWpqUftjtL5MdYNU6eW0j53puljLICdO3fWMlOnltLG32EsjJzDUf95Lec678W4AzmHjA9y88031zLTx5fSxhkYpZO/WiwjTs0i6ef/a4xSV49SUF9tLl26VONovPfee03diRMnavmhhx5q6mifuI9++umn5jo+K+NulFLKHXfcUcvUf2cMDdYdOXKklk+dOtVcx1hY8+6BRdYR43DR5mTsLsbrSPszitXzb5k9S9okfs54XEw5StuTKZ0Zh4X/U0prl9l+2mXaKMYxyzngOGcd22R7H374YXPdwYMHa3nfvn1NHZ8tn3NeVh0/J98p3nzzzVrOPl+4cKGWGf8n98DLL79cyxl/hs/Dce3FdSilPd8yjfjo/3oxqfJ9aBSjieuCaylT6jKmXMYI2mzbu2pGsbBE5G/yDOfeYTnPT8boy7Oe7y20c6P4lfzOmTaPZ1/ei/aW1+X3Fdq8fBbWLSPd/PXEvDE6r1X0xBERERERERERmQD+iCMiIiIiIiIiMgEWklOtra1V17d0EaX72SjNas8VN8l0lXQbpztbpkHl51tvvbWW00Wars/pjkxXN7oIjyRYIwkB20t3L7rR9eqW7SI2ay9d+nqp4Etp5RdM7Zdzzf/L1KSHDh2qZbo1Zppbyq7Yp3TJputiupBzbfH/Pv744+a6999/v9uPHTt21DJlDCnJIjke17rb+BSYgovl6dOnm8/ffvttLX/55ZdNHdf9SP5Cm5Spw/fu3VvLlBmljJX779y5c7Wc0gnKTEYuwPPORV7HNNsvvvhiLb/zzjvNddz3aRM2ex3kObNt27Zavummm2o55TZMFf/11183dZxXum+nLI52mbYsz62cV0IZKe1anq18zjwz2S8+f47NiFXPY76XUOZNiVApl6dQn5F95HvJ6EzotZdwv43et5J5Xc/nlZ2OruOzjOS51xu9OVhkHkWmzEjWxO933AP8XlBKG2ohv6/wnKFdGqUHZ93oPEo715Mx5/5l+5TBl9K+W43upU24nOt9TPTEERERERERERGZAP6IIyIiIiIiIiIyAfwRR0RERERERERkAiwUE2d9fb2rbR6lceNnagbPnDnTXMcUuRn/hJpBpo/bunVr917Hjx/vtke9dqaMZhvUsme8AOoVU/vfiwOxiH4vU+8ti9nzZTwE3i+1+S+99FItM25Gjv/PP/9cyxmv6Omnn77ivRLOL8sJ+5jtsY6xMUbpVzMmzoEDB2qZ8/b8888313FPjHSwIj1GcWQyDscolliPixcvNp8//fTTWmYshtxvIxtHRntgGbAfHI9MG87P88YXWSajZ2f8olJKee6552qZc8oYLKW051PaL85rL310fh7FI2D/R3HeRvPNPu7fv7+pu+WWW2r5rbfequU8n0esWgOf7Z8/f76Wc+zm7cu8sdFG6ahH89u710bTW3Pu533GPOs4p9dD3IJ5x5pztwy7eb2n2F0VjuvqGL0Xs457Jd9hHn/88Vp+8sknmzqemaO4gRcuXKhlfq/h99SE50Ep/bimGZfu1KlTtczvSaX0x8PvD/JP6IkjIiIiIiIiIjIB/BFHRERERERERGQCrC3oJvhrKeXHFfVF+txaStm+pLacw83DeZw+zuG1gfM4fZzDawPncfo4h9cGzuP0cQ6vDeaax0V/xBERERERERERkU1AOZWIiIiIiIiIyATwRxwRERERERERkQngjzgiIiIiIiIiIhPAH3FERERERERERCaAP+KIiIiIiIiIiEwAf8QREREREREREZkA/ogjIiIiIiIiIjIB/BFHRERERERERGQC+COOiIiIiIiIiMgE+D+dUxw+KUrWLwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1440x288 with 20 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "dark"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "n = 10\n",
    "plt.figure(figsize=(20, 4))\n",
    "for i in range(n):\n",
    "    # display original\n",
    "    ax = plt.subplot(2, n, i + 1)\n",
    "    plt.imshow(imgs_test[i, 0])\n",
    "    plt.title(\"original\"); plt.gray()\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "    # display reconstruction\n",
    "    ax = plt.subplot(2, n, i + 1 + n)\n",
    "    plt.imshow(decoded_imgs[i])\n",
    "    plt.title(\"reconstructed\"); plt.gray()\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Improvements\n",
    "\n",
    "Even though this very basic example seems to work, several improvements can be made over the original model. First, we can see that the overall framework does not depend on the exact nature of the `encoder` and `decoder`. \n",
    "\n",
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #192841; border-color: #779ecb\">\n",
    "\n",
    "> ### Exercise (optional)\n",
    ">   1. Rewrite the original class to accept any type of architecture for these (see code below). \n",
    ">   2. Fill the missing code to have a new AE model\n",
    "\n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AE(nn.Module):\n",
    "    def __init__(self, encoder, decoder, encoding_dim):\n",
    "        super(AE, self).__init__()\n",
    "        self.latent_dim = encoding_dim\n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "\n",
    "    def forward(self, x):\n",
    "        encoded = self.encoder(x)\n",
    "        decoded = self.decoder(encoded)\n",
    "        return decoded\n",
    "\n",
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################\n",
    "\n",
    "encoder = ...\n",
    "decoder = ...\n",
    "model = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second, we only used the train dataset, whereas the use of a validation and test sets allows to ensure that we do not overfit our model.\n",
    "\n",
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #192841; border-color: #779ecb\">\n",
    "\n",
    "> ### Exercise (optional)\n",
    ">   1. Write a separate train function, that can be called on different sets\n",
    ">   2. Re-write the training loop to test overfitting\n",
    "\n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "r4gv6G8PoRQE"
   },
   "source": [
    "## **Optional** exercise: Denoising AE\n",
    "\n",
    "Imagine (for the sake of argument), that we choose an encoding dimension which is of same dimensionality as the input one. Then, one huge problem is that nothing prevents the AE from simply learning the _identity_ function (try to imagine why). An autoencoder can also be trained to remove noise from images. This type of _regularization_ prevents the model from learning this degenerate situation.\n",
    "\n",
    "In this exercise, you will need to create your own denoising AE, by relying on a noisy version of the Fashion MNIST dataset (adding random Gaussian noise to each image). You will then train an autoencoder using the noisy image as input, and the original image as the target.\n",
    "\n",
    "Let's reimport the dataset to omit the modifications made earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "uJZ-TcaqDBr5"
   },
   "outputs": [],
   "source": [
    "class GaussianNoise(object):\n",
    "    def __init__(self, mean=0., std=0.1):\n",
    "        self.std = std\n",
    "        self.mean = mean\n",
    "        \n",
    "    def __call__(self, tensor):\n",
    "        return tensor + torch.randn(tensor.size()) * self.std + self.mean\n",
    "    \n",
    "    def __repr__(self):\n",
    "        return self.__class__.__name__ + '(mean={0}, std={1})'.format(self.mean, self.std)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aPZl_6P65_8R"
   },
   "source": [
    "Here, we create two new train and test sets by adding random noise to the images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "axSMyxC354fc"
   },
   "outputs": [],
   "source": [
    "# Load the datasets and use our Gaussian noise transform\n",
    "train_valid_dataset = torchvision.datasets.FashionMNIST(root=dataset_dir, train=True, transform=torchvision.transforms.Compose([torchvision.transforms.ToTensor(), GaussianNoise(0., 0.1)]), download=True)\n",
    "train_dataset, valid_dataset = torch.utils.data.dataset.random_split(train_valid_dataset, [nb_train, nb_valid])\n",
    "test_dataset = torchvision.datasets.FashionMNIST(root=dataset_dir, transform=torchvision.transforms.Compose([torchvision.transforms.ToTensor(), GaussianNoise(0., 0.1)]),train=False)\n",
    "# Create loaders\n",
    "train_loader_noisy = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, num_workers=num_threads)\n",
    "valid_loader_noisy = torch.utils.data.DataLoader(dataset=valid_dataset, batch_size=batch_size, shuffle=False, num_workers=num_threads)\n",
    "test_loader_noisy = torch.utils.data.DataLoader(dataset=test_dataset,batch_size=batch_size,shuffle=False,num_workers=num_threads)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wRxHe4XXltNd"
   },
   "source": [
    "Plot the noisy images.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "thKUmbVVCQpt"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABH8AAAD0CAYAAAACE4qKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzsvXm8VeV1/79AcJ5nUUEBQRxAEBVQtIpzEicwRqMmJmlSU2vbr0maaJIaY5rYXzWppmlN0phok6ix2jhPRJwHnEFxVgRHnHBCZDi/Pw7n4f18uHt7ufcA9+z7eb9evlzn7n322XuvZ9qb9VmrR61WC2OMMcYYY4wxxhhTTXqu6BMwxhhjjDHGGGOMMcsOv/wxxhhjjDHGGGOMqTB++WOMMcYYY4wxxhhTYfzyxxhjjDHGGGOMMabC+OWPMcYYY4wxxhhjTIXxyx9jjDHGGGOMMcaYClPVlz8HRMSgDnxvtYj4cpPP5YiIODYiNm3ycauOfVgN7MfWxz6sBvZj62MfVgP7sfWxD6uB/dj62IdLSVVf/qwWEb078L0eEbFWk8/l/oj4KCJ2aPJxq459WA3sx9bHPqwG9mPrYx9WA/ux9bEPq4H92PrYh0vJSqeffvqKPocy1oiIHSPi44iYs+hvK0fEeov+1iMito2ItyKitmj7OhHxUkS8HBEL5Xg9o+6QNSLi3YjYIOpOqkXEKhGxZkRMXnRs0thvSESsuui7ep7DFx1ft82OiLUXnfdz7bnoimEfVgP7sfWxD6uB/dj62IfVwH5sfezDamA/tj724XKiK7/82S8iPh/1G3tgRAyOiIei3jBOiXpj+OuoO3vkom0REcdHxGERMT/yG79mRJwdEe9FxLCIOCEiPh0RD0TdWbtGxP4RMSEirsf31o2ICyJij6g3kGERsV1EPLJo+4BFv/l+ROy8aNujci1bRBdvCMsI+7Aa2I+tj31YDezH1sc+rAb2Y+tjH1YD+7H1sQ+XJ7VarSv+t1KtVvtX+dvXa7XagEX2gbVa7bxF+7X1/VG1Wm1f+ds3arXaxvh8TK1WO7aN7363jb99u1arbYHP32/jfLeo1Wrr1mq109t5PlX/zz6sxn/2Y+v/Zx9W4z/7sfX/sw+r8Z/92Pr/2YfV+M9+bP3/7MPl/F+vFf3yqYDNIuIp+dtdUQ/fenbR599HxIKlOOaaEfE6Pk+LiO2X4vtvwGZo2fiov/17LurhYVsuxTGrjH1YDezH1sc+rAb2Y+tjH1YD+7H1sQ+rgf3Y+tiHy5mumvD55aiHfJG9IuLhThzz4aiHeDX4fCeORXaKiFMj4tcR8V+xpP6vu2IfVgP7sfWxD6uB/dj62IfVwH5sfezDamA/tj724XKmq+b8qUU92dOJEbFR1EunvRgRd0bEkVHX9w2PiBERsX7U3+hF1N/GfS0i/mrRtmERMTciZkbEExGxd0QcGhEHR71h9Ii6Vm+lqDtz3KLvbh11neF9EdE/Iv4+6gme7lt0zBOj/pZyZkSMiroGcPii8xwWER8s+r22zufjiJjRjJvUxbEPq4H92PrYh9XAfmx97MNqYD+2PvZhNbAfWx/7cDnTo1arffJeK46VImKrqGfy/qjJxx4R9aRMVzbhWH2j3nBnNeFYVcM+rAb2Y+tjH1YD+7H1sQ+rgf3Y+tiH1cB+bH3sw+VEV3/502zWifrbvbWi/ibxlKhn7Datg31YDezH1sc+rAb2Y+tjH1YD+7H1sQ+rgf3Y+tiHBXS3lz8bRD1U66OImBz18DDTWtiH1cB+bH3sw2pgP7Y+9mE1sB9bH/uwGtiPrY99WEB3e/ljjDHGGGOMMcYY063oqtW+jDHGGGOMMcYYY0wT6NWsA82aNas2ffr0Zh3OtJN+/frFRhtt1KMZx+oqPuzZc/E7yTXWWCPb9t577y318VZfffXs88KFC5P90UfNzinWMUaOHPlG1LPcd5qu4se11lor2Ztsskm27cMPP0x27969kz13bh6Vybaw0korZdsYtbjKKqsk+9lnn+3gGXeOqvTFXr0WTwsbbbS4SS5YsCDbj/2I6H5l9Oix+HbNnz8/2drPl2eEahX7ItExlX2Mdhm637x585L9/vsrXlJflb5IBg9eXAlX+wM/s099/PHH2X7cxnE3org/8zsREU8//XQ7z7jzNKsvdhUfEp3POG5yPuN4HJH7WtvBnDlzmnmKTaHq46mywQYbJJtjrfYj+lvXoW+88cYyOruOU+W+2F3obn2xqjTDj017+TN9+vTYZZddmnU4004mT56cPaB1hq7iQ7402HXXXbNtEydOXOrjjRgxIvvMh5OnnnpqqY+3LKjVak0bRZenH3VBw8XouHHjkn3yySdn+z388MPJ3nTTTZP9zDPPZPutueaayV5vvfWybXzg7N+/f7IPP/zwdp17s6lKX+Q1fPWrX0327Nmzs/2KHjR0P7YJfeBZeeWVk/36668ne9KkSdl++hC7LFkRfVFfpvBBXPsY6chLsVGjRmWf+YBCf6ivCB9OIyJmzVpc9OK2225b6nNqNlXpi4R9gi9KI/KX5quuumqyX3jhhWw/btMX8pwX6Xttm5/61Kfaf9KdpFl9savMiUTns7fffjvZAwYMSPaGG26Y7Vf20mDq1KlLd7LLgVZd22i7p1/L/oHj+OOPT/bo0aOTrS/x6O9p06Zl2y644II2j93etlX2vY7+Q0or9kWT06p90eQ0w4+WfRljjDHGGGOMMcZUmKZF/hjzSfBfHf/hH/4h23b00Ucnm/8ipv96S7nQ+uuv367f1X8dY8QC/wXn1ltvzfb79a9/nezrr7++Xb/V3Sj7l6jTTz892XvssUe23yGHHNLm8d59993sMyV7+i9nbAvc79Of/nS239VXX93mb5m2mTBhQrK/973vJfutt97K9nvllVeSzcirmTNnZvtRJjJkyJBsG/vmzTffnGyNSrjooovade6tiv5rbFG0T9m/2jJiMiJin332STajHw866KBsvyeffLLN4zPqLiKXM6gsYbXVVkv2aaedluyrrroq2+/KK69M9osvvtjGVRiy9tprJ3v77bdPNqPkFI6FAwcOzLaxv2n0AsdTRnaV/Zapw0gpva/sy4zQUtkd7z/70zvvvJPtx+9pBNivfvWrZH/rW99q17mbtimSQSpDhw7NPv/ud79L9l133VV4PPruH//xH7NtF154YbLZnsrmibK5wYV9jDHEkT/GGGOMMcYYY4wxFcYvf4wxxhhjjDHGGGMqjF/+GGOMMcYYY4wxxlQY5/wxy4yzzjor+8zKQZqfgnl4aGueEWrhi6qTROTVgailj8irODC3geaKOfTQQ5N99913Z9v23HPPMOW6+J122inZ6kfmDCnL6/Pmm28mW/MbUO/O3Bbbbrtttp9z/iwdG2+8cbJZLaiswgnz/2hfZK4Y5jCJyHM89enTJ9lPPPFE+0+4AnQ0lwPH1EGDBmXb6Afez0suuSTbj/2UOUm0LzI3kObm4hjLPG39+vXL9jvnnHPa/E5ExLe//e1kv/zyy2HyPHlsB+obzne0WVEoIm8T2hd5fPb7rlg+vKtRNjYeddRRyT7jjDOSrblimGvt3/7t35I9fPjwbL9999032cyTFhHxi1/8ItlsI2Vzp/PBtA+uK5iT7rXXXsv222233ZL9gx/8INna3zj+feUrX8m2cX3JfIm6pl6eVTCNMdXBkT/GGGOMMcYYY4wxFcYvf4wxxhhjjDHGGGMqjGVfpqlQhqClRl999dVkU7JVxsorr5x9Zqla2hq6TDmSllQtOp6eE0O5x4wZk21jCePPfOYzhcfvzrBUtJaGZgg0ZXiUnUTkMgVK9Nrat8GWW2659CdrEpRpzZo1K9ks5x6RS/ko49R+tO666yZbS5jze+yzU6ZMWdrTbmn0vhRJMU488cTsM31FqU5ExLx585LNPqalu2+99dZkH3744cnmeB2R9zc9P/qLpeSfeuqpbL/Zs2cnWyVhZ555ZrK/9KUvhYkYP358stdff/1kz5gxI9uPEp+y8ZTbKCnTY6yzzjrJ3myzzbL9dt5552Q/8MAD5RdgMsnVSy+9lGy294iIa6+9NtkHHnhgsrfeeuvCY+t4oGNAEZZ6tQ3b9mGHHZZtYz+48847k835LSKXqlMqSzl1RC77euSRR7JtXPdSYqtr6kmTJiVbpdK65jLGmAaO/DHGGGOMMcYYY4ypMH75Y4wxxhhjjDHGGFNh/PLHGGOMMcYYY4wxpsI4549pKj/84Q+TreWAmdNDS9VuuummbR5PS9XyGNTSr7HGGtl+zGdADXZEnkeGeX00pwzzcGg5T5bi3HDDDZPd3XXWLIFKmH8kIs85wDwUWiacPtay8jwG25pq683SMX369GQPGzYs2Xr/+Zn5C7T8LP2reWSYx4T7dbdS72U5f5jDqm/fvtl+zz33XLKZY0v54IMPkq199Nlnn23zeNtss022H8fR++67L9vG8ZB5TTSvzGqrrZZsLSHOOeC4445L9kUXXZTt153KVH/5y19O9iuvvJJs5uKKyMc8jplbbLFFth/7qfZn5r/jMbS97Lrrrsmues6foramuQhHjBiRbM0Bw3XFwIEDk7399ttn+x188MHJfuedd5JNv0dEDBo0qPB8Bw8e3Obvvvzyy9l+zIOoaxttF1VGS6dPnDgx2bqW45z02GOPJXurrbbK9jv++OOTzf7B/D8R+dh4yCGHZNtuuOGGZE+bNi3Zo0aNyvbbb7/9kj169Ohs2xVXXJHsZ555JowxpoEjf4wxxhhjjDHGGGMqjF/+GGOMMcYYY4wxxlQYy75MU2GJ2LIysyrz+sUvfpHsX/7yl8nWsHKGQDOk/b333sv2e/HFF5OtMiDKUli+c+bMmdl+PH+WJo/I5Qssgd3dZV877LBDm39X2RfvH6V3tCPyNqNQIkZfUYZnlh6G/T/66KPJpnQoIpdEDBgwINnrrbde4X5PP/104e9SckTZSXegTGpBqYjeF8pn33///WwbZR/sK7ofZSosN/0v//Iv2X6Uaalsl58pI1E5LsdRlc6wDw8fPjzZKvuqutSLUMbDuZDjZ0Qu4+GYqX1W7zmZPXt2m7a2zT59+nzSaVeGora23XbbZZ932WWXZKu8h2MeS3qrJG+ttdZKNsuMP/TQQ9l+nN+0HdDfG2ywQbI5hkTk87HOzVVfw3CNonKrf/qnf0r2Cy+8kG3j2Mu5Svfj/HfBBRckm+vEiNx3O+20U7bt3nvvTfbqq6+ebJXvUWKrxzjllFOSfeKJJ4YxxjRw5I8xxhhjjDHGGGNMhfHLH2OMMcYYY4wxxpgKY9mXaSqUGrB6SMSSFW3IqaeemmyGnGv1J4bATpo0Kdl777134bEff/zx7POQIUOSTRnCySefnO135plnJlurqzC0fvfdd0+2VsHpbgwdOjTZlNdpW6Af2WZUXvfWW28V/hbbE4+hUgezdFDqQCmk9iMyYcKEZFNuEJFXtbntttuybZSyMIRd5SmsUtTd4P3TfqQVCgn7QVGFw4i8z1FWe+ONN2b7Ufagx2A1GfZLlfdSHqaVwAhlNN0JypAj8vv1+uuvJ1ulzOyzHHdZKS4ibz8q/6N0jL+rbU7l3N0Rlbay/avUkX5jX9MqpJRbjRw5MtmsrhYRMXXq1GRvtNFG2TZKx1gpVX+LUj6VjlUd3tsDDzww23bCCSckm9K7iNw/rPxFaWZELiWjv7UqGPuwVnBjm+E2yqsjcr/q/HzNNdeEMca0hSN/jDHGGGOMMcYYYyqMX/4YY4wxxhhjjDHGVBi//DHGGGOMMcYYY4ypMM750yQ0Nw011WWlaZmzQbX0LM9JTXlXo6h8rJaILctPceGFFyb70EMPLdxv/fXXTzbz/JxxxhnZfu+++26yjz766MJj9O3bN9mXXHJJth9z/mjJcea8YFni7g7zE9D/zPETkecPWWeddZL94IMPZvuxfClzGETk/YXHnzFjxtKetgHTpk1L9rhx49r8e0R+/5lvQPNenX/++clW3zCnEP3LsuLdHZaEZj60iPIxlXkj2D+0TDtzxDC/0KOPPprtx3FTSw6z/DdLx2+yySbZfswpxN+KiHj++eeTzVxfOr/wfKuG3q+i/GWaP4/9hTm37r///mw/lrnW3DTvvfdesjnfcayOWDIHUHdhzTXXTDZz60Tk/UHXL1OmTEl2WZ4r5mBi/iXNycPS7Lou4VqTedI0ZxrHA52bq84+++yTbI45ERGPPPJIsrmGjMj9w7xL/fr1y/bjGDdx4sRkcz0fkft4xx13zLYxxyTHhNdeey3bT8dywnljww03TDZzFxljuieO/DHGGGOMMcYYY4ypMH75Y4wxxhhjjDHGGFNhuq3si2HTGkJNucrmm2+ebRs9enSyr7vuumR3tLx0WdnU8ePHJ/uss87q0PGXBwz5Jyr7Kispqve5iCOPPLLNv1M2FpGHpqskj6G9LK2rpW/byzbbbNOh71WRIUOGJJvh6doWGELPMOlRo0Zl+zGMXUPc+Znhz2Xl4c0nQxkAxzUt260yvAYaik5pkvqQ/ZTyEpVHdLfy0ir/acB+E5GXnFaZFvufjoGEfZP3WctZU36lcyZ9zjFV/cZjUB6msJ0MHTo026ZSpiqhZaPZJ8rWGBwnef9VavLQQw8lW8tLv/jii8lm26HEOaL79cUGbK8qt6QcR/suS3rTh2VyOkrw6IuIvO/pXEcZE/uQjruUHJWN11X0Ncuvb7nlltk2ji1cl0Tk9+Wdd95Jto5j9CvTNVDeHpFLNbUv8hw5z6rk8tZbb002nxci8r5PKahlX6ZKtPdZvqPsueeeyb7ttts6fbz2orLsjr5jKMKRP8YYY4wxxhhjjDEVxi9/jDHGGGOMMcYYYyqMX/4YY4wxxhhjjDHGVJhum/OHlOkCx44dm33ebbfdks1cN+eee26Hfpt68AMOOCDbpqUmuyosI1kGdeaqY2fOH9WnE2qcyQ033JB97t+/f7LffPPNbNvBBx+c7FtuuSXZzAUUkecA0nOirltzoXRnqGvnPSrL+XP55Ze369iat0RzUTTQ0tBm6aC2mPl/1Icc/5g3gnlFIvJ8JJr3i2MC/avjQ3dj6623TjbHIc01Ql0473NEXpqd97ms3DTHOe1f9P9GG21UeAyeo+YTYXtiXhP9HscO3ouIauf82XbbbbPP7Iv0teaLYZ6Zspwe99xzT7KHDRuWbaN/6QttVx9//HHh8asM5za9B8wBo7nQeC+Zo0fXFLz/7KPMDROR533RvszxleXduc6MyPvl7Nmzs23MN8OS41WBvtK8ZgcddFCytR/x3jLHk45PW221VZs28yFG5OtSrlcjIv77v/872Zxntc/utddeyR4zZky2jf7XecOYqsD5SeeqIvR5vW/fvsm+/fbbs23jxo1LNnOqzZgxo93nyPFW527yzW9+M9ma33afffZp9++1B0f+GGOMMcYYY4wxxlQYv/wxxhhjjDHGGGOMqTDdVvZFiYGGYY0cOTLZGqrJcE+W+L7iiiuy/Rjeq1KH6dOnJ5slGBluGxExc+bM4gvoQmyxxRZt/l3L7hGGpEbk0imGP+sxWAr3Jz/5SbIHDBhQ+FvTpk3LPjO0vl+/fsn++te/nu03evToZGtJVYZ9t7dMfXeA4eX0cVk45h//+MfCbSw1SxlLxJJyvgaUlpilh35jX6T8SOG2hx9+uHA/HQspYaCvu7vsi2HIvEdlklh+JyKfZzheqXySn+lHnRd5fD0G96UfVfbFMuQ6B9DntLUMcpXR0uyU5FDOqr6hNOS3v/1t4fEpJ/mbv/mbbJv6tOi3iuS2VYdjl8q+eE90jKMs/vXXX0+2zolFc6Tef/pJxwP2PX5PS4SX+VD3rRoPPPBAsn/3u99l2yidUjkX1+ocx1Q6Rkk7y8CvtdZa2X70o6ZO4Jqazxla/pnyW5XDUt6m61djujoc29o7ViqUU953333J1meOBx98MNk6NvI547zzzkv2YYcd1q5ziCiWeh133HHZ56OOOirZOl6oJLyzOPLHGGOMMcYYY4wxpsL45Y8xxhhjjDHGGGNMhelWsi+GkTEMS0MpmWWbYbQReXUFhmWpPIm/pdu23377ZDNjuFaJ0JD5rkpR5RetDsQwVw0xp9zgRz/6UbJZpSYiYv/99082Kx/ssMMO2X70jYbLUS52ySWXJHunnXZq4yraPl9em55jd4aSK/q0rC2z4ppy9913J5syvIhimUKRHMy0D7Ztym/KQm/LJGGsVqOV2FjNiGNyd5WWNKCMh/dCK0CyiovKhulH9j+9t+xH9KlWiOF+WqmL0gfKRlQCw/NXqQNlCpw/y8blqqE+ZN+hb3Q85Rz0s5/9rPD4lIbo/Fy0PiqTOHUnuPbTe8A2z8prEXnfoIyPMqKIYlmf/hb9UeZD9jVWhYrIKzLquF4m129VuD783Oc+l2yVf/DadV1H33G+02pp9CPtsnWirlk4Fra3LVx//fXZNqZS2HvvvZN90UUXFZ5HFWC/Uuken/UoY54yZUq239e+9rVk8369/PLL2X70vT6/EfbLsurSCttje6VOXQ0dT3gdZddXdp/Yr9jOKd+KiPj3f//3ZP/rv/5rsh999NFsP1bl0/cBjz/+eLL322+/ZKuU8sc//nGyNQ0M+/Duu++ebE01wv20+vRLL70UzcSRP8YYY4wxxhhjjDEVxi9/jDHGGGOMMcYYYyqMX/4YY4wxxhhjjDHGVJgum1SmTCeo5S25jbbmBCnSqmvJ01dffTXZWvaS2kBqwFkCXn9btYvMc0HNrur9mXNBdYg8xoqGZS9JmR69TE996qmnFv4W9+M932677Qq/Q39G5DmKysqalrWlIj1qe9tcd0P9XVQaWnnhhReSvccee2TbinITqAbfLB1vvPFGssvGXequy/oR8yOoz/g9apqXRhdfRVgumHOE5hVg3oI///nPhcegH5nHKSKfZ2hrn+X3NOcM50L6TtvFE088kexDDjkk28Zz5DXz2FVHfcN5nveHudUi8jnuueeea9dvaZ4R9k22M83N1J38Qdg3Pvzww2wb752u4+gblv4uy3HBdlC23tX2UpRbb8KECdnnp556Ktmax6SK/uVYyBwhX/ziF7P9Dj744GT/4Ac/yLbxnnHtqePk5ptvnmzmLNS14KxZs5Kt+UOeeeaZNvfTsvLMLTJkyJBsG3Nisrx9V8j5U7R2K8trw7U17yXzGUVEnHzyyckeMGBAto3jJueYZ599NtuPbeTWW29N9kknnZTtt++++yZb57N77rkn2e3NX6P51Vo1zw8pu4aybbreJ+ybHL++8pWvZPuxnW255ZbJ3nXXXQuPrXkKeYxrrrkm2fqcwfcIJ5xwQraNa2DmpGLO34h8vNBz1HmlszjyxxhjjDHGGGOMMabC+OWPMcYYY4wxxhhjTIVZ4bKvolJvZeFgZSF0RaGBytFHH51shvhF5OXiNKSTYbsMm9awTYZKs+y4niPR8F6GKG6zzTbZtocffrjNY6wIikq9KwxpnDhxYrZtzz33TPbMmTOTrT5kiCRDnLX0MFEfMgybIc56DIb1abnhonLilAVGLBlO2p1gH1YftPe+sC2Uhb+b5vHKK68kW0uzE45PZWVs2U9VrspyxEXjYneEEhOW+6ZcMiKfP1mSNCJi7NixyWbYscIxlvObSszY3/Q8KD8pKxVN6YRKl/g9SkF5TlVH1xFF/Yoyloglyzy3B5VDc11VJjXprv2UY6H2DbbdwYMHZ9sofaSt7b/ovpZJznVOLOrnhx9+ePb57LPPTrZKTbRtVQGOjUwrcOONN2b7sd2PHz8+28b1INcl+jxyzDHHJJsSzP79+2f79enTJ9kcqyPy9kW5ij5LcEy+9tprs2233HJLsnVu6Cpo+y17tuM8NWLEiGT/4z/+Y7bfk08+mexLLrkk23b//fcnm/6k3C8iYvTo0cmmlEj7F+V/l19+ebbt+eefT/ZZZ52V7CuvvDLbT/tfd2LgwIHJ1nmez+jbbrtttu3MM89MNlOi6LM8t3EdqnJ0jrHaJvmMyDng0ksvzfajX3UOoPzwxRdfTLY+B7/zzjvJ/uxnP5ttK0uP0REc+WOMMcYYY4wxxhhTYfzyxxhjjDHGGGOMMabCrHDZV5F0Q0Ov+FmlQDxGmdSLGbgZlqUZtynZ0hB2ZgJndRoNx2T4olaGYBhZkexNOeCAA7LPXUn2VRSWr+HDDJX93e9+l21j2KXeL8J2wHtXVOUiYsn7ylB6yitUynDBBRckW2VfRWhllO4s+6IURKvVTZ06tV3HYHb9b33rW9k2HSNMc2D/o62SLd7/9ddfv/B4/B77W0QeflskpewO6PjF8OIymQ37mFbtKZJfaTULzpnsp+qPMll2kexLz/3pp59Otspe2J54P3Ts4LxSJmdrRVR6TMkV74lWsTnllFPaPF6ZrIKyhIi8ShEr/mnb3GKLLdr8re4E5aoR+bi29dZbF+7LtZ9W1WIfop9UClO2xuV4zX6oEj/6+tFHH822VXFeZdqEQYMGJVvv5cYbb5xsHbv4mWOoHoMyLVah1WpcbDM6VnONyoqOOs8+9thjydaKw7zmoUOHJlv9vSJozB8drYbL6mWsnhSxpHS2PejziH5uoGkdvvvd7yZbnxH4TPid73wn2To+UGav/mW7YL/Utsn9VCr8l7/8JZYFq6yySpJtUaIVEfH6668nW8c53heeq/pt0qRJyaZcLyKvhMXxUcdlPtPx/mmqEsrF9LmVfZ19VtdRXOdSehgRcccddySbkk71N+W5ej922GGHaCbVG+WNMcYYY4wxxhhjTMIvf4wxxhhjjDHGGGMqjF/+GGOMMcYYY4wxxlSY5ZLzp0xDzNwB1C2qzrmsBCBh+cQjjjgi20aNHnMPqMaPuj7Vk7IsH89d8xcQ1bWyZBu3aU4NXvPuu+9eePwVDXWLZfeEZTS1VCrhPVb9akfKe+t3qJflNi1rfe+997brmCzFXFbmuLtRlqtE800UQX26+qeoDLL2I7N0cExiThUdx5kLhH1b4VirOmn6VLXh3QnNFVaUC07zr3Cs1G38TO275l2itpw5Q7R/0XfU9EfkbYbnrnMf8xuU5XvgmKrjN/X5zzwUvQLFAAAgAElEQVTzTOExWhEt+8s+wXWK9sWiUs5lZcKZLyQiz0XB3AmaH6Fs7q4avP+851p2d+211y48BnNWsX9pjkHm/OG4qDkQ2a+1b7D9MK/PZpttlu1Xlrep6jl/mGdOxziWV/72t7+dbWN/YUlmvV/01x/+8IdkDx8+PNuP56F5YK677rpk33333cnWHCE//elPC4/P9Tfblubo5LUsD1ZeeeXU/uiXiHzcpx2Rzxc/+9nPkq3z2ZgxY5K9zjrrZNuKynarD3fbbbdkc77RnDLM7XLzzTdn27juYb7Tww47LNtv7Nixhccvylmjcz23qX8nT54cy4KNN944vv71r0dEnlMqorwsOf04e/bsZOs8Q9/peoPrUvYdzYvDMZD3RdehbBdl+WN5Xdo+mYdql112ybaddNJJyeb167xdlr+42Wud6o3yxhhjjDHGGGOMMSbhlz/GGGOMMcYYY4wxFaapsq9GiLGGK7VXslUm6WFIWL9+/bJt2267bbIZ3qoh1AypYwiYhuwWlQKPyK+F56HhowylZMilHoPhehpGxpBtLQG7/fbbR8SS4WsrAt5LhsWpjIOhelr2krD9qNSHtFcCplKsopLFGi5ZdvyiEowautjdYHgrw471XmpZ6iI0NJ4Uycos+2oeHNdYdjoiD48tk4IwtFXlBhx7Vd7QndCxh/eW8gDdb8aMGcnWOYJyE5Z6LgsZ59ir43dZeWP2Ux5fJdX8rKHcnBd5DA3JZznmqsm+tAwzS9pyLUJJQcSSpbwblK29rrnmmuzz3/3d3yWbbWeTTTbJ9nvzzTcLj1k1iuYYlWeofIVwXcf1kbZr9g32L92vaP0Ykfebl156KdllZcAVztW8/o6W5e4K7LzzzsmmzFXTOgwePDjZuvbYe++9k/3UU08lW8e4vfbaK9kPPfRQslliPiIfy/U8brvttmSPHj062fpM8+KLLyZbZV/0P2XFKjFe3rKvBQsWpGcBnWP4TKVrBbY/jpNf/vKXC39Ln994//g8onPRpZdemmymKKBsuaOcf/752WeugbTNFUmQ9JmmLN3EsvLv22+/HZdddllELFmWfMstt0y2rhtZ6p3P65xzIiK22mqrZFMeFpFLvfg9vX9FEjN91mZ7Yon5iLxvMpXM/vvvH+2F11yWIobPLtrXdZzpLI78McYYY4wxxhhjjKkwfvljjDHGGGOMMcYYU2H88scYY4wxxhhjjDGmwjQ150+RJpiacc3XQ70ebc1lQ42fauaYU6esNDFLx/H4qhPk8TUPBTXbzEejWlD+lp4v82NQx6faSOr/WG4wYrEOsayc9vKiqHS6wrKIAwYMKNyPx1Afclt7y6qXlXqnP7UspOqAi47B81A9dXeDuQXoY22nqn8vQnWvpGi8KdPUmqWDemfNM3LwwQcnW3Xs5MEHH0w2c5hE5DmiusJYtqLQMYrzGMco7TdPPPFEm9+JKM6XpfeZeZ14Hsw1FJHr5DVXQ1F5aC1NzDltypQp2Tbq4jlHat6aZmvfuxLMNRER8aUvfSnZHO80T+E+++yT7BtvvDHZZXMk5+OIvC+W5ZUpK2teZXhPdF3InDLal7kv153arrmOLcuvw36txyjyt+bBY24bpaikdCvn/LnrrruSfe+99yZbS0PfcccdydY8dtyXY6b2j6J8kNpmmB+yLHcTf0vXQxzztV8yjwm3zZo1K1YkCxYsSHloWNK+u6L5w1qFOXPmxGOPPRYREdOnT8+2leVG4vqD64j+/ftn+zG330EHHZRt++1vf5tstnPNR1f2/NARrrrqqmQfeOCB2bZHHnkk2Touc+xkn9Xxmu9A9JlfcyJ1Fkf+GGOMMcYYY4wxxlQYv/wxxhhjjDHGGGOMqTBNlX012HfffbPPffr0SbaWPWdoF0Mfy0JitaQtw8AZKqUhVSz7x5DOspKbGiLP8Fmeh5ai43WVURbezhBhLXleVgJ7edPe0GCWx9xzzz3bdTyFPqXd3rLsEbm/y+4jw+BpRyxZmrMBpQvdkcmTJyd7yJAhyaZ0JSJi2LBhnf4tLeNZ9Fum47BsrUo1GYp73HHHFR5j6tSpyVYZ0EknnZRshu8+8MADS3+yLYyOJxyXOA9oqXfeM8oIIorHIh1f2Y84v+lYzvlJZdmcJzlX65zWt2/fZD/77LPZtjFjxrR5fErbIqotO9J7znvJdYnOW+x/lH2VzW9vvPFG9rlInq8yO5UDVpkieY+uNziu6XqDYf4M3dc1HfsQ5Qpl6yFtL+w3bDsqhyg7Jvtse6X1XR2WQee4s9NOO2X7sTw6y1BH5KXHX3311WTrOMsxjiWvmbpCj6fjKfsij6d+5JqabTUivxb6W9NL6LOLMe1hwYIFqe3onDxu3Lhk61jDcYll6LlOjMj7xM9//vNs23PPPZdsjqOadqNoDaT9jfIzfR/AOZR9Sp8zxo4dm2xKwCLyvsnj6fuFojQ4ERFvvfVWNBNH/hhjjDHGGGOMMcZUGL/8McYYY4wxxhhjjKkwTZN9rb322rH//vtHRMSXv/zlbBvDtjULODOdF4W96jaF8iuGgGlILEPTGM6qIWAMe9VQSsrKGJq5/fbbZ/vxe2XnzjB7rVLE8Gqt1tCoRKUyuhUBq8CUyb54X7fddttsG6+jqHLM0lAWos3zKDvfgQMHJpthvhF5O2Bb7e6Vpm677bZkn3DCCcnWdjpixIilPrb6qqhftXJVkq4Ax0be42222Sbb75lnnkl2mRSEYa5aUW+33XZLto613QntD5yTaHPOichlwyNHjsy2sboMxzyVYhXNu9q/+FnHaIZA01bZEeWeKjcoqiam4c+8zssuuyyqDCVXbAfa37SKXkfgPWd7VHmSrpeqDK+d7V/bNe+JrjfYztl/dT/6mr+r+/Gzjpnsl1xba1+jHEnhtTVjLdYV+NSnPpVszm9///d/n+13ww03JFulxxw3WcFS7+V9992X7EY1pIgl7yV9otIYykYo09IKZEwvcc4552TbWNFt8803T/aPf/zjbL8XXnghjOkMM2bMKP1M+FzFOYd/j8jbvY6BbNuUres4x+cOHkOfp9mvtC9yvOBzn1bN49qpTC6rqQ8Ix2yVealMvrNUY2Q3xhhjjDHGGGOMMW3ilz/GGGOMMcYYY4wxFcYvf4wxxhhjjDHGGGMqTNNy/nzwwQdJ6zpq1Khs24477pjs3XffvfAY1BprOXfq31QLR50ftdKqu2M5XWoGNU8LcwOp1pA5C1hmV3WzLHevJamLypKrjpxl5agFjFisDy/LJ7S8YI6VsvOhllJLGzM/RUeuqazUu0LtdtlvHXrooclW/7J0KI+nZTS7G3fddVeymZdC23YjZ9XSoGNCka62K/SJVoZ9ieOp5vrQUpdFMC+F6qmZA6is/HDVUQ06tfDM16ClSx9++OFka9lillEty0XGfsS5SvsRx3k9X+rd2dc1v9BWW22V7CuvvDLb9pvf/CbZl156aeFvad7AKnPnnXcm+5hjjkm2lnxmOfGOMn369GQzL4Hm/KlKHpj2wL5RtsZgWXDmQtPvsQ/pnMjPZWuqsnwS2t8aTJs2LfvM9a9SxZw/3/jGN5J9zz33JJt5liLyvBrrrrtuto3zE9c2HGcj8vyQXMPrvWSb0Vx4bDPMn8J5ISLvm7/+9a+zbXfccUebv82/G7O80fGxCC39bppLNUZ2Y4wxxhhjjDHGGNMmfvljjDHGGGOMMcYYU2GaFme/YMGCFP54xhlnFO6nYZYs9Tto0KBkjxkzJtuP4eJDhw7NtrEUbFmYLkNiKR2bMmVKtt9NN92U7Ouuuy7bVlbSmDCkvW/fvtm2N954I9mUsqisheG3KrF4+umnl+p8liUMUdawVDJkyJBkayg5r4/htRrGXBTyrH9vb7h2mUSIbY4Sv4iICRMmtPmd7lyuOiKXDlCqqNJHtpP+/fsn+7nnnis8tpaLL5IJWfbVPCjnoRw2Ykk5ThFFcoaIvL8wXL67ccEFFxRu45zJvhKR95fx48dn21i+lMdQ+QFlCxtuuGGydSwrk4QVlbrWcqiUhJ9//vnZto022ijZlDF1hTluRfHzn/882ZxzdF6kRKW946nC9QflheprLTddZbiOKJJUReRrvJkzZxYeg/Oezmfcxj6q65eybSrNbaBrS86dZfLOqkhxBwwYkGyuNfXan3zyyWSPGzcu23bEEUcke+edd052nz59sv2+8IUvJJv9Up8DuB7WdTMlYUwxoGWi+azC8TMiYpNNNkk219sqMdMx2hhTfRz5Y4wxxhhjjDHGGFNh/PLHGGOMMcYYY4wxpsL45Y8xxhhjjDHGGGNMhVnugl4tSTpx4sQ27f/8z/9cbue0LDjkkEOWy++U6dCXF8wLUlaGlGXQVZvOY5RdU9E21b7zs27jOdKePXt2tt/o0aOT/dRTTxWeE49fpLnvjpTlCKEGvb05KrTEM3MyMYdXVcrTdgXmzJmTbM1L0N5cLGXjA32lOTBMHc6ZmnuMuVk22GCDbBv7BHN3vPbaa9l+HLN4DPUV/ahjKvu65qcjLDk/bNiwbJvm1zN5qWjmZmKew4h8PN11112TvTQ5f+g3ztWan0/zt3UX9D4Q9qFGPsYGzKFTNmZybcP+Vfa7mkOtiA8//DD7zPNln4zIc7SV/XYrwf7C3DiaJ+f+++9P9oMPPpht4xrwzjvvTLbmIOWceckllyR7++23z/bj8XXN8sc//jHZDzzwQLI158/1119feHxeM3O+qb+NMd0PPyUZY4wxxhhjjDHGVBi//DHGGGOMMcYYY4ypMNWo42hWKJRrMOSVoaYREWeffXaytYwmw5DbG8rcXmmXQgkSf0tLWU+aNCnZV199dbbtn//5n9s8RlXCpNuL3mf64Yorrkj2Mccck+3HMOc99tgj2TfffHPhb5WVFud5UB5hOsemm26abJXutVdeR9mSyjZ5TI4d3R22Z95nHRvZd8pkc7y36reBAwcm+/nnny88BksHa7+nJJASE/UpZUx77bVXto2yLx5fx/YqUzae3njjjclm2feIXJJ36KGHJvviiy9u929zfGUb0fZSNrdWDbbrsnUJJch33XVXtm3rrbdONkt4qwTs7bffTnZZKXZu6927d+E2ov2Q5b71+JR9VQXKY7fYYotkc+yLyMeuAw44INvG+8T7TJ9GREybNi3Z7L8qvaOEl6XoI/I1zOuvv55sjsH62++99162rV+/fsnmWlzl28aY7ocjf4wxxhhjjDHGGGMqjF/+GGOMMcYYY4wxxlQYy75Mp2H1AIZGqwyBkqg33ngj27bNNtsk+9lnn012e6UlZaHouo3SE4Y4ayUFhtvq+RJeM0NtuwNlMoU///nPyT7++OOz/dg2xo8fn+zTTz+98Lc0pL1I9tfeKlTmk2FlqI033jjb1l55AOUMKp1g5SD2t+4O23OZ3GTw4MHJ1mqFHG95jEGDBmX7vfDCC8mm9KdPnz7ZfpQL6LhM2S7HBMqR9DMlhQqvv2yMqRp6X+m3a6+9NtlHHnlkth9lPZS1LA1sP2w7rBoXsWRVuSrDOYfzikql2DdYMSqiuD+or1lhjf1Q239RFaeIvG/wnLRy1auvvppsbS+saqWyslZlypQpyb7nnnuSzfEzIl+XUCqm2yibGzVqVLYf14r77bdfsrVCHyvx7bbbbtm2m266Kdn0D+WFEbmvbrvttmzbdtttl+x333032VxfG2O6J478McYYY4wxxhhjjKkwfvljjDHGGGOMMcYYU2H88scYY4wxxhhjjDGmwjjnj+k0LG06evToZGvuFeqTNe9EV6d///7ZZ5bVZN6SyZMnL7dz6gpo3gLmU2LpZuZ9icjvmZb/LmLq1KnZ5x133DHZzHmhuUpMx2GekZEjR2bb2us39hXmHojI81Iw94xZDPOLaP4f5hhjnpaIiKeffjrZ9NWTTz6Z7cecLswTof5l/g89D/q4KHdMRN7vmStOt82dOzfZ3SnnT1mfuvPOO5P90ksvZduYg4S5lIYNG5bt98gjjxQen32TvtHcXjqWV5mivHI6x7CdX3bZZcv+xBbx5ptvtms/zUPE/DPjxo3LtnGe1Tw1rcr06dOTvc8++yS7b9++2X7sf9p3Xn755WSzf2y99dbZfkU57jR/Eo+h5deZb4g+2HLLLbP9ODZyzIzIy8JzvOhO/dcY0zaO/DHGGGOMMcYYY4ypMH75Y4wxxhhjjDHGGFNhLPsynea+++5LNkNZtcxve2UiXREN2aVEgSHf77///nI7p65AWRlq8uKLL2afWR6VYc1jxozJ9qOksKy8Lv2z4YYbtuuczCdD6aaGprfX94QlwSNy36uUxdQpkzmdeuqpyf7mN7+ZbTvooIOSve666yb7+eefz/ZjCWP6Z9asWdl+LEWtZZDXX3/9ZFNuoOXnWQb5vPPOy7apbKFBK88bS0t7JW06nn7mM59JNmVaLDUdUS77ok+1nxL6t+pQFkRpHe2IiB/+8IfL7ZyawbnnnptsHQ8oG6Ssu5XlQpSynXzyycneZZddCr9z4YUXZp+5ZuHct+aaa2b7UYrHdAEqn+RaWeV1HPO4vlQfPPHEE8keOnRoto2yeEqqqyybNca0D0f+GGOMMcYYY4wxxlQYv/wxxhhjjDHGGGOMqTB++WOMMcYYY4wxxhhTYZzzx3SamTNnJvvBBx9MtpZ6/+CDDwqP0avX4qZIPbWW+V2W6G/xPJ555pls2zXXXJNs6v/vueeeZXR2XZP26sd/+ctfZp+pVb/44ouTzRw/ykUXXZR95n1nqenbb7+9XedkPhne87Fjx2bbrrvuuqU+3pVXXlm4bcqUKUt9vO5AWc6bOXPmJPuMM84o3I+5S1jOPSLP4bL22msnm/k+FM3nxnwWzEfD8uQR3S8n2rLiRz/6Ufb51VdfTTZ9M2nSpHYf85JLLkn2a6+9lux33nkn22/ixIntPmarwzULc69wvolo/33mGmNF5l753//932RrX9bcelWA49Pll1+e7FdeeaXwO8wT1NbnBr/5zW+yzw888ECymXdNc9oxD4+ex+OPP97mfldddVXh+fJ3I/J5Y8aMGcl2zh9jjCN/jDHGGGOMMcYYYyqMX/4YY4wxxhhjjDHGVJgeTQwBnBUR05t1MNNu+kXERk06ln244rAfWx/7sBrYj62PfVgNmuVH+3DF4b5YDdwXWx/3xWrQaT828+WPMcYYY4wxxhhjjOliWPZljDHGGGOMMcYYU2H88scYY4wxxhhjjDGmwvjljzHGGGOMMcYYY0yF8csfY4wxxhhjjDHGmArjlz/GGGOMMcYYY4wxFcYvf4wxxhhjjDHGGGMqjF/+GGOMMcYYY4wxxlQYv/wxxhhjjDHGGGOMqTB++WOMMcYYY4wxxhhTYZbHy58DImJQB763WkR8ucnnckREHBsRmzb5uMuKARFx0Io+iUXYjx2nq/jRPuw4XcWHEfZjZ+gqfrQPO05X8WGE/dgZuoof7cOO01V8GGE/doau4kf7sON0FR9G2I+dYbn4cXm8/FktInp34Hs9ImKtJp/L/RHxUUTs0MHv79jEc2kPvSNi9eX8m0XYjx2nq/jRPuw4XcWHEfZjZ+gqfrQPO05X8WGE/dgZuoof7cOO01V8GGE/doau4kf7sON0FR9G2I+dYbn4caXTTz+9PfutEfUb8HFEzMHfV46I9Rb9vUdEbBsRb0VEbdH2dSLipYh4OSIWyjF7Rt0Za0TEuxGxQdQdVIuIVSJizYiYvOjYpLHfkIhYddF39VyHLzq+bpsdEWsvOu/nPumi2+CrEXHbUuy/TtSve/OI6BMRb7axzzYRsUkb29aIiHkRMW3R/5XNImL7iPhg0fYatq0ddX99FLm/7Mc6rexH+7BOK/uwcUz7sbX9aB/WaWUfNo5pP7a2H+3DOq3sw8Yx7cfW9qN9WKeVfdg4pv3Y+n5sk/a8/NkvIj4f9Rt6YEQMjoiHFm3bMSJOiXpD+OuoX/BIbD8+Ig6LiPmR3/Q1I+LsiHgvIoZFxAkR8emIeCDqjto1IvaPiAkRcT2+t25EXBARe0S9cQyLiO0i4pFF2wcs+s33I2LnRdselevZIjreCPaMpWsEZ0S94fSKumOOjIhJi7atFhH/FvXOs1FEfC0i7o664yLq1z82IvZp4ze/EPXrez8iDom6XyYu2nZERHwq6vf2gKj76+GwH0mr+nGjsA8btKoP3RdzWtWP7ouLaVUfui/mtKof3RcX06o+dF/MaVU/ui8uplV96L6Y08p+LKZWq5X9t1KtVvtX+dvXa7XaAHw+sFarnbdo37aOMapWq+0rf/tGrVbbGJ+PqdVqx7bx3e+28bdv12q1LfD5+22c8xa1Wm3dWq12ejvPp+y/vovO47u1Wu2mRf8/tVarrd2O7/5VrVabUHCu36jVapvh83oF19vW3/6pVqsNxz3fcdH/16jVat+Tfb9Wq9W2tB9b3o9/U6vVfmEftrQP3Rer4Uf3xdb3oftiNfzovtj6PnRfrIYf3Rdb34fui9Xx4+Zl59nrE95gbRYRT8nf7op62Naz+NvvI2LBJxyLrBkRr+PztKiHMrWXN2AzrGx81N+MPRf10LAtl+KYRbwYEWfi85lFOxZQdK5rRsQr+Px2xCf6o8G/Rf3N6gFRDzFrvCXdPCJ2iojTsW+viNg67MdW9+N6UQ+lJPbh0rGifei+WKfV/ei+2Po+dF+s0+p+dF9sfR+6L9ZpdT+6L7a+D90X61TBj2uWHeyTfvTlqIcPkb0i4v/ad66FPBz18KYbF33+fHxSiFL72CkiTl1k94qI/68Jx1xWvB0RfaPeyCLqIWAfFe+ecUhE/C8+/zAibo16438m8kYQUddBfkb+Zj82h+Xlx54RcZZ83z5sDu6LxdiP7ovLE/fFYuxH98XliftiMfaj++LyxH2xGPuxbT/2KDvYJ738WRgRf46IH0ddXzco6m8Epy/afmTUS5ItjLr+7NGIuHzRtp2j3oAaWrs9IuKGqOvb/i8ivhIRP4i6Bu4hnOhKEfHtqGe83nPROb4ZEedFRP+I2DfqyY7Oi7r2b5+o697ujogPo66L6xERA6OeHOrwiLii4HxujPobzfZyx1Lsu2pEHBf1hEz3RD2R094R8WRE/Cki/jPq2sDno66NHBKLnbdRRHx90XU07sG0iLhk0favRP1eTI+63vHxRX+fH/X7/6OImBJ1reSOUX9jaD8uplX9OCnswwat6kP3xZxW9eOksA8btKoP3RdzWtWPk8I+bNCqPnRfzGlVP04K+7BBq/rQfTGnlf34fNGJ9qjVakXbyEoRsVXUQ/ra+5ZqaRgRdedc2YRj9Y16putZTTjW8mDjqDv55aX83lpRD/eaGfUEUKRn1P31kRzXflx2LC8/2ofLDvfFtrEf3ReXN+6LbWM/ui8ub9wX28Z+dF9c3rgvto39WOzHNmnvy59lwTpRfzu1VtR1bKfEkhdjuj72Y+tjH1YD+7H1sQ+rgf3Y+tiH1cB+bH3sw2pgP3YRVuTLnw0iYnjU31JNjoi5K+pETKewH1sf+7Aa2I+tj31YDezH1sc+rAb2Y+tjH1YD+7GLsCJf/hhjjDHGGGOMMcaYZUzPFX0CxhhjjDHGGGOMMWbZ0d768p/IO++8U3vllXr5+pVWWinb9tFHi3NFrbrqqoXb+L2VV14522/evHmF295///02t2lUU+/evZM9f/78ZC9cuLBwv7JrKaNXr8W3lr8VEbHaaqsl+8MPP2zzO3oevEZu69OnT6y33nqlJd3ay9tvv117+eV6nqgePfJD8h6tssoq2baPP/442T179mzzOxH59el9pN/oa20vc+cujhLkefAc9Px5H3VfthGeu56H3o+itqQ+5D3Q9shtw4YNeyPq2d47zaxZs2rTp9cT86+55prZNrbnOXPmZNu0XzXQ+0IflPmY/tlwww2z/fjb7A/aLnhv9fx4Xtw2c+bMbD/ed+3PCxYsSDZ9qtelfZg0vrfJJpvEuuuu25S++N5779VmzarnruMYEZFfg7Z79hdem7Zf3hPtz9zG+7rWWmtl+9GHvD+rr756tp+eI+H3eL5l/bksWpX7lfmsbG4YNGhQ0/ri7Nmzkx/L5hK9XvqE56p9Vu81YVtgny3zD32s58Q+wT4bkV8Lbb3PPA8dK+l/3iv1I7+nx29s22STTWKdddZp2rzYWNtoP+Lnsjar94t88MEHydZ7wvuw2WabJVvHdd47XSsQjpn8jv4W0fnzmWeeKfwO50yi58t7pefBdtusvkgf6jny+srGhTJ4PXr8It/rHMNjrLHGGsnWNsc2oveOv81tul/R8SLyNsK+rOsAoufI8auZ4+k777xTe/XVVyNiSV+VjRncl9vK7m1ZXy9bl3AM5Tb1AY+nbYbjPNuFthneZ46t+nu8rqXxIxk8eHBT/PjWW2/VXnrppSXOS39f/Vu0nimaAyKWXE+yL/K+qg/Z73kMHQ/K5jr+VllfLJvP1N9tnXtEPl5oW+K1DRw4sKlrm9deey0ilmx77ItlbZbbtC2091mb91P7Pdc6ZWMg73vZ+ojtTq+Z/tY+xmOwDek5ca1fNB9HRAwdOrTTfmzay59XXnklvvjFL0ZExLrrrpttmzZtWrIHDx6cbeNCYu211072lltume3XaGQREZtvvnm27fbbb0923759k60NoU+fPsl+/fXXk62NjIusddZZJ9v21FNPJbvswX6jjRb7pbH4b7Djjjsm+8EHH0z2xhtvnO236aabJvuuu/LKdI1z/P3vfx/rrbdeNIOXX345Pve5z0XEkoMhF6j9+/df4nsNuHDT+8rzfPrpp7Nt9HdjkRYRMWTIkGw/3v+BAwcm+8UXX8z240Ci7aXxYiQi75S6QOV56P3YZJNNkv3WW28lW9s+74G2R07wM2fOnB5NYvr06bHLLrtERMSee+6ZbePC5LHHHsu2FfUdHQxfeOGFZOuDxgYbbL8n1soAACAASURBVJDsAQMGJPuv//qvs/2mTp2abPr4ySefzPZ75513ks3+G5FPeFtssUWyTz311Gy/oofbiIj33nsv2fQp23tExBtvvJFsXSA1+un555+/hP87yqxZs+L73/9+RERMnjw528Z+NGPGjGwb7+Xbb7+d7LKXkltvvXW2jb5n3xk3bly236OPPprsN998M9nDhw/P9mPf1HvHsXH27NltfkfPv70PMjruEu2LHPNvvPHGpvXFWbNmxSmnnBIR5XOJvrDs169fsukr9puIiGHDhiVb7+0222yTbN7PoUOHZvtx/N5jjz3a/HtE3id22GGHbBv7La9LX1Y9//ziyqOcIyPycZRtnH1Pv6d+bIw/55577hL3u6O88sorcdxxx0XEkot/fua8EpHPGZzzdTHM/q33hHPS9773vWSPGTMm2+/dd99N9t13311wJfn8rGM3x0aeI9chERGHHXZYm9+JyNdVhO0qIp9DeO4R+Rh2/fXXN6Uv0ofarjl36EMVr533RPsafV+23uMxdH3Ee7LbbrslW9cenGM4f0XkaxaOp2UvBDlnR+QPMU888USyOd9G5PdAH3Y49jTLhxERr776alpL6EMu2yLXDRG5XzlX6XnzXug/PPJhjHOrPqvsvffeyeb8qefEh2A+30Tk4zzbha5LuAbm2BqR+5/XpS8jeQ/0fvCab7755qb48aWXXkpjiI53Rf+oGlG8ntH1AP+h8fHHH8+2sS9y7cS1b0TuN/YBHQv5bKvny3mWY5y2A/pG2zT7La9Z10ccLzgGROTt8/LLL29aX3zttdfipJNOiogl2x7HKB3f+VzAlx3aFnjf9UUIn7d5fJ1/uBblfdfxm/e9bP3KsZj9MiJvWzpmc33H5xi26YiIhx9+ONn6LMH2NWPGjE77sWkvf3r27JkG3+eeey7bxo7UeGvfgB2VjVsHKDaMe+65J9vWeNBtnEcDvliJyCd9vmjSf/nmRK4OoBP/6q/+KtlsqBF549TBiYszDt76AomLYe0YjYmp7C3+0rJgwYLUQfSesPPpfeU1sBNxkm0cv8GRRx6ZbRs/fnyyeU06oHKQ4QOSLtr4wkJfXnBSLPv7L3/5y2TrgMoBgRNC4180GnBS0fvBB85msvrqq8fOO+8cEXkbishftiq81xyU9QUD77UufDjoHXPMMcnWxT99xzFg7Nix2X58+NEH36J/wdOXfWxP+oDGF668V9pnecxnn30229ZYuBX9q3dH+PDDD9NDoV4PFxz6cpRtnb7QBT59qv6dMGFCm7a+HOV50dYJmA/h++23X7aNLw34kH/TTTdl+3HeuO2227JtHKu4YNf+rC8Pi86xmdRqtdSW9IU3xxD2h4i8bW+33XZt/j0iv0a9Br4Y4vGPOOKIbD+2Wz5c6IP97rvvnmyNRuNYdvrppyf7jDPOyPYri7LgQrcsQojz/4gRI7JtZS8GO0NjDNFFI19S6WKwvS/feJ/Vv5xP2Lb1BQ/vHReh+uKMLzr0/nNtw/WRPmiwna2//vrZtm233TbZvB/qFz4A6JxZNkd1lPnz56fxXcdCPjzoupPXwD6kL5A4j+jCn/MK1046NxP6s2yNpw9dvHdXXrm4gjIfKiLK105sZ2wHel3cpi+G9KGmmTTuR9l4rg+LvJ9la3+OT/qQzjGJ+7FPReT/0MWXc/oswXlX14IHHHBAsjnGc0yJyNcC6oOi5yl9Ecjr1H8IKlord4ZVVlkl9X9tUw899FCy+VwRkf9DAs9Zo8q5/iiLjOXYrWNCUVCCoveL8Bz5opf/yBiRr2s5dkcUvzTi3B6R3xttZ/q81iw++uijtH7bfvvts21c12n/4Mtxzi36fMRxr+x9AMcabTMcYzkmaLvgb+k7Cr6YZZtR3/OaBw0alG3jOMP+p880vFe6ttdnl87inD/GGGOMMcYYY4wxFcYvf4wxxhhjjDHGGGMqjF/+GGOMMcYYY4wxxlSYpuX8mT9/ftLyMQdPRK5V1LwY1BdTd6laOGryNBEbdfjUSKoGlvp0agFVb8ukfKqHphaX2j3VHzOZrh6feVKo1dSkYzxHPX5D69vMPCMRi3XCmtSZ+mHqciNyLTQ1vEcddVS237777pts1aXSN7znZUmjqeHUJF3UWmsOEmqE6RvNycEEm5oc+bTTTmvzPDTBKfWj2pZUC9tMGvkcttpqq+zv7Eeq/eZ9Yn4X1UMzX4DqiXfaaadkM1fJI488ku1XVFlMk6FSB6y5P3g/2Xf0mnkM7WMcZ5i7SdsdtfV6Pxo5O8qy8y8tPXr0SHkFNMkjxwLV5PP+MU+V5skZOXJksjW3AXOBMLeFtleOT9TFa7tijhPtA9RJU9fN3CEREdddd12yVQvNMZB5FJijJiJvq9pG9B40iwULFiQfaZtizji2r4i8rTO/i+aoYP4eTcLMtsG2fdlll2X7MR8Xx0C9R0yeqduK8jUxt0tEnutN+9E+++yTbM7p2q84j+iY3cinUFa1Zmnp2bNnGq807xVzqWi+BuYGY9vW+ZN9VnMRcH7ieKpjAnMFcT7W+YhjoY4dzG3AOVL3Y//TvDX0Kfu6ni/nU13blFVG6yi9e/dOfUDbLscMzT9UlENMj8FxTK/nwAMPTPZ3v/vdZLP/R+RjMsc7zRHFz3pfmWOLc72eb1n/YB4hjutlc7P25bJ8Rp1h3rx5yUe6hmAeMs2TxLwgbF+6Dr3vvvuSrTk9OP8xx5P2sRNPPDHZjYS4EUvmsuL3/vSnP2XbmBuP+VTOP//8bD+ODzqH8dr4W5rDi/ljdHwry5nTUXr06JH6lSYs5/pM88qNGjUq2SwqoDl5OEfqfMl7RB9qnhfmlOG6WH3NcVefN3nvmCdNnyU4R2q/4fMs53fN01iW/6pZRUiUVVZZJfURXduwr2guQo4jfBbWgkdsp2X+4fpAC2ewPfF4WriB46HmOuS+PIYWQOG4yfYZkV8n9zvooIOy/biO0rxymji7szjyxxhjjDHGGGOMMabC+OWPMcYYY4wxxhhjTIVpmuwrYnE4l4ZiEy0NylAmhtxqCB1DuzRskSFhDEnWUDSGvzFkVGUKlBbpMaZMmZJshlyWlUMtKytHiYWGA7I8qoZjNsKyVQrTGRjeriFmDFXTEEOGt1PqxZK/EXm4u4YdF4Vx6n5FPtT7wJBRPQZDPxm2qaVvGebNkPuIiO985zvJPuuss5I9ePDgbL+y0qGUYjSTXr16pRDKBx54INtGWZaWY2UoKSUAek3sp7qtSKalJeEZnsn+oRIDhmBq2WKOEQwZVZnlL37xi2RriUz2K8qHtPQ2y1NqWGgjLFjLNHeWxpinEii2Iy0vTRkAr1XLxZeVoGWZSo4DGobK8+J+6ieiIdqUI/AYKidkf3700UezbQyL59haVt5YpaBFMsTOMm/evCVClhuwdKeO75QmsB9R8hSRh1dTehKR3zPKeHUsY9seM2ZMsu+9995sP4bQq4+578EHH5xsLfH9xBNPJFtL3p577rnJ/s1vfpNsHTfpY+3rjflaJSqdgWXCVbbIuUXnD8rAOC7quWkIOmH/4P3XEt8cC9mfdV5kH9YSyQyRp7xBQ+k5/nHuj4i4/fbbk812pVJNnqOOP8tCgrnSSiulsUfHBa7xKJmLyOfoyZMnJ5sSlIj8GnS9wXUdUyKcffbZ2X6//vWvk81+qHMR5xnKzyPyOYz7qZSOfUilJpS6cW5WiRnPkb6OWPI+NouVVloprc917mNbVwlJWT8llNSNHz8+28Y5guOfyrmKUiRwXIzI5bcTJ07MtvE5ifdy9OjR2X708V133ZVt4zPIjTfemOwJEyZk+91xxx3J1jFZn9eaAaV76if2I10rXH311cnm2kYlfly76njNcZLf03GSa2G2HZV9lbUlzn1ca5eNyfqsyPGC16V+KUvnUHaOnYGSdr0vHCtUesc2xnWpruu4JqIMMiLvV5xPdR1VtA7QtsX3CzqnsQ1xbayl18sksjwvzrv6fkHndaIy1M7iyB9jjDHGGGOMMcaYCuOXP8YYY4wxxhhjjDEVxi9/jDHGGGOMMcYYYypM0xLG9OzZM+kpVdNIzSHLD0fkGmVqyVVHyzKwml+H+jrmjdAy6My9QN2hlhSkxk/ze1AfzeNryUKWz9Vrpl6YOU60XCY1hKprbWism6nn7N27d9Jgqoab+lXVd7JUIUvXqSab2lnVj1NbT1tzf9BvzM2k7YXtQDWnPF+WINf7z+vUcoV77713sq+99tpka8lmaphVs9ns0n2k0S5Uw81yzapLZbtnngE9T+bv0XZCDTqPwVwfEbm2me1ENbtsJ8ybEZFrZ++///5kswxkRJ7DQPXQhOOWlvxl29XxonEt2kc7w1prrZVKAWvuFfZFzeVA3/B8dHxi39GcD8yJwdLcOtawz/G3NL8Q85bofWUuGvYj5g6JyNuxjh1sg7vuumuyVUP+3HPPJVvnKG0zzYK5RjQnD8cG7YvMEcB2vueee2b78Ziaw4X3hbm/1D8cA5lLgKWnI3If33nnndk25hGifl7zuTCvjN4P5tugtl5zCPK6mPMoIuKee+6JiObnHGn4R/O08Z5MmjQp27b//vsnu2jOj8jzFunYctxxx7W5Tfszfcr1keY20L5DeHzm9eMYEJHnRNCccuzPRM+X86mWLdY8Ts1g4cKFKb+CzmfsN8yJF5H3Q+YWY86JiIjDDz882erDouvRvJaHHnposi+99NJka1/71re+lWxdn7J9cl7Vc2K/2X333bNtzK3CMZ7tOSLPf6lrm2Xhw4g8XwxLX0fkaw9dlxCOtZ///OezbbzvOp7yfrLNaq6///f//l+y77777mSfd9552X6cF48//vhsG33A+ZjtLCLPA3Psscdm22699dZkMzejlqHmml1z5CyL/FsLFy5M7ZFzckR+T3RtM2LEiGTzWU79xDmHa7+I/Nq5ntFcOxyj2Vd0ncBnQH3eZE4YjouaB4/3WNe/mpezgT5Tcpwv6+vNhLnwNI8g1/u6Bih6NtP9mGPrf/7nf7JtHIu5ZtFcjuw7fL7TZwlu0xxrXEvw+UHz9bAN6bMk5wvOrVx76fE1n6r6vLM48scYY4wxxhhjjDGmwvjljzHGGGOMMcYYY0yFaZrsa8GCBSlETcvAMtRbwwpZ5pSh+Bo2xTBOlfEw7JLyDA35YygqQ5c1LI5hcypTYKgpz0lLFjIMTyUMDHlmmK6G8DJEUcMBGyGGzZR9LVy4sDAsfJtttkm2SiYYxsdr0NA6XoOGM1KOxNKT3/jGN7L9+D22A20TlLWodKyoJKjK2fhbWmKzKOyU9ykiL92nocjLqgTj/PnzUziw+qos/JRyAYYfUo4RsVha0fgtwnKmlBNpqDHvE/u6hsIzFJRlLyPy0qYMF2ZbisjbpEqzdtttt2TffPPNydbS9AzpZNuKWHwfKXfpLB988EGSe+k4xnBolWyxbChDwvWcGfquMhTec8oYNbyaoe8MvdXQaB5f5R8cO9jftB+1V8570003JVvDdzk36DlquHCz6NGjR5rjdJ5pyPoilpQMsTQ22562bbY5lUNTKkdZpEqiGE5PiSFlKBH5vKUh2pRE/PnPf062jh3si1rinOfLNQP9G5G3IZX7Nq5Fw/g7i46jDTieUHIYkd+T9krJdU6gJIeh4xpyzs8cuzSsnOXntaxs0fiv6zmG+F9zzTWF58ExQaWGlGRqX9S1TjOYP39+6uMsEx2Rj2Nlaxa2Ad5H/azzPMcr9i+VZLMvX3TRRcnmfBuRj+WURkTk18LrVPkOv6dzCCVclB5oW2Kb1v5RJq/uDKuuumqShGpf0fGPcO1PaZ+WkD777LOTfckll2TbONbweHr/KPX6wx/+kGxd81LGwbQWEbmkjvOYjv+c7/We/+3f/m2yzznnnGSr3JbrHl3D6NqjGay88srpmnQsLBt32F+43teS22zb+jzDNsO2rT7k/MZz0nUs5ymVkzL1B/2k18xxUtdH7Ld8LtXnHa6bdU2un5sF1zZ67Ww3OlfxuYPXW3aeOrfye2Uyc46HvO+63uOaSCWrPH9u077CY3LtHZG3SV7nzjvvnO3HOVmfVcrGt47gyB9jjDHGGGOMMcaYCuOXP8YYY4wxxhhjjDEVpmmyr1qtlsKZNJSJ4WpaNYnh2QyXZUbsiDwEUMPnGWJ1yy23JFtDcxlSxVA7hSFsKvti2BdDuTTsleer18KQOO5XJjvSagozZ86MiGUn+1KZCMMi9Z4w5JAhfVpF4oYbbkj2tttum237/ve/n2zefw1rpkyLobGsiqLnf/HFFxceY/z48ck+4ogjsv3Gjh2bbK3iRfkf2yPDOyPKwx811LRZzJs3L4UUawhjkWwuIpf18L5rKDxDXTW8kdc4efLkZLPCWkQezsrjUZoTkbcnDTvnbzG0XkN9Gd6rlWUoHaPURsNCKUNSiU5DZqEyus6wYMGCdI9UBsExU/3LUHyGsqqMjSHnuo2S1bLqcGwvDMNViQfDte+7775sG8cSjs8qU+CYMHz48Gwbx3z2bQ1Z5zillRN0XmoWPXv2TP5SmROlL+rjIumr+oroXMB7S8mknsfAgQOTfd111yVbQ9BVJkQou2SlrrIqTxquTz9y7qAsISIf21nNI2Jx2yiSaXWEXr16pbau4wdDyTX0nfJM+lfHfcpzdJ5h1RTKvniPI4qrkGi4fNl8xLGD46n2Dc67CkPVOU5pBRrOG9pPl0WFoTXWWCP1Aa1QRumvbuM9oQ9V5k9pospVOB9Rdq3VUOlD9htdR7FCkrY59g1KPSnHjsjHcp3DWV2PvtfKcdp+SDMl0GThwoVpXtN1Ccc1HU85dnEsPOWUU7L9WP1L+xjve1F12oi8EhGfaVSSwvvJ8TMi4uijj042+5Q+W02dOjXZKgu54oorks3rV0ls0TgVsaTkthnMnTs3pd1QeTf7jsoiuf5j29PnQa73dO6gDx577LFk6/MO1wpck2qbILoWYz/i+Kd9lnOVPityrGUbUYkUz1crqC0L6V5E/bwbKRtUcshr0jbE9QbnGX1e57XrfE6f0NYUBuw7TC+h949jiaYV4DMd761WXuU4r/MDz59rBq0ayd9uPOM3UGloZ3HkjzHGGGOMMcYYY0yF8csfY4wxxhhjjDHGmArjlz/GGGOMMcYYY4wxFaZpwvgePXokXZuWNKQeULXA1JhSH6vlP7lNtZXU8jHvgepXmVuExyvTYGqpWmoyWcJdNaPUk2oODOZiYN4M1UZSA6n66oYeVvM3dIaePXume9bQ5DagTlhzADDXCe8x8xVE5Hlf9FpZYvNnP/tZsrW0IvXvv/rVr5L9ne98J9uP9+UnP/lJto2lPqkr1txAzAHDHD96/jwG9dMRuQ5b262WXG4WvXv3TrlBVM/Pc9V+Sh9TJ6x5KJh3RDXQhx9+eLKZ14n9QY9BvbLm8KAPtAw883axfCnHg4j8Pqu2nhpbthkts8g+y9LYEYvvm+an6gyrrrpqKqmsPuTYpfeEumPmFNDyw9pOCfPysFyp5uthXiTekz/96U/ZfieddFKymUMjItde8xw1jwL7kZaqZb/i/dDcZGzTmluqLH9FZ6jVaqmNaa4XXofOEZwXmJuLc05EPj+ptp/XSP24jmUc69kvmYNHf0tzJrE9sSQyx4OIPM+PXjN9x3wJ2v6J5oW47LLLIqLe5rSvd5RarZbah+YK5DihOT04njBXgOYb4P3n/BmRXx9zJeg6irmIinL3ROS5XrR/MFcG/aTtlmOM5nqgr7hW0nxUvE62dT1+s5gzZ07KIaF9nXnfdD3FnCTMQaFzDPORaO7Av/zlL8nWnFGE4zXbla6V2Pd0/uWal/OezuGcqzQHzMiRI5PN3EOay4j5NXRNqO24WSxcuDC1OW2/zKWh83djLo3I15DMixMR8fvf/z7ZJ554YraN+e+Yv1LbL3NIsT3de++9ejmJz3zmM9lntjXmKNKcVGyH6keO5WwX2p7YxnVNqrlGm0GvXr3SOKo54fgcpeMk50GOtbru4lqTfo/I81BybanjKccrHl/HB66htc3x2ngfOd5E5POx5nk59thjk805UfOu8dlK26OeV7Oo1WpprtF8YMxZqOsNPttz/tDncPpE2wLnGa49NDcQP3NO03cDXL/qGM31Jf2vz7c8pp4H1058xtE8mjyGrm00R1xnceSPMcYYY4wxxhhjTIXxyx9jjDHGGGOMMcaYCtO8eqixOMxcw7eK5CQRedgXQ/0ZOh6Rh16phIHh4wyh1tB+hqkyZFTDJRl6qOG9DMtiiVYNS2Momp4Hw7kYZqlh4yw7qZKkRtib/m5nmDt3bioJqbIQSrtUksFtvJcMw43Iy+mpTI7hmAz309KHp512WrIZgqcSBYaIaqjmVVddlWxKDTQEj6Ga2h4Zbs0wacrX9Dz0nqpkp1n06tUr3d8yOYv2U8oF2D8oB9D9NEyY4d5szyrjZJ9gSdr/+I//yPa79NJLk63yA95P+uCnP/1pth/DdjUElrIIluNUSQpDNVXCMGfOnIhobl+cN29eCu9XaQXHUP1Nlklm2LeGnPM+qDyG7YLHU8kW4Xlou2YpVg235bUwdPnWW2/N9uN4oeMKJQ0Medaxg21VZa2UKDYTlkPVeYZ9oKyf8n6qnILh1toXeXy2Zw3RZgg1x6t+/fpl+7GssI6plDXtvffebR47Ivcj5bcRuQSDYdLa/tmfdW5tyBS1jXSGnj17prFG1x78rGWLCe8X+2VELnMaO3Zsto1SE45/KnVgG6Hs5Itf/GK236mnnppsvf+UIfJ8uc6JyMvTqsSA18YSuWUyfm0jLJXdLGq1WprfVbrHNZ7KGSndYPleHS8oS1U5Ne9R0diq50HZgK4b6HuVArCk96hRo5Kt6xeOw9oeKe9l21QJEMciLS+t97hZUEarkjf6TiUTvF7KlbV0Oq9Xr4nzH21dy3K8uvrqq5Ot63uuo1XGQxkdfadr1DvvvDPZhx56aLatf//+yWYb0nmIMhftzzr2NoMePXqkeV+lM5Tf6G/z/nG9oWs6popQWdlee+2VbI5PKtMhbDu63uJzKfteRD6ucX2qYyGfi3RMppyU7ZaStYjFa9CIJdc9y0rSvsYaa8Suu+4aEUtK2bjma2+qF11X83ld7zvbNsdefTbl/eT8pmsbjstaLn7fffdN9nXXXVf4W3x+oEQvIr/OMqkb99MxVNOGdBZH/hhjjDHGGGOMMcZUGL/8McYYY4wxxhhjjKkwfvljjDHGGGOMMcYYU2GalvOnd+/eSeenenRqIbWUHbXT1HSqjpZlJlXDSM0qdcCao4Iaa+rpVO9JHR61hRERw4cPTzb1hP/1X/+V7UftsOY5og6V+nzNM0K9p6K5DppBz549kx64rDy96hlZOpNlJLUd8DN1mhH5faYOWPWx1EbTv40Svw2o/2Yei4g8r8yRRx6ZbNVCU7uvem3qhelfbXNsI6o/Lsuh0hnoR+0rbPeaP4T+oX5Vy7jy3mreAu7Lvq66bOZ3YV4LLfVO3bTqdNlOmLdA2wxzwmguEGqJec16b/jb9H3E4lKWmgelM6y00krpXFXnz/vP8s8ReRumb/Se8Hp0fGLuCR5DteT0Nb/z6KOPZvtRq645U3htHON1TGZupkmTJmXbmB+G+nKW9ozI5xQtP9psPXWDjz76KGnNNbeSatzJgQcemGy2Uc0rw206nnAuZG4QzUnCHEAsk6tjHvfTNsN7y76u+YW4TfOQ7Lnnnslmnz3qqKOy/ZhHQzXzjXPWPAOdpXHeHCMi8vuguVPYd3gvNT8CxxrtY7wO9gHNRch2wHFX86Qxj4mOV+wDvE7NgcB1iR6feXG4RtG2zvFf5xBdNzSDHj16pONq2fP9998/2bru4ZzNtaquzThelY1xXIvoHMN5kHkUNbcNz0nXNmxn9JvOe8xXwbwxEXneKeZi1PGU90rH62W1tlmwYEG6fs2FxHwxmq+Hc9CPfvSjZF944YWFx9B2z7Lh9IHmqGK+E85VOj5wTtbcQ/QJ+xSfFyLyOeWzn/1stu1Pf/pTm7+la2/6Ufsi+2mzmD9/fnpOKMtdo+fC+8f8dpprhz7kfBaRtwPmheLvRhSXcNd5j79Nv+u+ZflOuSbn+UXk8zZzb+q8yuvU8XPHHXeMZcGcOXPS7+qzDe8F562I/Fw5Nuqzyi677JJsHW+L8sxq22Z/4fjNPD4ReVvQa+GYyvUGx8aIfCzWZ0mOh5wLNY8az/Hggw/Otp133nnRTBz5Y4wxxhhjjDHGGFNh/PLHGGOMMcYYY4wxpsI0TfbVq1evFB6l0i6GKGvoO2VCDPtXGA6lsgKGJrLMn0oiGDrGcrQaBl9WmpjXwhC9r371q9l+DNXlNUbkIbjcpiHHvI8avtg4L5XTdJZGSJqGvPJ39J4wpJTyLQ3r5zEYBhmRh38y7O6QQw7J9qPEg+Gdn/vc57L9GE547LHHZttYzvLMM89M9mGHHZbtx7A7yvgi8jA+ltnVsFCGxWvou+7bLGq1WgqL1PvMcEQND2X4KWUEGsbOcFwNdWX/oA9Yzl33u/LKK5Ot7ZllEVX+xHBoSrE09JPyJA1/Z/gw91PJC8PIVZrRCNFuZlnUuXPnppBibSef/vSnk82yvBF5OPAdd9yRbA1NZwg/Q2gj8r7I0HoNcx09enSyGQ77wAMPZPuxvWg4LNsnz13LxXOc5PVH5L7hWK7lQSkT1FLNWvq9Way++upJKqz9jdIalRNR7sPxSkOSuU3nT953yjK0D1BaSd+rpKdsDrjpppuS/alPfSrZKhllGPsXvvCFbBvbDccHnYs4Fut5NNqClr3vLI1rV7k15zsN2ed5Umaj8g+OoSqDoK/YVzSUnrKRK664ItkqUb399tuTPWHChGwb1xtsOzqH8Px1rGXoPtumlq2ldFXbtM43zWDVVVdNsgmV8nPNqPIlzjnsk5x7IvK1DuVB7Z3k0gAAIABJREFUEfnYxVLBKuHk2Mjj6zqKfUPnKa5LuN6llCsi74cqt2A75hiqfZn+1Xlbfdos1l577Rg3blxE5G05Ik8JcNxxx2Xb/u///i/Z1157bbLvvffebD/2Iy17Tn+x/d54443ZfhzX2O91fc/jq0yV8xPnQh3X2BY09QTbAo+vcjnOPXr8ZbFGXW211VI6BB0zeW4qj+WYR8mqzt18xtR1D7/HcUYlZpRk8p6ozJKfdR1Fv3He0LUNZai6jf32oYceSraOyfS1rrFUGt0s+JyhaQUobVPJFudsziU6z3ONqhJZXiP7kUoAObZxDPjxj3+c7cc+oH7k8U844YRka9vl+euzCj+zjR9wwAHZfpw7dEzVdVtnceSPMcYYY4wxxhhjTIXxyx9jjDHGGGOMMcaYCtM02deCBQtS2JNmM1dZFWEoPuUUKidhmKqGYzLcilm7VYbBaiIMZ9RQ37322ivZGuZNiQGPx9D5iDzkW0OEGXrI0DYN5WOomIazNcIDNbysM6y88srpvmvoNSUZGkbIMGqGmev9p39HjhyZbWO2fYbWaajx4YcfnmyG3TETfkQuBdTM+GyPDPlWOQ9DBsuqVfE69XzLwj2XVTjmhx9+mEJEVfrCe6EhrAwdZQU9DU8n6mP2K4bpap9lqC7btu5HH2uIO8N7eZ933XXXbD9Ko/SaGZ7O3546dWq2H8NOVerQqHanEpTO0KNHjyXGpQbsY3pPeB/Kxji2U5Uj8fp4L7Ud8LcoOdLKGfxtlVzQv/yeSnspG2HIfUQeDsv2rfMO/avh7NrumsXHH3+c9SXd1kDbDqVtrAKkUiyGKOv1FoWdM3w8Iq/8xjFPq+vRB3pN3JdVo3T+pBxa+xjnTI5bOm7yXul42wg/137eGebOnZsqtnF9EZGHi2u75/jO+6Vtj31H511KhcsqxlD+MXHixGQPGzYs24/rLT1fyld4/1ixMiLvYypJYjug1EQloxwTtJpVWZXTjjJ37txUpVTXWZzLVX7J+a2oeltE3pfLqj6WSSfZzjmmqeScMhSdizgGcFzXVAz0oUoqOAYUjfF6DL1v2raaxUcffZT8qM8IbIu33HJLto1rdVYP1Epn7EdllZ24tlGZCO8nfaCyDbYnrTbJ8YJrZZ2nuH7VcZ2/zXWzyrL5TMNnn4glK/s2gw8//DAefPDBiCivAkj5cER+rew7e+yxR7Yfj6lyU95zVnzSuYjVNjkeaX8j2gfYLjjmqMSPzx1aiZA+ZH9TaRflijrWqqyzWSxcuDCtn/U5gDIwrfxHaTPnI+0fHEf1GuhHrhUuuOCCbD9KPH/1q18lW9eXlFmqf9gXue2cc87J9mNlP+2LlKQydYnOrXymufvuu7Nt6vPO4sgfY4wxxhhjjDHGmArjlz/GGGOMMcYYY4wxFcYvf4wxxhhjjDHGGGMqTNNy/vTo0SPpxDUPDbWP119/fbaN+jfupzpaaog1BwD1t9Rdaik2lv2jnpAliyPyvApaApCadmryNB8Gz0N1otTRsvyj5qah1nuHHXbItjXuj+br6Azz5s1LuYu0HC31znr/qdV86aWXkq3aSeZ50WulvvrII49MtmqCmSOJ3xk4cGC2H7+n5TypnTzttNOSrflxqMPWtsScBcyPoNp66r9VF9vM0uCkd+/eSXOrfqT2VPXdzCVDTTVL1UbkeQVUs8q+w3auOTjY/9jHNA/CrbfemuxvfvObhefLc3r44YcLf0vLrfK3eZ2aT4DjkWpvG7pnHbM6Q+/evVN+Ec1RwT6m94s5CzjeadvmmMk+FZH3HeZ80PK9HP84hmq5WJ4/89dE5GMj2462W+ZP0XwbzJNCrbmWw+Y9aOQcaFCW16qzNOZDLbvNMURL0rLMLfX82hepEdd2z7bAe6TjDvsAx2yd+zjeau4hjgPUyO+4447ZfuzPjZLNDTh2so/pGMPz0NxDjXlK8y90ht69e6e8DNrHeZ6aI4RtiuepYxC/p/ldmLOCZaO137Ps9cUXX5xsnWeZc0tzQ7DPsa9rfiHmxxgxYkS2jX2d7Xvrrbf+/9s7m1jLqqpdj3NOnVOg9sxtmfApqBiDJKBIVfFXFKAUxlKEKJZIiJEGSkNjYmIkxgaiNggxGpWoqMSgoBKpQooCCiIUvyL/JkYTkptow9yuGKrOz74NsyfPfIq9P+Gsc7/rzvu01qm1au215hxzzLF3xjtGdx3t0S1+vZcPwdLSUtuLPYesSWE/yXfnOa9DjrPrQu3du7cdM3byeiWsw+L7MYb2+HMO6W9Yw6Sqr4XhtULbYrzr+lqT1usrPfNQrK6utr3e+wxjZMdhrLXEWM6+hX/v2bOnO8f6JLwH12VVP9/33HNPO3b9LZ7zu3Bt8vuU67BdddVV7dg1crjm6B88V7Qh+1OvhyFYWFhoNuYW4dyzWbusqq/bRJt1zbBLL720HXtuGEfQtzIurqo65ZRT2jHX+q5du7rruK4uv/zy7hy/77BWkz+LcbjXGOuwfehDH2rH/l7EeTrppJMmPseQcF90DMl43HWqxvXzqvrvjrYFtj23v73iiivaMWsy3Xzzzd11/H7MusH2m/yO4PXBmOiPf/zjK/57VdV5553Xjv19nb6YdTS9F7EWketVDVnftyqZPyGEEEIIIYQQQggzTX78CSGEEEIIIYQQQphhBpN9LS8vNznCtFamTrNk+hpbojv1imlZlh2x9S/TPZ1Kz+diihnTy3wPp7cz9ZCp0W55ypRnt/5k60amOfq9KD1hqlzVyymRG9XGzxIPjh3lBL6W8+b20kzxs+yC8gWOlyUvRx999Cve36mNTK23hIdSLNqmbY7phGM53Jj3v//9r3hu3PZ7DFO7PVcex6FYWFho0php6emWDDKlmJItjnlVn35IOWBVP8dcV279ynHhfFj2wPRxPy9TKdkuc1rbbrcu5fMyzdJzRfmTU27HqdJuablexvZtiQefzTIqyhQ4JobvYEkez3Hd+71p6/Td9v9MbbUtTWqZ63Rzpna7ffikdsS2F/oL+/xp7ZnXw2g0au/iNGHKqm6//fbu3He/+91XvM7riHuGffakttJOLadUhL7Ma5F2YkkdZQv8XPuHJ554oh1/+tOf7s498MADr/iMlgDSvrgfV1Xt27evqoZNkZ6bm2u2ZJ9NP+a0dY4J5QeW07GFsdPFabOUZGzfvr27jq1lGfdQ5mWOP/747m/6r5///Oft2HbF+IUSl6peUsg1NU027Ra/ts8hePHFF5vtOZ1+2hris1F+SQlBVS9ftS9hzMtWzrYXxpq8n+eQsjjv72yjzLngs1f1bYkZP1f1+wvvwXXt53Jpg42QC40Z70P2hRw/S4MZn/PdKa+t6vegbdu2dee4Zxw8ePCI5xmze/fudkw5jNt479y5sx27XTnX+o4dO9oxZSdVfezk1tO0a76/49xJZTOqhi0rMWZtba35Gu8x/J5jyTBtnfG9x44xwI9+9KPuHGMF7k2WRfIeW7Zsacf23ZzTa665pjtHeeyjjz76ip9b1cdw/g7I+aXc33I2rjfLaDfCn1b9K54Z39vyJY6f34n+l/up78H96YMf/GB37re//W075rr3/sx3Zzz8+OOPd9fxO6LjV8YSlGzdd9993XXcg73f8fsiP5vjVNWvvzPOOKM7N7SUNpk/IYQQQgghhBBCCDNMfvwJIYQQQgghhBBCmGEGk31VvZzK+453vKP7d3YuYBpbVZ8Kz3RZp5VTruKUKqZ9saK+O10wfY8pt04BY5q507AnyYScLswxcCV/jgdTrZ2GTSmTJRzjv4dMy2T1dssziFONOV6U3DhNmJICps1W9Sm1tANLx3gPyvMsDWBKsDs/MF2ZUganS1IidPHFF3fnOIe0JafN0n7Y4aRq41Kj//nPfzbJmlPLaWNOs+TYMm3aMhumhU/risVjp2MyhZVj5nXEbgeu0M/0d6Zmeg74jE7XZwou0/Mt9+T9LSsbp4kOuRZXV1ebPVoWwfebJh3gmvL7cE7ZAaOqXy9Mc7bv5rlp8lOOiyUX/H9c6/Y/vM5SUEJf67HhPb3W2Z1iSObn59v8HXfccd05pnt7/Nixg/uTuyZxnbqr4Sc+8Yl2zHHxZzH1nhIGy0G4/twdjWt90n5Q1dsau+dU9enWTBX3eqbc0/KucYfGIWV8Kysrbd/3femDLA2mrVNCbL/PVHLHLNyfKLeyrOXWW29tx7SRaR3m3G2K96SswGuFz+i91fY5xnN46qmntmOvvY3YF9nty/EY7dx7NNcGJYuOcbluLP1m9zXug7Zdfha74Hq9Uspgm2PMwvvZDuiTPf7cNyZ1Na3qJU2Ov4bsfEk2b97cJMCMwap6WQTji/H/G8N16c6HlFvZlun/uLfs37+/u47nKPewZIjSJftaSvC5T7jzHvcGx8q0BfotvzPnznsr5dxD8frXv77JEG2/lKz6Obk//PCHP2zHlmzxe4djMvoWdvP9zW9+013H/8f7uZMW70HJZVUfc9GWPve5z3XXTSpfUNV/32Ts7rXI74qWl7ur41AcPny4rZdPfvKT3Tnaum2I33kpR/eapaTOkkn6LI67ZYQcC8rK7aP5vcO+nTELn8NSWs6xy0vw/tx3Pd+MEV3uwXvTeknmTwghhBBCCCGEEMIMkx9/QgghhBBCCCGEEGaY/PgTQgghhBBCCCGEMMMMVvPn0KFDTedmjfLZZ5/djt3ajNpgah9dK4D6Xre5Yy0fautcQ4ftKKlvd+tD6imtV6T+ne0T/Ux8F9cx4bU8Z00f73/FFVd058at0q3vXA+j0ajpMV3Xh3VHrJnnM/A6tzSkDtI1nViDhHpb6/9Zl4X1W9xmm9pl10xha13qv12XgK38Lrzwwu4cdd3UJtte+FnWtG5Ue+mFhYVWl8F1Wrj+3KKU+nHa6Jvf/ObuOs6P6xZwHqlVd5tC2j3v4bVCLTx18FW9Xpy6XOuhqbH1c7AODDXa9j/U2rtGzlhv7XWxHpaWllpNANsNa25Q613V1xHgmnA9AD6rtepcz/RJ1lPTRvh/rLNn+1bXfuLaZA0J1z6h73C9k0mtTMc+cgzHxjby/2Ituk0v9xLWqquqeuSRR9oxx901EngPr1PaMG3I48V7cM90PReOp22ddbU436yTUtX7bNev4Frke7pVLPX6brc69j9D1hx53ete1/T9999/f3eOY25fSD/BGkncE6r6Ncz2z1X9XLHmhdcY1xFrInqfZRzlPZ61XviMvgdrEbguCt+Zz+RaGfRN3ncdjw3B3Nxc81He9+wbCX0c1xN9VVVfF8R7GOdwUvxY1ccR9E8ef+I1P6kdNGtT+W/Xg+O7cV3bXvhcjus3qp7haDRq3y9YN6qqb7Xs9T+uE1TV+zW3U2btku9///vduRtuuKEdcyzsu9lynS2+HRvS13r/YR0hrnXXGWG9ngsuuKA7x3iOa5at7qv6teg21xsxj8vLy82/s2ZOVf/dkXU7fe0vf/nLdszvl1V97O/9knEFfdU555zTXcdYc1odU+4/3os4roxBd+zY0V3HOfSeSLvgWmR8UNXPr++xEXWbqv61L4/r1bk2LuNGx2tcf4wbHYfSl7leFq+lf3XcznN8xmn1aFmDr6qPS26//fZ2TJ9SVXXvvfe2Y/4mUVW1bdu2dnzppZe2Y8eojHVc29gx8XpJ5k8IIYQQQgghhBDCDJMff0IIIYQQQgghhBBmmMFkX0tLSy0F1XIrppG6hS9lHUzZcntpprA6NZEpYEz9dEohJQdMuXW6GVM6Latimi2f16mzTINla/GqPuWMqV1O5WZ64B/+8Ifu3Djt3inM6+Hw4cNNGufx55i4Pd0kGZBTajnmTotjmiDn2ul5lA3QlmxzbCPstG6+G9Na3VqP72UpI1O7mbbpFEfKw5xC6zbpQ7Fp06b2uZbeccycSknbZlo411RV/04eF8odt27d2o5/97vfddfRnpia6fRnnrP8ibbAtN2bbrqpu2779u3tmJKUqj7Nkn7Fc0XZkduVj6VkQ0qHDh8+3N7XchKuebafreqlRbRny8PYUtU2S2kCfdy0VvaTpA1Vvb3Yd7N1Jsfc636aJI/j89xzz7Vjp8nSVi19smxhKNgO1enYlCBajsixZvq75Qw8ZzniJFms5ZO0Dfo1p4tTjmOfyr+nteulX7EtcE44p/bLnFdLbGzLQ7C8vNxS1S1F4Hx4fdDe6O88rrRtS6C4JhgrWILJFtW0Zbfgpv93TPHhD3+4HXMdufUtn9FxD+VKnDc+X1U/NrZ933MIRqNRi5m4Lqp6u7YslX9T5mTZF2V9jp24b9EX2o/R573rXe9qx4xlxu8yxlIs3pO2472T4+/9nXEJ41+vefozS9Mc5w/FyspKs2+PM+N9t7Z/z3ve04737dvXju0L6T/8vpQucu04Hj7xxBPbMeMG+zFKTbxO+Z2B+6n3kGeeeaYd29fShrg3fOADH5j4HJaabMQ8rq2tNR/CMg5VVc8//3w75pxV9eNHOfGuXbu66xjzTpN48n5ucf/www+3Y9qSx5j3cAzBPZd+5Hvf+153HW3Qz8uW5ozXbS/82/egzO9b3/pWDcXy8nIbG7diZ+xv6R1j1mm2TV9mv8xxpy/z9xHOAefOfpNzTP9Q1X8fpbTyoosu6q7j94Jvf/vbE5+D37P8zpxvr71p8t/XQjJ/QgghhBBCCCGEEGaY/PgTQgghhBBCCCGEMMPkx58QQgghhBBCCCGEGWawmj+j0ahp0tjiuarX0VrfTT059XpuxXvccce1Y9dKof6TrRBZT6iq19BRn0edXVWvwXS9jQceeKAdU//n2gOsbcAW81W97pQ6PreYY1tW1w0aazyn1eF4taytrTVdpMeYmlK372XtBs7hY4891l1H3blrelDfS92vNcHUgfKz3DaVzztNF089PW2sanLdjapef0sbca0Y6m9dE4Z68CFZWlpq7+/noXbWWlxqW2nPnivarzXQrNdALbzHlj6Cn2VdK/9m6+Cqfk3wOVzr68Ybb2zH9j+T2ie69gbf2bWSrOUfgrm5ufaZtj3alGsY0bapQfeYcE24HgdthP+PbWur+hard955Z/fs5JZbbmnHH//4x7tzrDfD1r2uUcR2uvbXtE+2q3a9K9blcG2DjWz1PrZbtnut6m3RdecmtUR3nZwDBw60Y9dK4bhwvt/3vvd11/3+979vx6yB5do0HM8LL7ywO/fQQw+1Y/oR+x/uyQcPHuzO0a7ZQtx7MNeb62247tEQvPTSS22fcI0QzpPHlezdu7cdsyZS1cv1+6qqrr/++u4c/6b9uq7DpHonroFEv+ZxffbZZ9sx6xK4vgTrDbnVOG182lxwPTte9J4yBKxn6BpGxG3mWQuE8YD3Ir4P57Oq9y2MN7zHHHPMMe2YY8c1WdXX13DMyLU8rYYd66Rw7Vb1dawYu7o+Bf2N93fHFhuB43vauutl3HPPPe2Ya9a2xpp53mcYb7CGh+vHcW0ytnGsTp/vOoKsubVz585/63ntV9hGmj7B9Z8Ia1JVHVkLZwg2bdrUYowbbrihO8d3+PGPf9yd47pijOJYgWPCfaSqt23GL657RR9HP+k6UFzD9h1cz1zDXkf87uLaNpNiBNe0cq0bMs3frYeVlZVmH25zz+9+/D5X1a9T2pvXEesB7d69uzu3Z8+edkxfbJ/AmJVz7++VPOf4lfE+19RZZ53VXUe/7D2Tc8Dr/NsAf/dwrOHxWS/J/AkhhBBCCCGEEEKYYfLjTwghhBBCCCGEEMIMM5jsizIFthis6tPf3A6P6VFsK8nW31V9eh1b11X1qVJMsaUEwPenpMqp0UyRdcotUz+ZzuZ0QEo1nEr5pje9qR1TtuEUMKaPetzGKb1OUVsPS0tLLU3S0hbOoVO93XZ2jNOamUJKeV5VP4dMfXQaH9MgmRZIqUFVnxbplFqmXtPOnFbH9FHfg/IaphG7feuDDz7Yji1pss0MxaFDh1oqrFP7aUdOIZ7UytTjwrH1O3BNMPXaKbFcmz/72c/a8fHHH99dN629Mc8xRdtpwJwTr2e+G9/ZqbJMH3aKsMd4COhPvd6IZWxcc0wd93qmrNPp7ZSXUDpw2WWXddedccYZ7ZiyFtsE16nTba+99tp2zPG3nM3SQ8K0X9qIU6gps7Q9nnLKKRPvvx5WV1fb+Dp1ef/+/e14moyE9uz2t4888kg7tlSXPpvyKO+fXIvcg2zXTEO2BJD+kfIwy0H4N5+9quozn/nMK97DY0O/Yvsfr+8hZXxzc3NNsmE5Ots6W15MaQj9mv0upTuWAtOeuQYcD5x//vntmLGN5TeUfNiffvGLX2zHTIP3vsX17JbnjOcoC6IMvqpvpXzCCSd057znD8HmzZub7fm9OR/2M4zdTj/99HZ88803d9dNkvoY+nLPIaVe9Ncef/4/34Pyv2n+jlI7ry/aOO3H8mbGXLZpx7wbgaWyfF/LYCjxoTTR0g3uO/Z/XMMf/ehH27HbXPPd+bmWA/L5bTPcu/h9xPdg/O84jddS2s14taqXo/m7kO1rCObn55vNOV6mHVkmRxkyx8eyIsZqjvcYm1A65HXPuaEv9HUcS0vr6Sfp07i3V/Vz4/IVb3/729sx427bJtcp952qjZHRVvXfM/zubOfuc3/961/bMef/mWee6a5jK/Wf/vSn3Tl+b+P8uETCueee244ZU1iWNUkiX9X7dvpRfm5V/572f7w/7dg+lf/Pa3GaXPO1kMyfEEIIIYQQQgghhBkmP/6EEEIIIYQQQgghzDCD6U7e8IY31GmnnVZVfbpWVZ/e7TQ/prwx5ckpVUxdc8cTduxhGrLTZZkKypRIS4b4/5yOyVREpjK7ewzPOa2W6WeUl7jz0J///Od27JT+cYqwx2k9jEajljbntDWmf3r8CcfVz0ycej2pq4A7zjA1kM/h9DmmQVqGwudi+iBTLKv61E9LJc4888x2zNRSdxegzMEp+JYnDcXq6mpLubf0kWmqTiukXfJZ/U4cd8sumDZ9ySWXtGN3J6A0lN0PvI4o1bSUY+vWre2YqZqWmkxLXabkifPv/8M17LUxfrch1+LCwkJL43YqsFOZCdcR7de2Rj/pziiU/lAC5m5SnCvKzdx1hFjWQvknbccdMZgq7NR3prtzn3AaPH2JZXC2z6FYW1tradiWSnHfmZbOT+mA1welIva3HDP6QO/B3Be5Brj/VPVyEI8fU835Lpb5Ufbi57jjjjte8f9Z9sjuRO4oOZa6DbkWN2/efIRsaQz3bO9p9LVM07dPpmzEKfLc1zjmjikonaJPdsq/x5wwpZ3+35JIPoftgH/z2d35izJB+zO/2xCsrKw0n+7OV/Q17qI2qSssJR1V/R5paQjHhD5p2nrl/mOpJzvOWrLLuedcu6PjJHlYVb9v0Gc6RqMM1LGNZWZDsbS01PwBZcJV/T7mGJU+lGPxhS98obvummuuaceWHjMWoaTa+wyfi3um1yJ9vKVY/A7C8gN8hqr+e4blRIxTPvWpT7Xj6667rruO+71lrUOWlRizsrLS4m6X9+Basc+Y9D3PY8K9aNp3Ffpr+3eOP7sbO7ZhDO29iD6Be6LXLCXZ9oXs3jf+fl115DzxOza7plYdKWMaisXFxVbCxDIqjpnlSozP+WzubkUJl2N/SqW5tzqO43hu2bKlHTuWpSzPkjrOMf+fY1TGUY6dCH204z7+vmAZnPfa9ZLMnxBCCCGEEEIIIYQZJj/+hBBCCCGEEEIIIcww+fEnhBBCCCGEEEIIYYYZrObPiy++WI8//nhVHak5pGbZGk/q5qhjZP2Rql6X7Nos1HVSG+iaP6wdQP2o9YTUSluXTU046y9Yq0ntnnV9u3btasdsuWn4XNaij2tsDNlmejQatftZB06dpjXzrK9A3aPrC7DuxDTtLO9ne+Ec0nasMeeYu/0fNcGsPWOdJu/pOgTUqtLOrMuk3tXjRq3vkCwuLrZ7u0YI7dK1CXgta0i4FhV1yNYeU/dM7Ttrx1T1NWI4Lm7RyrlzC0vWSGBtBj8TbcG6e96T9jmtro415uMWsNP05a+WtbW1pkm33fB9rKdmHRnW+aFGuqrqT3/6Uzt2TRX6GtaesP6fn836Hq4vwVoEXousf8JWuq6Pw3Xqei58DtZf8JqlTzvppJO6c247PhQLCwvtOVx3iXUGXLNs0jt572PtEbfG5f1Zr8M10GjrrC/lWiOsPeH1Mdb+V/V18qyLpx+gDVZVffOb32zHd999dzs+cOBAdx3rlTjW2Ig24aPR6IhYYgzn1OPP/Zs1EVxLhGPp2gncg3h/142gr53mH2gTjh1Yd4Rr0T6Z7+y6cfTJtGnbAcfAft11/oZgNBo1v2a/wM/3+PNa7vms1VjV27xrOrG2CPcI+0LOKcffezjjKs8N1yjXueunsO6Ex2NSq3rvQ/Qp3nOntbtfDy+99FKL01yrjr7R7dcvvfTSdsz9236M88M20VV97HDssce2Y8eotN+bbrqpHXucd+zY0Y7pP6uqnnzyyXZM/z8tLmHNtKqqZ599th1zzfr7Dtepv3d5rxiClZWVZjuut8qY1DEj9336uPPOO6+7jvGpYxbGshxXx7isHUO7d90rriPXa6Mt0eZYr6iqX7Oukce/6R88bvw+5dpzGzGHxvEa/ZfnkTbG/cN7mm2RcC/kdf6+zjXAWmYc86o+Bna9Tz4XYxb7GNbYst3RT9OP+LsE7+lzjv3WSzJ/QgghhBBCCCGEEGaY/PgTQgghhBBCCCGEMMMMlg82Go1aKt4PfvCD7hxT9JxqzFQmpnGef/753XVM7XNK6Rvf+MZ2zNQ7p1kyZY+pjk6//c53vtOO3WaZ6bNMv3UK6rXXXtuOneZ98ODBdsx0M6fVMoXd7STH0gqnTK+Hubm5liJoWRbT3ZxqzzQ2voPHjinVflemuDFlzimDnMMXXnihHXt8eD+nEzJlkHPodo/XzLw5AAAU50lEQVRMuXz++ee7c0xjP/HEE9ux2z3StpwK6NaGQ7G2ttbSDKe1+rVMgemhTB21lI2pmtNkR0wFt1SH0iLK33w/pjm7hTvlRbw/Uz2rettwmjzTjJmmaz9FSSClDVUv+xxK2dbLUUcd1XyjW3XStm2zfG6mwFpiwzRnSwKY9kq5i6WVvD/TVe2fzzzzzInPy3aW/Fz7B7ajZXptVW8jnCeuy6p+H3JrXadsD8XCwkJbL3532h5bDFdVbdu2rR1zLPzulBxceeWV3bn77ruvHZ9++unt2G1Ox3Ltqt4nea1wjOwTKF/mOrXUjW3Df/GLX3TnuMdTmsD90s/odP0hZdBjlpeX2xp0KjZ9nCUBjG1ov96r6Ls8Xl//+tfb8Te+8Y2J9+AaY7xhae80CSztk//Pkh6OuX0hJS/cTywj5zxRslt1pMxpCFZXV5vdW2LDZ7MP5zvQPrm2qqquuuqqdmwJKeNT7nWWetInMVawHJ02aJ9M/0B5iWWL9AGWn9N/T9vT6B/YRrmqly0Nyfz8fNunvQYYLzu24VpkjGHZIn2hx5ZzvG/fvnZsWc3JJ5/cjumTfT/aBY+r+vG7/PLL27HjfcqT/H2H64r/z/6fklHvu2effXYNDefQ5SwYi1g+zvVCe37ooYe66zjOjkW4PijZss9hHMvvdvanXAOUAvoc/b/XGyXo9sncZ2mrbmlOf0opeNWR32+HYm1trT2vbY8+yp/POJvv53fneHrfZUzMObGfo69/7rnn2rHjBt7Pz8u/jznmmHZs6RilwJaC0p9zrBx7M5b1eAwtpU3mTwghhBBCCCGEEMIMkx9/QgghhBBCCCGEEGaY/PgTQgghhBBCCCGEMMMMVvNnfn6+aTLdvpy1XqzFpc6Z2lm266vqtYFsHVzV1/hge0a3YqOuk7pDtyGl1s66bP5NHR81nVVVX/nKV9qxtcl8Z2o13RKeNRZct2ZcE8F1ZNbDyspK0wNb50/NtFvw8f2ozfS48m/qbauqnnrqqXbMts6eQ2ouWSPENZeoe7XGls9B/bN1tNRus75QVa8z5TmPG+uOeF24vs1QjEajpvF2jRCOi22K8zitzT3rAVmHylpR1N9aS85zfEbXx6KduG6NfckYt6i+66672rFtktpczr9bgbPuwhlnnNGdG+vWPffr4R//+Ec9/PDDVXWk/6Cf9Ppgm2H6TNcUIK4zQj007cD+ie/L+ljW6rMO1kc+8pHuHO2C69T1Zvi31w39Nfcat0Ol3bo2zEbUiqn6V62R8ZrzPHKcP/vZz3bn2N6c2ne/O2uiufYE1xXXqduhUtPO2kCui0TfYW09/T7rS7jOBWvEuJXybbfd1o657qfVfbN/G69vtzNeDwsLC23fdzzAMbdvYQ0S7luuB8DaHF47J5xwQjum/bpeHN+X5/bu3dtdxzXmeiesP8C5dp0RjoH9pMdnjH0j7cDnGJsNyThWcp0F1qFyHYdJtdG8x/C97U8ZH9CPuUYU9zPOp2uJsB6Ma/LQR3PtOU5k2+Npvpaf7RiI8+Q6RxtVz3B+fr7Feh5n+n7XZuH+Ny32Z4zmWlRXX311O+YYOQ6hPTOudR1N1rtzDRLOK2ueuY4mY2B+56jqff5FF13Ujr/61a9217FOl23BtbqG4PDhw83PsU5KVT9PjpdZr5J7ltcRr/MexvHi3uG9k+uIY+I4gevUsTDrzbCWi+vXENd54T7IeGb//v3ddbRVfz8eulbMGNaIdXzPNebW85wf2rlrsf3973+feI7xB+MD75+EzzGttpLjez4H/cpb3vKW7jrumZ5jXkt78vpiDWDu/VV9DbchSOZPCCGEEEIIIYQQwgyTH39CCCGEEEIIIYQQZphBW72PU7+cdsZ2fm4Ry3Q7pqc5nZht2pz6zhRAnpsmdZiWzsgUOqZuV/Wp0kzzYipbVZ8q7ZQ4pgCyzavTR5kebunS+JxbJa6Ho48+uqWmOuWS6Whu78o5pGxn586d3XW33HJLO/b8Uh7FcXBaLtMEmSo7ba6ntTJlOrTTq++44452TOlQVdV1113XjjmfnmumJluKwNTrIdm8eXNLheS6qerTMSmvq+pTISkNsTyMKZduv067Ycqt03t5Dz4HU6Gr+vG0/GBS60v7B65Tyw1pu1zDbq+7ffv2duzU6LFUwP++HjZt2tRs2r6K62PHjh3dOY4rJWG2PcokPYdMWeX40M6rqp5//vl2/Ktf/aodUyJa1acrU/5S1fs43s+p3JRmWGJLORVbdtpu+f8sS3D7243A0gumKFumwH3y6aefbsdM0a/qx9p7K+eR6e5uZcp19Oijj058pmmtTCmJoY/2XNGeLNXl3NEXWdrl5yLjtsXTWlS/Wubm5tqaczo3pajTJGEcc7cVppTA0kDuVZTdeI3RT1L6OU1u6z2Tz8V3sfyF0hPbAT+PsYD3YM6vx9TzPQQLCwtNvmxpMcfYMhGOJefJ8jDOteXJjHW4R3gtc/3yOksn6Ru3bt3aneN8cM177+Seyxi06t9fO3wur8kh1x9ZXV1ttsOyCFX9PmNJGO2Pdu4YiLbOmLSqH0PuVX4OrnVKNSxd4XcLStaq+vbl9M9+JsYplFdX9fEMvyd4D2aJDfsf280QLC4utjHzWuQz8ztHVS+/obTO8hvuRfbJnF+WM3CswNIWk+TTVf2e7rGiL+Q9LE1661vf2o7tJ+lruZ5PPfXU7jr6C5fl2CgZ7eHDh1vc7eemTVlezDlm3OX4kr7XcsdJMqppa4zzbakx78E4xJ/FsXz88ce767gHOL6kbTAu997K5/AeM0lS/VpJ5k8IIYQQQgghhBDCDJMff0IIIYQQQgghhBBmmMFkX6z87UraxF0SmKrI1McHH3ywu44yFKeUTkrptCSK92fKudOMKSHxPZjSyXQ2piT6mZxGxrQ/ppQ5HZxpXq6YvhEsLy+39/CYMNWY6XhVfWV8psC6OwRTJJ1qzBRiprtZ9sVUSqZq2iaYUm0JG+2T42/ZBOfeacRMc2aKruV/fEan2Ttlfijm5uba8zmNl/PqdEzOF9MxnSbMe1q6wfmi5MOSF6bPbtu2rR1/6Utf6q5jRyCPF9NlmcbpNFemXDq9l3NCm7QPo09g+m3Vy+PojhHrYfPmzU2G6Cr/W7Zsace2bdolfdC07gaWhnCt0685DZsyiLPPPrsd287puzw3tEc+h9Pgec7vwlRZyvosJ+EeMq3r25Csra21PWNa1w1KE6v6seYYWRZJH+VOJLRH7iVesxxb2gU7uFX1+5htkmPL+bacjnIl2xPtlTZo6TXfy+ngY6mxOxSuh+Xl5bY3u1MU0++9R9NPcO7tT+l3HW/QLu688852vGvXru46ypU4b5Yy05am7WncG9zlj3um/SnXN/d0+0z6a8cajpeGYHFxsdmX9yLGLJbAcq1wDVnmRLmBZfFcA+ymR+lQVT9XlEBYAst9z3GPffkYxihVVVdeeWU7tj/gu3F/sbyG69KSZ0s4hoLfMzwunB931uL64F5+9913d9cxNr/gggu6c+Pum/4s71W0L5bAsL/jGvNexf2U82GfTNmUv4NQfki/5fV2/PHHt2PLdKdJbF8rhw4dar7H78M9zD6DklWOuZ+ZY+lSGrwnYwXL4rmOaNuObWiD7i7KeJKfaxkn/YO7t1KWyLGxBIi2ZP82ySesF3bec0zG557WTXBSCZSqvpSAZWUcM7677ZWxP8tsjOXhYxiX+HsR1xHH0r6X9/d3X8Yj9AOOCfm3pWn+frpekvkTQgghhBBCCCGEMMPkx58QQgghhBBCCCGEGSY//oQQQgghhBBCCCHMMIMVqVhdXW2aNNc9oT7viSee6M6xJRr17dZqUmtnvSM1vDznVprU4VF3aB0/6xdQZ1o1uX6FdZXUZ7oWCLXT1Pi7ZSSf13rh8TtbJ7keVlZWmjbV96Uu2NpDaiLPPPPMduyaArzu85//fHeO+slptZQm1U5wy0i2gXfrw/POO68dcz7ZnrCqb21Mnb0/j/Pme1C3aY36RvHSSy8dMR5jqIn183i9jHFdB46ta6Xw2l//+tft+JJLLumu47qiRtmtV3k/Py91/LQLt7Kmntv1Dfj/qFv2O7Mlo33CuE6ENbrrYXl5+QjN+xhq3K3fpw/l/3dNAWqoXR+Fvpfzy3oVVX29FY6jbY/6fPsErnvez+1Q2dKW67eqr0tBv/vcc891102rJ+CWm0OxuLjYagu4Pg3HhXtYVV9DhPup6wrwHvbZ1LFzjztw4EB3HWsQnHPOOe3YtVe4B1nHz/tz7bglOfdn1yPjPbhmp9U38HOM18aQLYo3b97c1rh9C/+2bXNtug4DoU92i1jen+PzyCOPdNe9973vbce0bcclXFeOe1gHgXNtX8h6T45ZOO6TfHxVHxO5LpRjvyE4dOhQW3+uyUM79B7Id2c8wFouVVV/+9vf2rFb9HJf4R5mu3722WfbMde1n5c2772I9sL7u8YZY7iLL764O/e1r32tHTMWdm0V2qrfeaNq/lS9bEuOFWjbrqHG5+He59pAtEvXf+JexTjP+ydthvuKvwfQzqe1K2eca1vwnBDGXKyh6r2P9/Ae/9RTT028/2tlfn6+zcfTTz/dneNatO9iTVTGca7XQx/EWHX82WM4H65Zwz2X4+W6enxe34P76qS6qFX9XJ988sndOcZf/L7pOmy0d7+z18JQLCwstL3YtRxZY+zd7353d457AfcZ29rHPvaxdkzfWNX7NsY902pFHjx4sB1Pq3vodco4iHbhd2YNNN9jUs1Kx8qsQ+U5duyxXpL5E0IIIYQQQgghhDDD5MefEEIIIYQQQgghhBlmuN7E9XJKnVPjKJlxejtTJinx8T2Y6uq0xbe97W3tmKmfTv1mKhZTx51eyBacTilk2iBTtNwCkHIup2oyXZbv7LRBpvS6Jdw4ZXRaOvmrZXFxsaUkWtrF1GzLASi/YWqsx45pck45Z+oe0xQ9dpQRMM3OqY5MkXNKLdPpeJ3Td2kvloUwlZH2aIkZ8VxZ0jYU8/PzLeXU7YdpU54DjgvXqVOBuTadPskUZd7D0gGmzfMZnYZN6Q6lMFV9Sjpt0HPFZ7TvoJyRtuuxof+xNGMs93Ca5npYXl5u9u0UUj6zZYacXz6z5UJcw5YEcK3TN3KMq3pfSB9sKRqfyfbIseRnnXbaad11lKS4vT19B9flfffd1133zne+sx27ZbqfayjW1tbavTlvVf369xwzHdi2SG6++eZ2fNlll3XnKPVj62DLJbi3cI3dc8893XUcZ8tg6du599nuKP298cYbu3OUQXCu/vKXv0y8h+Ul4zXo+GE9sDUx27lW9XZjX8j4w7JLwrn3eDGe4Wd5f6b8gJ910kknTbyf2+Ju3769HXNc3dJ2miyLey1twhJMXueY0GUDhuCoo45qe4v3B86T5Utbt25tx/SL3ou4VizFosyD0kCPI8eE+6V9A9crpWhVfQzJ9WrZBNso279w7hljOZ7mvPkZHUsNBUsTOG7n/rRz587uHPeCq6++uh3TV1X1c2f/xLXPOOLUU0/trrv33nvbMWWEXm9ci44pKGuh/MzfM9h+3r6J64/74vXXX99dx/Xm2N5S5SGYm5tr9uI9kT7U64hjRF9oO5hmexwTxgD2OZRb8Zyv495k/891RZ9s+Q73Ktsj7YC+49hjj+2uY7zgEiGWJw3F4uJi26/8XYZ26nNcE5TB2i9zj5smG6afo7+q6n2Wv48SxsqML6r6tcP5ti1Mmu+q3i9TdnrKKad011Eq6Hh76HlM5k8IIYQQQgghhBDCDJMff0IIIYQQQgghhBBmmPz4E0IIIYQQQgghhDDDDFbzZ9OmTa2OjtuvUhvsujZs9UcNHWtIVPVaSNcAYAtAau1cg4P6Quqr3XKTekLr0anT5v1dS4RaQ2tSOT7U6rt+AevYuObP+Jmn1RJ4tYxGo/ZOfm/WYnG9Bz7DXXfd1Y7POuus7jpqUa2r5LtTO8m6LlX9XLO+jq9j7QS3QOZn85zHknpn2+2jjz7ajqnNtO3zvVxrY6PqjCwsLDTds7XkbNNrDT/XAduWu14Az/l9aetcz3v27Omu49jef//97djtHqljv/XWW7tzXKdsZeo24axB4+d94YUX2jH1ttTlVvUaZrb1rRq27taYpaWlVlfC88QxskacOmm+N31fVb+Grcmmxpljbj9GW2Irdn8W7cDjf9ttt7Xj/fv3t2PX8GJ9NX5uVV9LgTWQ3LaZdYNcp8EtN4dieXm5ja/HmVp/+xfWCeGeQ3ut6lvDfvnLX+7Ose0p9eLW1p9wwgntmGNm/8Q15rHleqaf37dvX3cd/Tf9SFVvG2zL6uv4LtbMs17fULD+lmMFvo/3RcYHXjuE4+VaVPQ79HdcK1V9TER7cQ0E1qWgv6uq+slPftKOWXPrySef7K6j/+P7V/V7HGMixjlVvW25XoxrfQzB4cOH277gWjv8fM8Tn5M1I1yLkPNkH8daQayJ55iR+wifg/Up/dkeO/pe+lDXiqG/eeKJJ7pzXNuu90Roc26xPGTNLTIajdp+6BoYrM/E+ih+Htb/sV1yDryvM4ahXezdu7e77s4772zHu3fvbsePPfZYdx3n2H6Ffox+xDV5zj333Hbsd2YtP+6fjkM5x967bTdDsLCw0N7X32toz15HjG3oIzx2jOm9r/L9GBvb5/D7BOeJa7mqH2Of49/8/uS1wfu7bhDjki1btrRjvxd9hGtLbdRaZD1DfyZrkTnuod/jc7POkvFexe9qtF/XYmMNN/6m4NqsnCvXzqIdMjZ2fTjGUa6VxTidPsfxHK9zPaxp9WRfC8n8CSGEEEIIIYQQQphh8uNPCCGEEEIIIYQQwgwzZ9nAOvg/VfW//9urwtD8V1X9r//2qn+PzOH/HJnH/3wyh7NB5vE/n8zhbDDUPGYO/+fIWpwNshb/88lanA3WPY9D/vgTQgghhBBCCCGEEP4/I7KvEEIIIYQQQgghhBkmP/6EEEIIIYQQQgghzDD58SeEEEIIIYQQQghhhsmPPyGEEEIIIYQQQggzTH78CSGEEEIIIYQQQphh8uNPCCGEEEIIIYQQwgyTH39CCCGEEEIIIYQQZpj8+BNCCCGEEEIIIYQww+THnxBCCCGEEEIIIYQZ5v8CMwyFKF5KumoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1440x288 with 20 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "dark"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_test, labels = next(iter(test_loader))\n",
    "x_test_noisy, labels = next(iter(test_loader_noisy))\n",
    "n = 10\n",
    "plt.figure(figsize=(20, 4))\n",
    "for i in range(n):\n",
    "    ax = plt.subplot(2, n, i + 1)\n",
    "    plt.title(\"original\")\n",
    "    plt.imshow(x_test[i].squeeze())\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "    plt.gray()\n",
    "    ax = plt.subplot(2, n, i + 1 + n)\n",
    "    plt.title(\"original + noise\")\n",
    "    plt.imshow(x_test_noisy[i].squeeze())\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "    plt.gray()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Sy9SY8jGl5aP"
   },
   "source": [
    "### Define a convolutional autoencoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vT_BhZngWMwp"
   },
   "source": [
    "In this example, you will train a convolutional autoencoder using  [Conv2D](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D) layers in the `encoder`, and [Conv2DTranspose](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2DTranspose) layers in the `decoder`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "R5KjoIlYCQko"
   },
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################\n",
    "\n",
    "class DenoisingAE(AE):\n",
    "    def __init__(self):\n",
    "        super(DenoisingAE, self).__init__()\n",
    "        self.encoder = ...    \n",
    "        self.decoder = ...\n",
    "    \n",
    "    def forward(self, x):\n",
    "        encoded = ...\n",
    "        decoded = ...\n",
    "        return decoded\n",
    "\n",
    "autoencoder = ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write down the full optimization loop to optimize your denoising auto-encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QYKbiDFYCQfj"
   },
   "outputs": [],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A7-VAuEy_N6M"
   },
   "source": [
    "Plot both the noisy images and the denoised images produced by the autoencoder to check that your implementation is correct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "sfxr9NdBCP_x",
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 1440x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "######################\n",
    "# YOUR CODE GOES HERE\n",
    "######################\n",
    "encoded_imgs = ...\n",
    "decoded_imgs = ...\n",
    "plt.figure(figsize=(20, 4))\n",
    "for i in range(n):\n",
    "    ...\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Attention layers and Transformers\n",
    "\n",
    "In the following exercise, we aim to implement attention layers and Transformers in PyTorch. Transformers are a powerful deep learning architecture that has been used to achieve state-of-the-art results in various natural language processing tasks. \n",
    "\n",
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #410819; border-color: #cb2e47\">\n",
    "\n",
    "> **Warning on the exercise difficulty and goal**\n",
    "\n",
    "> **Note that this exercise is a lot more involved, as we now move on to an almost research-level type of exercise, where we mostly provide a research paper as reference. Your goal is to try to reimplement the architecture and method proposed in that paper as closely as possible.**\n",
    "\n",
    "#### Paper reference\n",
    "\n",
    "[Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.](https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)\n",
    "\n",
    "**Link:** [https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf](https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)\n",
    "\n",
    "</div>\n",
    "\n",
    "***\n",
    "\n",
    "<a name=\"intro\"></a>\n",
    "## Introduction to attention\n",
    "\n",
    "Attention mechanisms enable models to focus on specific parts of the input sequence while processing it. They have been proven to be highly effective in natural language processing tasks such as machine translation and sentiment analysis. Transformers are a type of deep learning model that incorporates self-attention mechanisms. They consist of an encoder and a decoder, and utilize multi-head attention layers, positional encoding, and layer normalization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Defining our own attention layer\n",
    "\n",
    "In the following, we re-implement the *attention* layer, which is the basis of the infamous `Transformer` models.\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #410819; border-color: #cb2e47\">\n",
    "\n",
    "> This is defined in Section 3.2.1 (pages 3 and 4) and depicted in Figure 2 (left) of our [paper reference.](https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)\n",
    "\n",
    "</div>\n",
    "\n",
    "\n",
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #192841; border-color: #779ecb\">\n",
    "\n",
    "> ### Exercise (**course**)\n",
    ">   1. Implement the simple attention layer as defined in the slides\n",
    "\n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AttentionLayer(nn.Module):\n",
    "    def __init__(self, n_hidden):\n",
    "        super(AttentionLayer, self).__init__()\n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE \n",
    "        ######################\n",
    "        ...\n",
    "\n",
    "    def forward(self, X):\n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE \n",
    "        ######################\n",
    "        ...\n",
    "    \n",
    "\n",
    "######################\n",
    "# Solution: \n",
    "class AttentionLayer(nn.Module):\n",
    "    def __init__(self, n_hidden):\n",
    "        super(AttentionLayer, self).__init__()\n",
    "        self.mlp = nn.Linear(n_hidden, n_hidden)\n",
    "        self.u_w = nn.Parameter(torch.rand(n_hidden))\n",
    "\n",
    "    def forward(self, X):\n",
    "        # get the hidden representation of the sequence\n",
    "        u_it = F.tanh(self.mlp(X))\n",
    "        # get attention weights for each timestep\n",
    "        alpha = F.softmax(torch.matmul(u_it, self.u_w), dim=1)\n",
    "        # get the weighted sum of the sequence\n",
    "        out = torch.sum(torch.matmul(alpha, X), dim=1)\n",
    "        return out, alpha\n",
    "######################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<a name=\"setup\"></a>\n",
    "## Setup and dataset\n",
    "\n",
    "First, let's import the necessary libraries and prepare the dataset for training. We will now rely on a text dataset, which is readily available in the `torchtext` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "from torchtext.datasets import Multi30k\n",
    "# Translate English to German\n",
    "language_pair = (\"en\", \"\")\n",
    "# Import the text dataset\n",
    "train_iterator, valid_iterator, test_iterator = Multi30k(split=(\"train\", \"valid\", \"test\"), language_pair=language_pair)\n",
    "# Potentially use the GPU if available\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"mha\"></a>\n",
    "### Implementing the Multi-Head Attention layer\n",
    "\n",
    "The multi-head attention layer is a key component of the Transformer architecture. It allows the model to focus on different parts of the input sequence simultaneously.\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #410819; border-color: #cb2e47\">\n",
    "\n",
    "> This is defined in Section 3.2.2 (pages 4 and 5) and depicted in Figure 2 (right) of our [paper reference.](https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)\n",
    "\n",
    "</div>\n",
    "\n",
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #013220; border-color: #03C03C\">\n",
    "\n",
    "> ### Question 2.1 - Implementing the multi-head attention layer\n",
    "\n",
    "> 1. Following the definitions in the paper, propose your implementation of the _multi-head attention layer_.\n",
    "  \n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MultiHeadAttentionLayer(nn.Module):\n",
    "    def __init__(self, hid_dim, n_heads, dropout, device):\n",
    "        super().__init__()\n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE \n",
    "        ######################\n",
    "        ...\n",
    "\n",
    "    def forward(self, query, key, value, mask=None):\n",
    "        ######################\n",
    "        # YOUR CODE GOES HERE \n",
    "        ######################\n",
    "        ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"transformer\"></a>\n",
    "### Building the Transformer architecture\n",
    "\n",
    "Now that we have implemented the multi-head attention layer, we can build the Transformer architecture. We'll need to create the encoder, decoder, and the final Transformer model.\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #410819; border-color: #cb2e47\">\n",
    "\n",
    "> The transformer architecture is defined in Section 3.1 (pages 2 and 3) and depicted in Figure 1 of our [paper reference.](https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf). Also more details about the architecture are provided in subsequent sections 3.3 to 3.5.\n",
    "\n",
    "</div>\n",
    "\n",
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #013220; border-color: #03C03C\">\n",
    "\n",
    "> ### Question 2.2 - Implementing the Transformer architecture\n",
    "\n",
    "> 1. Following the definitions in the paper, propose your implementation of the _transformer architecture_.\n",
    "> 2. (Optional) Implement the other parts of the architecture (sections 3.3 to 3.5).\n",
    "  \n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encoder Layer\n",
    "class EncoderLayer(nn.Module):\n",
    "    def __init__(self, hid_dim, n_heads, pf_dim, dropout, device):\n",
    "        ...\n",
    "\n",
    "# Decoder Layer\n",
    "class DecoderLayer(nn.Module):\n",
    "    def __init__(self, hid_dim, n_heads, pf_dim, dropout, device):\n",
    "        ...\n",
    "\n",
    "# Encoder\n",
    "class Encoder(nn.Module):\n",
    "    def __init__(self, input_dim, hid_dim, n_layers, n_heads, pf_dim, dropout, device, max_length=100):\n",
    "        ...\n",
    "\n",
    "# Decoder\n",
    "class Decoder(nn.Module):\n",
    "    def __init__(self, output_dim, hid_dim, n_layers, n_heads, pf_dim, dropout, device, max_length=100):\n",
    "        ...\n",
    "\n",
    "# Transformer\n",
    "class Transformer(nn.Module):\n",
    "    def __init__(self, encoder, decoder, device):\n",
    "        ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"training\"></a>\n",
    "### Training the Transformer Model\n",
    "\n",
    "After building the Transformer architecture, we can train the model using the dataset prepared in section 2. Note that this exercise is left optional, as this requires several notions not covered in the course yet (embeddings, positional encodings). However, this is of course more rewarding to see your model being trained :)\n",
    "\n",
    "***\n",
    "\n",
    "<div class=\"alert alert-success\" markdown=\"1\" style=\"color:white; background-color: #192841; border-color: #779ecb\">\n",
    "\n",
    "> ### Exercise (**optional**)\n",
    ">   1. Implement the training loop for the Transformer.\n",
    "\n",
    "</div>\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Transformer model\n",
    "INPUT_DIM = ...\n",
    "OUTPUT_DIM = ...\n",
    "HID_DIM = 256\n",
    "ENC_LAYERS = 3\n",
    "DEC_LAYERS = 3\n",
    "ENC_HEADS = 8\n",
    "DEC_HEADS = 8\n",
    "ENC_PF_DIM = 512\n",
    "DEC_PF_DIM = 512\n",
    "ENC_DROPOUT = 0.1\n",
    "DEC_DROPOUT = 0.1\n",
    "\n",
    "enc = Encoder(INPUT_DIM, HID_DIM, ENC_LAYERS, ENC_HEADS, ENC_PF_DIM, ENC_DROPOUT, device)\n",
    "dec = Decoder(OUTPUT_DIM, HID_DIM, DEC_LAYERS, DEC_HEADS, DEC_PF_DIM, DEC_DROPOUT, device)\n",
    "\n",
    "model = Transformer(enc, dec, device).to(device)\n",
    "\n",
    "# Train the model\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "def train(model, iterator, optimizer, criterion, clip):\n",
    "    ...\n",
    "\n",
    "def evaluate(model, iterator, criterion):\n",
    "    ...\n",
    "\n",
    "def epoch_time(start_time, end_time):\n",
    "    ...\n",
    "\n",
    "N_EPOCHS = 10\n",
    "CLIP = 1\n",
    "\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(N_EPOCHS):\n",
    "    \n",
    "    start_time = time.time()\n",
    "    \n",
    "    train_loss = train(model, train_iterator, optimizer, criterion, CLIP)\n",
    "    valid_loss = evaluate(model, valid_iterator, criterion)\n",
    "    \n",
    "    end_time = time.time()\n",
    "    \n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "    \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'transformer-model.pt')\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train PPL: {math.exp(train_loss):7.3f}')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. PPL: {math.exp(valid_loss):7.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"evaluation\"></a>\n",
    "### Evaluating the Model\n",
    "\n",
    "After training the Transformer model, we can evaluate its performance on the test dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the best model\n",
    "model.load_state_dict(torch.load('transformer-model.pt'))\n",
    "# Evaluate the model on the test dataset\n",
    "test_loss = evaluate(model, test_iterator, criterion)\n",
    "print(f'| Test Loss: {test_loss:.3f} | Test PPL: {math.exp(test_loss):7.3f} |')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Optional fun** - Style transfer \n",
    "\n",
    "Style transfer is the technique of applying the style of one image to the content of another, creating a new image with the desired content and style. The following exercise aims at replicating the results of the original [Neural style transfer paper](https://arxiv.org/abs/1508.06576) by Gatys et al. \n",
    "\n",
    "Note that the following tutorial is a plain simplification inspired by the great [tutorial available in the Pytorch documentation](https://pytorch.org/tutorials/advanced/neural_style_tutorial.html). Hence, all credits goes back to the authors of the tutorial.\n",
    "\n",
    "## Introduction to Style Transfer\n",
    "\n",
    "Style transfer is a technique in computer vision and deep learning that enables the transfer of artistic style from one image (style image) to another image (content image) while retaining the original content. This is typically achieved using a pre-trained deep neural network, such as VGG-19, and optimizing the input image to minimize the content and style losses.\n",
    "\n",
    "<a name=\"setup\"></a>\n",
    "## Import images\n",
    "\n",
    "First, let's import the necessary libraries and prepare the images for style transfer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import requests\n",
    "from io import BytesIO\n",
    "from torchvision import transforms, models\n",
    "from PIL import Image\n",
    "import matplotlib.pyplot as plt\n",
    "# Find our device\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "# URL to load our content image from\n",
    "url = \"https://thumbnailer.mixcloud.com/unsafe/300x300/extaudio/6/b/a/b/d576-35c3-48a9-a01e-7dc4f0255e62.jpg\"\n",
    "response = requests.get(url)\n",
    "content_image = np.array(Image.open(BytesIO(response.content)))\n",
    "# URL to load our style image from\n",
    "url = \"https://t4.ftcdn.net/jpg/02/44/21/17/360_F_244211780_VFoZhDiuxyWpnTalr0DFilyYqNokEoVZ.jpg\"\n",
    "response = requests.get(url)\n",
    "style_image = np.array(Image.open(BytesIO(response.content)))\n",
    "# Display content and style images\n",
    "plt.imshow(content_image)\n",
    "plt.show()\n",
    "plt.imshow(style_image)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"model_loss\"></a>\n",
    "## Pre-trained Model and Loss Functions\n",
    "\n",
    "We will use the pre-trained VGG-19 model, and define the content and style loss functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision.models import VGG19_Weights\n",
    "# Load pre-trained VGG-19 model\n",
    "vgg = models.vgg19(weights=VGG19_Weights.DEFAULT).features.to(device).eval()\n",
    "\n",
    "# Content and style loss functions\n",
    "class ContentLoss(nn.Module):\n",
    "    ...\n",
    "\n",
    "class StyleLoss(nn.Module):\n",
    "    ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"prepost\"></a>\n",
    "## Image Preprocessing and Postprocessing\n",
    "We need to define functions for preprocessing and postprocessing the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Image preprocessing\n",
    "def preprocess(image, max_size=512, shape=None):\n",
    "    ...\n",
    "\n",
    "# Image postprocessing\n",
    "def postprocess(tensor):\n",
    "    ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"implementation\"></a>\n",
    "## Style Transfer Implementation\n",
    "\n",
    "Now that we have everything set up, we can implement the style transfer algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Style transfer function\n",
    "def style_transfer(content_image, style_image, content_layers, style_layers, content_weight, style_weight, iterations=2000):\n",
    "    ...\n",
    "\n",
    "# Perform style transfer\n",
    "content_layers = ['conv_4']\n",
    "style_layers = ['conv_1', 'conv_2', 'conv_3', 'conv_4', 'conv_5']\n",
    "content_weight = 1\n",
    "style_weight = 1e6\n",
    "\n",
    "output_image = style_transfer(content_image, style_image, content_layers, style_layers, content_weight, style_weight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"visualization\"></a>\n",
    "## Visualization\n",
    "\n",
    "We can visualize the output image and evaluate the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize output image\n",
    "plt.imshow(output_image)\n",
    "plt.show()\n",
    "\n",
    "# Save the output image\n",
    "output_image.save(\"path/to/output/image.jpg\")"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "TensorFlow_3_Autoencoder_Dimensionality_Reduction.ipynb",
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}