Report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Q-Learning to collect yellow bananas using Dueling Networks\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will learn how to train an agent using Deep Q-Learning and Dueling networks.\n",
    "### 1. Prepare \n",
    "\n",
    "Make sure, you followed the installation instructions which you can find in the [Readme](./Readme.md) of this repo.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "import os\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "#make sure, Environment is not going through proxy, when running on local machine\n",
    "os.environ['NO_PROXY'] = 'localhost,127.0.0.*'\n",
    "\n",
    "# define function for plotting scores\n",
    "def plot(scores):\n",
    "    fig = plt.figure()\n",
    "    ax = fig.add_subplot(111)\n",
    "    plt.plot(np.arange(len(scores)), scores)\n",
    "    plt.ylabel('Score')\n",
    "    plt.xlabel('Episode #')\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Start environment\n",
    "\n",
    "Next, we will prepare the environment!  **_Before running the code cell below_**, change the `file_name` to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Banana.app\"`\n",
    "- **Windows** (x86): `\"path/to/Banana_Windows_x86/Banana.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Banana_Windows_x86_64/Banana.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Banana_Linux/Banana.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Banana_Linux/Banana.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Banana_Linux_NoVis/Banana.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Banana_Linux_NoVis/Banana.x86_64\"`\n",
    "\n",
    "Please enter the corresponding value before executing the next cell..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name=r\"Banana/Banana\"\n",
    "train_mode = True  # Whether to run the environment in training or inference mode\n",
    "env_visible = True # Whether you want to see the agent doing training/inference.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... now we can actually start the environment.\n",
    "\n",
    "The UnityEnvironment launches and begins communication with the environment when instantiated.\n",
    "\n",
    "Environments contain brains which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: BananaBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 37\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: discrete\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "env = UnityEnvironment(file_name=file_name, base_port=64738, no_graphics=not env_visible)\n",
    "\n",
    "# Set the default brain to work with\n",
    "default_brain = env.brain_names[0]\n",
    "brain = env.brains[default_brain]\n",
    "brain_name = env.brain_names[0]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examine the state space\n",
    "\n",
    "We can reset the environment to be provided with an initial set of observations and states for the agent within the environment. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Agent state looks like: \n",
      "[1.         0.         0.         0.         0.84408134 0.\n",
      " 0.         1.         0.         0.0748472  0.         1.\n",
      " 0.         0.         0.25755    1.         0.         0.\n",
      " 0.         0.74177343 0.         1.         0.         0.\n",
      " 0.25854847 0.         0.         1.         0.         0.09355672\n",
      " 0.         1.         0.         0.         0.31969345 0.\n",
      " 0.        ]\n",
      "\n",
      "State size is: \n",
      "37\n",
      "\n",
      "Actions size is: \n",
      "4\n"
     ]
    }
   ],
   "source": [
    "# Reset the environment\n",
    "env_info = env.reset(train_mode=train_mode)[default_brain]\n",
    "\n",
    "# Examine the state space for the default brain\n",
    "print(\"Agent state looks like: \\n{}\".format(env_info.vector_observations[0]))\n",
    "state_size = len(env_info.vector_observations[0])\n",
    "print(\"\\nState size is: \\n{}\".format(state_size))\n",
    "action_size = brain.vector_action_space_size\n",
    "print(\"\\nActions size is: \\n{}\".format(action_size))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, we will use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Once this cell is executed, you will watch the agent's performance, if it selects an action (uniformly) at random with each time step.  A window should pop up that allows you to observe the agent, as it moves through the environment.  \n",
    "\n",
    "Of course, as part of the project, you'll have to change the code so that the agent is able to use its experience to gradually choose better actions when interacting with the environment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.0\n"
     ]
    }
   ],
   "source": [
    "env_info = env.reset(train_mode=False)[brain_name] # reset the environment\n",
    "state = env_info.vector_observations[0]            # get the current state\n",
    "score = 0                                          # initialize the score\n",
    "for _ in range(100):                               # maximum 100 steps\n",
    "    action = np.random.randint(action_size)        # select an action\n",
    "    env_info = env.step(action)[brain_name]        # send the action to the environment\n",
    "    next_state = env_info.vector_observations[0]   # get the next state\n",
    "    reward = env_info.rewards[0]                   # get the reward\n",
    "    done = env_info.local_done[0]                  # see if episode has finished\n",
    "    score += reward                                # update the score\n",
    "    state = next_state                             # roll over the state to next time step\n",
    "    if done:                                       # exit loop if episode finished\n",
    "        env.reset(train_mode=False)[brain_name]\n",
    "        break\n",
    "    \n",
    "print(\"Score: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Watch a Smart Agent!\n",
    "\n",
    "In the next code cell, you will load trained weights from file to watch a smart agent!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Run: 0, score: 3.0\n",
      "Run: 1, score: 7.0\n",
      "Run: 2, score: 7.0\n"
     ]
    }
   ],
   "source": [
    "from dqn_agent import Agent\n",
    "import torch\n",
    "\n",
    "agent = Agent(state_size=state_size, action_size=action_size, seed=0)\n",
    "# load the weights from file\n",
    "agent.qnetwork_local.load_state_dict(torch.load('checkpoint_vanilla.pth')) \n",
    "\n",
    "for i in range(3):\n",
    "    env_info = env.reset(train_mode=False)[brain_name] # reset the environment\n",
    "    state = env_info.vector_observations[0]\n",
    "    score = 0\n",
    "    for _ in range(100):\n",
    "        action = np.int32(agent.act(state))\n",
    "        env_info = env.step(action)[brain_name]        # send the action to the environment\n",
    "        state = env_info.vector_observations[0]        # get the next state\n",
    "        reward = env_info.rewards[0]                   # get the reward\n",
    "        done = env_info.local_done[0]                  # see if episode has finished\n",
    "        score += reward\n",
    "        if done:\n",
    "            env.reset(train_mode=False)[brain_name]\n",
    "            break\n",
    "    print(\"Run: {}, score: {}\".format(i,score))\n",
    "env_info = env.reset(train_mode=False)[brain_name] # reset the environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train your own agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Deep Q-Learning Algorithm: Deep Q-Learning with experience replay\n",
    "\n",
    "So you want to train your own agent. Let's start with an introduction to the Deep Q-Learning algorithm. You can read more details [in this groundbreaking paper](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf).\n",
    "\n",
    "### The algorithm\n",
    "\n",
    "From the [paper](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf)\n",
    "\n",
    ">The basic idea behind reinforcement learning is to estimate the action-value function using the Bellman equation as an iterative update \\[...\\]. Such value iteration algorithms converge to the optimal action-value function \\[...\\]. In practice, this basic approach is impractical, because the action-value function is estimated separately for each sequence, without any generalization. Instead, it is common to use a function approximator to estimate the action-value function.\n",
    "\n",
    "In this implementation, you are using a neural network as function approximator. The weights of this neural network are updated at each learning step.\n",
    "\n",
    "The agent does not only learn from random actions, but also from experiences.\n",
    "After the agent takes a step in the environment, it's experiences (current state, action taken, reward, next state) at each timestep are stored in a dataset called experience buffer.\n",
    "\n",
    "This is the implementation of the DQN algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from collections import deque\n",
    "from dqn_agent import Agent\n",
    "import torch\n",
    "\n",
    "def dqn(n_episodes=10000, max_t=1000, eps_start=1.0, eps_end=0.005, eps_decay=0.995, train_mode=True):\n",
    "    \"\"\"Deep Q-Learning.\n",
    "    \n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode, \n",
    "                     in case the agent gets stuck in a state\n",
    "        eps_start (float): starting value of epsilon, for epsilon-greedy action selection\n",
    "        eps_end (float): minimum value of epsilon\n",
    "        eps_decay (float): multiplicative factor (per episode) for decreasing epsilon\n",
    "    \"\"\"\n",
    "    scores = []                        # list containing scores from each episode\n",
    "    scores_window = deque(maxlen=100)  # last 100 scores\n",
    "    eps = eps_start                    # initialize epsilon\n",
    "    env_info = env.reset(train_mode=train_mode)[brain_name] #initialize environment\n",
    "\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        state = env_info.vector_observations[0]\n",
    "        score = 0\n",
    "        for _ in range(max_t):\n",
    "            action = np.int32(agent.act(state, eps))       # take an action\n",
    "            env_info = env.step(action)[brain_name]        # send the action to the environment\n",
    "            next_state = env_info.vector_observations[0]   # get the next state\n",
    "            reward = env_info.rewards[0]                   # get the reward\n",
    "            done = env_info.local_done[0]                  # see if episode has finished\n",
    "\n",
    "            agent.step(state, action, reward, next_state, done) #save experience in replay buffer & update weights\n",
    "            state = next_state\n",
    "            score += reward\n",
    "            if done:\n",
    "                env.reset(train_mode=train_mode)[brain_name]\n",
    "                break \n",
    "        scores_window.append(score)       # save most recent score\n",
    "        scores.append(score)              # save most recent score\n",
    "        eps = max(eps_end, eps_decay*eps) # decrease epsilon\n",
    "        print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)), end=\"\")\n",
    "        if i_episode % 100 == 0:\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))\n",
    "        if np.mean(scores_window)>=13.0:\n",
    "            print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode-100, np.mean(scores_window)))\n",
    "            torch.save(agent.qnetwork_local.state_dict(), 'my_checkpoint.pth')\n",
    "            break\n",
    "    return scores\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training the  agent \n",
    "\n",
    "We need to reset the environment and then reinitialize the agent. The environemtn is solved, when the agent reaches a score greater than or equal to 13.0 over the last 100 episodes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 100\tAverage Score: 0.46\n",
      "Episode 200\tAverage Score: 3.22\n",
      "Episode 300\tAverage Score: 7.60\n",
      "Episode 400\tAverage Score: 8.34\n",
      "Episode 500\tAverage Score: 11.44\n",
      "Episode 565\tAverage Score: 13.09\n",
      "Environment solved in 465 episodes!\tAverage Score: 13.09\n",
      "Duration: 0:15:16.452048\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXeYXVW5/79rn3OmJJNJT0hIILRACJAQAlIEqUoRFDvgtfy8YsEreL0qWAC9oly9oqIognhFVGxIUUpo0msIAVIIKSSk92RmMu2cs9fvj73X3muvvdYup86ZeT/PM8+cs88ua5/yvuuti3HOQRAEQQxdrHoPgCAIgqgvpAgIgiCGOKQICIIghjikCAiCIIY4pAgIgiCGOKQICIIghjikCAiCIIY4pAgIgiCGOKQICIIghjjZeg8gCePGjePTpk2r9zAIgiAaipdeemkb53x83H4NoQimTZuG+fPn13sYBEEQDQVjbE2S/cg1RBAEMcQhRUAQBDHEIUVAEAQxxCFFQBAEMcQhRUAQBDHEIUVAEAQxxCFFQBAEMcQhRUAQBDEA2bCrB9c9uAxvbttT9WuRIiAIghiAbNzdi+sfXYG3dnRX/VqkCAiCIAYgnHMAgMWqfy1SBARBEAMQ29EDsFj1NQEpAoIgiAGI7VoENdADpAgIgiAGIp4iQANbBIyxqYyxfzHGljLGFjPGLnW3j2GMPcQYW+7+H12tMRAEQTQsnmuo+peqpkVQAPBlzvkMAMcCuIQxdiiAywE8wjk/CMAj7nOCIAhCwosR1EATVE0RcM43cs4XuI87ASwFsDeA9wC41d3tVgDvrdYYCIIgGhV7sGUNMcamATgSwPMAJnLONwKOsgAwoRZjIAhi8LKnr4C5330YT6/YFrvv4g27cfjV87ClsxcA8Mfn38JZP32yamNbs30PDr9qHtZsT1cYJhQBGjlGIGCMtQG4A8BlnPOOFMddzBibzxibv3Xr1uoNkCCIhuf1TZ3Y1tWHH85bFrvvLU+9ic7eAh5f5siVr9/5GpZuTCyaUnPHgvXo7CvgzpfXpzpOqIGGtwgYYzk4SuAPnPO/u5s3M8Ymua9PArBFdyzn/CbO+VzO+dzx42OX3CQIYggjiq8yCaRmLbJwdHgT/MT7C9dQA1sEjDEG4BYASznn10kv3QPg4+7jjwO4u1pjIAhiaFC00/vTU8rlkilVjNu2878WiqCai9efAODfALzGGFvobvs6gGsB/IUx9ikAbwH4YBXHQBDEECBNFa63S600QYnUsqCsaoqAc/4UzMrwtGpdlyCIoYedwo1SH8dQer0jlBtVFhMEQSTAThEjEPAamQSlC/JBECMgCIKoFSJGkERmin3SBm9LpdTrUNM5giCIFKSxCETWkCqfea00g8Ki9bu11x50BWUEQRDVpBIZNnaV9EDUkB5YtBHv/tlTuGthuMaAYgQEQRApKKYJFhtcQ/WwCFZs6Qr8l+Fe1hC5hgiCIGLxC8ri9/UUgeIcqroa0Lp/3DFpcpk4xQgIgiCSU0zlGtLvY1fJIoiqZPaFvXk8FCMgCIJIgCc006SPhlxDlRxRMjzlo1FglDVEEASRglQFZZ5rqLborhfVWK5aFooOUgQEQTQ8XvpokjqCmHNUmijdFNlYbjAsTEMQBFEr0sUIXBTBX0/XkG7UFCMgCGJA8fyq7fjV4yvrPQwjaWIEQlf0Fzm+ddcib7usBwpFG9+6axE27OoxnmfR+t247qE3Eo9RVjS/fGwlXly9ww8Wa8ZdyxhBNbuPEgQxSPjwTc8BAD7zjgPqPBI9doo21CKL57FlW/Dkcn9FM9k1NH/NTtz23Bqs3NqFP376WO153v2zpwAA/3nG9JjrhfmfB14HAHw24v2MshYqDVkEBEE0PMUULSYEhaLZNdSSywAAOnrzsecppxCNRzSW8xaqpKwhgiCIeGyv6Vxy11AxlD/qP2zJOaJxT18x/toJ9YCu22lUHQGnGAFBEERyCrbIGkq+HoGtSHDZNSR6F3X1FWLPV062kR3RNdV3d5FFQBAEEUuapSqF1aAKcPmZeG1PlRWBX0dABWUEQRBlkS+mryxWXTqyr18olu7+eNdQnB5QryNbIlFKxK86jh1C2ZAiIAii4Sm6vpw0s+diyDXkPy6k6Ekdrwh4YD85NuF3mDCPm2IEBNGA5It2yP9cD/oLdt0WW0kD5xz9BVv7mu691O3vxQgMUlN+L4TMVYW9COZu6exFXyHeEhDEuYbE6+JyRY1FoBt2XyG9cisVUgQEUUE45zjoG/fj2/9YXNdx2DbH9G/ej2//Y0ldx5GE219Yi+nfvF9bvHXQN+7HJ3/7YmDbd/65BNO/eb/WlaNjx55+TP/m/bj5yVWB7cKK8ODAgrd24phrHsEn/y94zSjiFIEYm68QwhaBKuwfWrIZP5y3TPtaNSBFQBAVRPzof/fcmvqOw5Uwt9V5HEn456sbAABvbtujff3xN7YGnv/phbUAghk9Ynavs4A2d/QCAO54yVkFTBSUqXUENge2dvYB8GfjSYgz/lRLQFZafh1B8JhHlm72HtMKZQTRYAwUR0wtO1eWi3DnJB3ziBanIUJHr68Iip4iiD+/EKx5xSLg4KW59BLGCHSKwHuoSHv5vSBFQBANxkARwELGVVqGVCPmIAKlUe4dmTahCHr8ql8xu9edQsy21QIyXWVxqMgsAWldQwGLwFBQVpR0FLmGCKLBED/seusDdfZbKapxX6J1dHKLIAcgqAiEv19XvSsUjZqNqQaLbc4TKyP1uCSvexZBIEYg+gkFPyhZ4ZIiIIgGo94KQFDKzDYJ1bB4hOummNAt365xDRUiXEO+cnb3EccoF+Q8uVUiExsjUIPF0mW91NKI9ZMpfZQgGoyB4xrSzzTLpRp3Z6r0NdHWHHYNFSOCxd6MXMna0bqGSlAEce6yYqRFIMaoHCNtoKZzBNFgDBhF4AUhK33eKlgEQhGorhqDUPaDxVKMwJt1h/f3fPTuTFzcgjZYXML9xR3hZw25/yUF5I1XuW6tv0ekCAiiglSjjuyeVzZgS2dvqmNKmdkmQZZPr6zdhfmrd5R9Ts81xDkWb9iNZ1Y6awTIPvzbnl2N3z27Gr35ItqanRjBbo1F8I9XN3jpouprXLEIevM615B5nM+s3IZF63eHtv9w3jIs3dgR2LZpd6+XFiu7hv46fy227+nzr4mgAtu4uwf/O28Z3tjcaR5IFaCFaQiiklRY/u7uyeOLt7+MGZPacf+lJyY+rlqLmsgz1ffc8DQAYPW155R1TuYFi4Fzrn/KO6eszL51t1Ogt72r3/OZyw3hxL67uvO46NfP4+H/fEdozOJ0ptm2zXlkbOXCm5/3xibzt5fWoae/iBsumuNt+8CNz2Ddzh6cOXMvb2yvb+rEnS+vx/7jhnv7cWVMf1+wHj//1wrjGKoFWQQEUUEqbdKLgOam3eYlE3UI4VNp93I1DA0vz185eUGt/AXQky96ujYvuVhkAa5WKBfUYK3hHjiAosYkSJIy268ct25njzcucb3d3f0AgC2dvkWg9iGS4x61hBQBQVSQSisCv94onUSvlo+5GnUEGUMdgc69Nawp4wlNWVHI41KHaCvxA9M9OHUE4e1J3GymT8e2/c+iO+/0L8pm/L1ViyDJimjVgBQBQVSQSs+YS+1E7BeUVdYkqIZF4BWUqQVfmosNb8p6QjNfkIKu0oRcTcUM9foxxAE411cWJ0nFNenpolSb0OWmu2alfFBVYXf0xK9/UA1IERBEBan0jFkItbQuHiG8Kl9QVo06Aud/EotgeLMf1pSzfgKri6mpmEqQ2GQtceiFfpJbNincYtHPRBKKLWv5YlecWuyze7C5hhhjv2GMbWGMLZK2Xc0YW88YW+j+nV2t6xNEPai4mPROWJprqPLB4gqfEH7lbL6oby0tk7WYp4wKgTRM88CEvlBdRCqmOoKizWMVYJRFoI4t6BoKjmkwuoZ+C+BMzfYfc85nu3/3VfH6BFFzKu2b95crTHlc1dJHK39esapYXnHQFzUO+4LtB19lxRG4Xa4e47afiIkRmFpMFDmPXajGqAjs8DkDriGltmHQBYs5508AKD/JmCAaiErL31J7BvmuoYEfIxDBYrXlgy5rqGjbnrssb7AI1BiB6hIyuoYMFgG307WlVq+tnlJeVMevIxDB4qETI/gCY+xV13U0ug7XJ4iqUemZeKmFYUm7jy7esBvTLr8X6zWLwuhIYxH8cN7rOP8XT+POl9dh9nceDAl6gZggq64h3b0XbK7NGtJ19Lz4d/Px1b+94lf0xqSPnn39k9oc/uOvfQSHXTVPf5ALA8OWjl4c8PX7sOCtnYHxqt8JkT2kjoVzPvgsAgO/BHAAgNkANgL4kWlHxtjFjLH5jLH5W7duNe1GEAOKSntO/KUMq5M++sfn3wIAPPr6loTnTT6GG/61Ei+/tQtX3rUYu7rz2GNYCJ4xvWtI544pGlxD8u2Khw8u2Yy/zF8nZQ2JezDfhE75mMat8vTKbSjaHDc+ttLbZtvhIrVu6XxejMB1IaVZK7mS1FQRcM43c86LnHMbwM0AjonY9ybO+VzO+dzx48fXbpAEUQa6NsjlIORCWgdPsdQDYygpBhIzBiEMk1oEiHMNKWNUW0zE3UJTNlosmhrbiWygHmnGLysugewa8pWTXvHVipoqAsbYJOnp+QAWmfYliEak0r9lv0J4YGQNVUNUifdMFYQmi8BzDRX1rqHQMQlbTAiyMZF5XbyAc/+4XlkRGGoTBEL5cfCQIqwlVes1xBi7HcDJAMYxxtYBuArAyYyx2XC+T6sBfKZa1yeIelD5rKESYwTVWo+gHE1nOLRotAjCgrEg5eXLFoHONSQIrQcQ895kYpRuX8FGSy4T2MbBkXULImTXj3D5mBD3bPOwa6yWVE0RcM4v0Gy+pVrXI4iBQKXTK0vtGSRkaqWzhsrxDJncZkJQh+oINIKxaNt+G+mivqBMHaO6nnGcLrNiLYIigFxgm839+gDZNeSku5ov6LmJODcG02sBVRYTRAWpVrA4vSIo7bik4ykF06FirIXADF8/ky7YvjqRXUeRAWDltThlHesayke7hnr61RiB+XrCzWRzID9UYgQEMdipeB1BiT2DqlH4BZSmCOLbPzv/+xWfvxojYCwoWGWLIMqroiqU8i0C3czdj130BNJDo11D/Z4i4Mi7j1tytRfLpAiIhmF3Tx4rtnRixZYu7O6uT751HEkE5cqtXdjltiSOQ8xm01YWewVlceNImbcedXfLN3dqe+XIfnAAWPDWTk9R9fQXsXiDs9hLQWkrrQrQpoyFbV39WLl1T2j/KMUnn2fh2l3YGfPex8UIVmwJLxpjc/8939UdXDAnSvH0e8Fi4JV1uwAAw5pqv0wMLUxDNAzv/+UzWLGlCwCw//jhePTLJ9d3QBqSKILTfvQ4JrY34/mvn574fGl9/UmyjU770eOpzglEC9wzfvwEDtlrBB647KTAdn9heY77X9uIz/1hAX7w/iPwoaOn4st/XYjXN3W6+0k+fzucNdSUtXDHgnXec1OMQEV+7b3uYjpRZGK07i8fX4UzD5sU2GbqXGpreg3JCItgS0cffunWH7QqgehaQBYB0TAIJQAAq9xZ4UAjqedkc0df/E4oPUun1PbVcZiGI4SyEOoyckHX6u3dAICV25zP8qU1fhWunCRUsO1Q1lBTJiiuAq6hiDirzjVz2N7txv0tg1S8/oIjceCENm0MgRuuUyhybdBbIBSBnHLa2kSKgCAamkq75kvPGqptjKAzQY8c3bHBlsxS8FdjEagz9VJcQ4KWrFnYZg2aYMywJkwa2aI9n83196dzccmIeIN8b2QREESDU63uo6kri6u2HoF+e5IeObr3JhNYpMXfXuThmbQ6EzetR6CiE8RRLTtMnqHmnAWLMa3S4YYOpY5CM5srwiKQh5M2HlQJSBEQRAWpVkFZ2hiBL6wq3X1ULwST9NHXvTWycJcFrC5rKJNRFEGgxYT5ujqlGPV2mmIEzVkLGYsZVywztbCOzBpyfVryPvUoLCNFQBAVpGotJlIfV9lxCHQy0ObJVtZKYxE4aZfBm1BdNkWps2dULEW8lpOOj7IITLq8OZuBxZj2vTUFhe2EjeTkY6MsiGpBioAgKkq1LIJ0x1XLNaT1g9s80Vq7OnmYMazfW9BZBJqZunAPRbuGnP/yymCmgLC4tg7HItDHI5y1DPTnShKvkfeJCi5XC1IEBFFBqrYwTdqlKqsWLNZtC7qGTNfWBosDyzZK+2oEqC5bRwjNSNeQqyxkKyBKOJtm5BmLuRaBSRHoFtJJZhHISiRPFgFB6Hlk6Wbt9nte2YDfP7cGnHNce//reGXtrhqPLIgsBD9z23z8558XojdfhG1zXHX3IizfHE6vjEIIiDe37cG37lqUWMCXEqt4avk23KBZmEXmuofewIurgwsP2sqCKqYMIt1MOiNNzW01RlCMtwi+/JdX0F+wIwX79Y869yQfvafPvMZA1Izcshj6CjYu+eOCwHbHlRXeP66yWN7Pe1yHlkOkCIiG4FO3ztdu/+LtL+Obdy0C58CNj6/EexIUDFUT+Tc/b/Fm/P3l9fj7gvVYu7Mbtz67Bv/v1hdTns85YX/Rxm3PrcHand2JjisltvDRW57HD+cti9zniTe24oM3Phu6ltxWoTuvVwQ2l1JE3X/BYLF0Th5uy6yzCB5YvAmPvr4ldUuNrr4Cbrhwjva13nwRN350Ds6dNdnb9onjp2HvUa3IMIYNu3pw76sbA8dwhHsaAUKhxUt2WVn84ANH4IJj9kl4J5WBFAExKKi9V1WPTiAVOZcKvMpz8ajtj43HxcQIKtmLyOZBQaZryiaPScYUIyjaPNTTx5TNo1sXWIe8S2dvAeccMUm7356+Is48bBI+cfy+3rarz5sJy2KwmCGGwPUuMd3i9Trke99v3HB8/32Hxx5TSUgREIOCavXfT4tuFEVpRlhq0FeQ9D7tGMVTyRCCmhljWujdtqXxuP+MFoHN3XbPPqZCL5uHl4OMo6vPnOUkUjp1mUWmhnS2oY4geYzA3yebqXCEPwGkCIhBwQDRA1pBLbdOTkvazpmm41QqqTjVXHlVgEddM9IiyCezCJxVy+LvRz6612C1yOgUgakhHYfBIkgYI5D3UVtp1AJSBETDIv/4B4pFoPvNl9PuQb2ttMFikwWSZkxx17SVwK7JIogvKJPOyZO7hjivfLaW6Xqm+gNusEpUa8n0ecjf3ywpAoJIjrwk4ADRA0aLoFRUgZ1UgAvhbXIypHm/4pSsraROposRmLOGVMvCNI64xV9KJZ1rSP/ZiIC3OJVpti8fmyPXEEEkR65mTesjrhbaYLH0I087zFJjBHE1SWkEZ5zuKXI1RpDcNSQLvbhgsUmhJk3RTIvOIjBN1jn0ikDcg1AAptXP5M8rF1XtViVIERANi1zEJLfxBYA9feZK16jX5H26+wvY01dIJWR08lX2YaeNFqiKJXGw2PbTTnXpi1GKs6e/CM659z71x6Q/qsVfxmCx5pLyDFtdoawvb6OtORvYpqO3YButkHLQyWzjojUGZSQWwWnKOqLWmPkkHRu3Qlo1IEVANCQWCxYuyYrg3lc3YuZV87Bo/e7QcfMWb8LMq+ZFFp5t2t2LmVfNw6FXzsPMq+bhy39ZmHhccTGC1BaBHf3cPA7nQtu6+vGZ214Kvc4jzjPjygfwl/lrMfOqeVi1tQvn/fyp2GvJs3XxWajyTGctyTPkRes7vMdF7riG2lt8RTB6WJP2+t+6a1GsslKJavUsxqQTyKbmf6INNWNO+qfghn85i800u4rA5P+vVtvwpJAiIBoSi7GA8Jcf/2vZFgDAkg0doeOeeGMrAOBVjZIQrFOKtu5auCHxuOJiBLVzDfn7PfL6ltDrofMozx9b5rxPizd0xC4CJHLlxaxXWAQ5RejpZJ0xJdR2ztPemvO2XXP+YThq39GRY9Fx4kHj8Nh/new9/8q7DsaTXztFu+8TXzkFz15xGgD97N8YsIbzHmQYw12fPwH3fOGEwOvCNWQKNovP40unT/e2PXP5qXj2ilMNd1VZSBEQDYnFWMAd0NPvP47qwCyEURrrO03uvz5GYEe+nuZ8aYPFxtdDCib4uihc60rgRhMWwXB3Za0+VymHFYEuWKw/Z9HNGmpv8RXBxPYWnDlzr9jxqIwf0Yxp0ix92tjhGNfWrN13n7HDMH6E85o2fTQic6nIOTIWw8hhOUyfOCLwulCSphiBeG+mT2zztk0e1YpJI1tNt1VRSBEQDQljQV90rxSgFH543Q9ZCMioNsQqbSkWE9fJebmOIK0HIFxHkNAiiPGUxFkaLTlHNHQlWnnMUXZi0XXxuagyLy5rKDA+20ZfoYi2luB7X0o3VbWoLm5NYoFuaOb0T6BY5N651e9XXIxAWI2V7habFFIERENiMRbITumRUkmF1NX9poQwipIFqvAd3pxcEZhiBKUmNZVaUBanMEL1CcrzZncpx84EFoFoEDdMWASuIlBHoBuSKVWyaDtpqEIhCdIocP+Y4POkikDsJ4/RWFDm1hGI19VrxCkC/72pjyYgRUA0JJZiEchNzzzPUIRrKGrFLzXwOLw5+Rqyxspid3vafHd198Suodjc/+DrqgvKcw0lsAhEjKA5ZyFr+QpadU/pxmT6FET6aLOytnApCTXqR520XksoHTmOEaVEbJt7AWZ1t9j0UTt+glJNSBEQDYkTI9AHi6OqarlnEZh/cWoqYlsqi0ATIyj6rqG0hkHJweIYhRHnchICy9STR1YcnDvKLmNZaM5a3vsXHrvmPIbx2W7WkMi2EZSSWqm6hpJaFZ4ikCwCc9aQYxGI903dL84i8LrF1sk3RIqAaEhCMYJ8uMpY13BNCLyoWaGaB5/GNWSKEYjrpk8fVYRpYosg+vU415CoiDUFi+X9RT+drMXQnMt47596Tv16x/rxFTyLIPhBlSImxYReXCqpa0ggVwObXUPOZ2VSVE2uZRPVQRWol2OIFMGQ5J+vbsCa7dEpgfVg0frdeGxZONVRh1ggRCA3EYtyDRUjlIRArYxNpQg0c9w7FqzzxifPpF94cwdufHwlvn//Umzv6guO0+b4zVNvBmMf0BeC/XX+Wtz4+MrI3kt9hSKue3AZHl6yWft6eH9nvCKNVCW0iIxtI2MxxyIwuIbk8f3umTXo6isYC+z+9MJb2NWdR7OS7590xrxsk7wAUGnBYrFSmWwRxFUWmxSFUCZxFkEdiooBAMm/4cSg4Qt/fBmtuQyW/veZ9R5KgHf/zClcWn3tObH7qsFivWvIbBFEyRPVImjKJv91mlaX+vWTqwAEXSEf+pW/wMv0CSPw/qOmeM//8coGfOefSxKd/yt/exUA8L45e2PCiBYA4VW2Fq3fjesfXQHGgDe/f07I0lD1i3gPuvvj20WIFg+5jIWmrOUdG3INSWPvyRdx9T2Lje/tMyu3AwBashZOPng8po110j+TunVuf+Et77E4RBxprA5WGDO8CaOH5XDVuTOlc0W4huygoD944ggsc1eka06YPpp2vYpKkfgbzhh7O2Psk+7j8Yyx/ao3LKLa9OTNS/U1AhYL+vIDQicia0i8Fh0j8N+bjMVS5f6bfPheJo3hdXWdXLmPUpLzA0FBq55PKAZxeMhto2zoV5ThrCkjA88DnULdDpsZiyFrMRSKTnA87H4Kbtixpx+cA+PamvGe2ZOho701h99+8hhcfZ4jjJN6deSAf6lZQ83ZDF6+8p04+3B/ARvjsdy5P/n1eV86yXssrIo4i6BevqFEioAxdhWArwG4wt2UA/D7ag2KIOJgLOgakgWZcDdoXUN2fPaOfN5huUzZvYYAPwvHdCq1oZq6TKMgqkeQfE95ySKwWPi4cA+j4LnUzCl15i4fLscIchkL+aJtWOReN14Oi5kVs1xZ7NxLvKRsyloBRRYKFpeRmhO1HoFQhjKilYXIPDJVUvtZQwPbIjgfwHkA9gAA53wDgBGRRxBEFXHSR/2Zu66Ng7agLEEap6wIWpsyifv7RJ1XuAZMFkHSdtNRweKgIvAHnbWskEspLpDbp1iMahqnuth6oehkDQlFoBu/TvnYtqOwTeJPriwGkGjG3KwqAuUYk3smCVErlNk2D1kfIvVYXNMcLHbHWvLIyiOpIujn3FXfABhjw2P2JwYoSbNOBjqWaxGIH5h8X1EZGGK3KOEuK5jhzdlAi4g4TO+umFGbXld9+uaWyxHXll6TO45mLBYS9OEYgeIairEIdDGCrMWQzbBAllTU2Dnn4OBgYEbfe3trMIyZZMasKgL1mHJm3UbPkJs1pAp6UW2dSegaGuiVxX9hjP0KwCjG2KcBPAzg5qgDGGO/YYxtYYwtkraNYYw9xBhb7v5P30GKKIuBspJXuYheQ8L01lkEUXUEUcJdjj205jKxvf1ljIuniJMYzqUKZqNrKEKABywCab+sxWLrEUKuITVgHtFAzlmPwEYmI7uGdIogPAbO4bqGQrsDCFsESSbzzdlMZDfStOmjSY7lcOoIVCUjqq1zikWgfjfF5zOgXUOc8/8F8DcAdwA4GMCVnPOfxRz2WwBqWsrlAB7hnB8E4BH3OVFDBolB4NYRFNHi/tACrZ7d/3o/tVAE5nPLrqGWnJXKijLtKgSTSVGoFoBqIfjnNwtw+XFeuodMhmlSOaPPG5c5FVBAthwjYMgX9b35Q2O3nTEzxowCcGQJMYI411A5isDYPdR2voPqwvMi9Vh8vl6La+k8FkPsinLVJjZ9lDGWATCPc346gIeSnphz/gRjbJqy+T0ATnYf3wrgMThBaKJGDCqLoOBbBMGe/+Y4gOcaiowR+K6hrGWlDBbr9+039N8RqBZK1GpcpufyY/n4DGMhBRVX9RuyCCKCxTaHlDVkoVAs6KuINconKrAPhC2CJBNmNVhcWdeQ+VhdHYGwCEQarogxyHtlM5bvyhuoriHOeRFAN2NsZNy+CZjIOd/onncjgAkVOCeRgmopgn+9vgXTLr8Xu7v1aY+V5q0d3Xhm5XavIZjONSQE+M1PrMKR33kQgBQsjhDu8mzYspzzLN/ciWmX3+v9HXPNwwCAi383H7O/8yAuSlGpAAAgAElEQVS2uQVhprc3Li+/YHPM+vaDmHb5vfj+/Uu1q4oBTg3Iq+t2Ydrl92L+6h1aBQgEXUsZi4WU2XtveDrwPBQjUBeOVwSc/D369O/mY93OHilriHuL+ciT78v+vDCwqpyIOlqWOT8/3H00mUXQJ92/ekQ5wWJTQdn6XT14/I2tIUW/z5hh7nEscO2p7nbAea/99ugD2DUEoBfAa4yxWxhj14u/ag6MMXYxY2w+Y2z+1q366kYiPdVyDf3isRUA4BXQVIupY4L92c+btTcAvWtIyKpr7luKna6C8l1D5jeiv2CjKWPhlo/PdSwCznHvaxsD+2zpdAT/g0s2Y1d33qvUNtcRRNdtFIrcqx341eOrIhe8/+nDywEAtz67RrEI/H2CWUPBYLFOyegqkWXUCmDd8DKea8jGw0udCnE122jFlq7ANW3uBIt1svmdh04MuXESxwhq4Br64mkHhV4/Y8bEwPNvnnMorjn/MJx40DgAzuz/hgvn4PZPH6s9/0DPGroXwLcAPAHgJekvLZsZY5MAwP1v7CfAOb+Jcz6Xcz53/PjxJVyK0FHtJfHSLrySlgkjWvDuI/wCn+MOGAsg6OqIcg2p1oKOfJHjgAltOG3GRFjubFpXASvfa0dPwb2m6ZzRmUeqpRC1vwgE5wu2EiOQhb3/2FIUgS6QGuca0mX8qPQVbMfNIe1sajMtxsvhCGrdTPgc6XMWJJkx57IM/ZIiC7mGKqQIzj9y79DrJ7gCX9DalMFFb9vXe845xzlHTMJeI1u0569X07lELSY457cyxpoAiHXUlnHOS/EB3APg4wCudf/fXcI5iDKolqAWRTvVOLs6ZvnHKIJzRUnwiUc6YS+2RcUICraNJpHuxxyBpc5sgaBAFS6PuBiBie7+YHM3U7DYec32xinfoxxmCGUNSc97NQu9q0pTVRbqbelG191XxLDmTECJRbXnsDncrCG9RaATiklkeC5jBcevWgRlCFvZmtC5mEyKKqmAH9BtqBljJwNYDuAGAL8A8AZj7KSYY24H8CyAgxlj6xhjn4KjAM5gjC0HcIb7nKghVTMIqvgFVscs/1hybqVm0CII/ve384QWge0tMp5xWyboBJocS+joCbqeovbVsUexCKJcQ0JJ9BVsbf0EoGQNKRaBzk0V6jWkKIuwayg8vj39BeQsK6AIousehGtILyj1Ajv+i5a14iqLY09hRLYmdC6mcgV5veoIkjad+xGAd3LOlwEAY2w6gNsBHGU6gHN+geGl01KNkKgojZg1pAptrUWgEYi6IiqhMKKCxfkiD6T5GYV73saI5iw6+wrocBdw0e3alLHiLQKl3bPaKygwPvc1NV8/UFBmK4pAOp0q5AFNjEB1H4UyfsLj6u4vOgVlkjWjdk9Vz8EBgOkFoE6oJrMImJI1FHzd1OYhCfK5dLP/ONdV/K9vYAeLc0IJAADn/A04/YaIBqPalcXV0DOqkJJnj7qsIYEuRdILFkcMNF+0/cXGM45bRW25ADgza6GIfIsgfD65I6eJPSlcQ2LGnS/ywH0YYwQsWFCmG0vINVRQLQJlf82N7ukrhNwyanPDkAXjuYbCAlDvGkoQI8jE1BGU4xqS8/9TrGmc9Ir1cg0ltQjmM8ZuAXCb+/wilBYsJupMtfSA+P6a+suXQ9gi8B/nXBdOoOmc+1DXT8eLEcS4XmSLoMi5Vnj2FXwfvYgR6KyHpqxV0WCx2LdQtAMz/aiAsPz+6FxDcd8LNfah06Pd/UXkFItAJR9QBPBcQ0ln/0km89kMC9y/6hpiFXINpbEIkv4qBvoKZZ8DsBjAFwFcCmAJgM9Wa1BE9aiWayjN93d7Vx/e2NyJxRt2G+sOlmzowK7ufgDh2X7QNWS5+8h+aWf/Z1duCwiwl9/a5XXllBWBOg41RmDbekUwf/UOT4Au3diJZ1Zuwxua9NkkriF1JbC1O3vM+7puqH7FInhu1Q4AwJbOXqyTjlctgr/OXxc653Nu/38T6rdGp/C7+wvIZqzIFudyNs+u7n4s2djhNJ3TxQg0miBJv34nTuGPr5KyNWARlGixRDFgK4ul/X7KOb8O8KqNm6s2KqJqVDt9NMnU56yfPunl4R+y1wg8cFk47+Ds65/E/uOH49Evn6zM9nlgVih6uMgTYCHz7nttE37/vL9AyQU3P4dxbU3O/pJgPOf6pzBjUjvuv/REAK5rSCgCxtxlE8PC7Wt3vObtt3DtLlx48/Pa+81lGbr64mIEwfMv3dhh3FfUG+SLwWDx/zzwOmZPHYV/v/XFwP7DmjKBGf9tz60JnXPVtugV606fMRF3L9zgPdd9jT44d6pnoQmO3GcUXn5rl/dcVogbd/cCcL4D+hiBzl1kHqMonMtlgzsdf4CT0nnhsfvgV4+vCi1/mQb5u5cmhnHopHYAwLtm7hXYfuQ+o7B2Rze2dfW7xw9si+ARAHIlTyucxnNEg1GtWHGalZWEEgCA1zeZC9BWbXWEk+rPZxqLQG7RIM9W39waFHBeYZkryYTFIAvegtQzxhIWQd7GsKYMPvX2/QLn6y/a+OQJ03DyweZal5wVbxGoMYJ3TB9vFHrCOilomrtt6ujxMpA+fty+mD11FHKZdP2SVN47e3JgcRYgbFne+fnj8dV3HewpZgD4j1MPxHmzggvOmJrBJRX6uv1Wfu9svPHds7D8u2dhxTVnBYLB15x/mFdrcvmZh2D5NWdpU4GTEugRZDEsv+YsfOL4adKY9R/agRPa8MZ3zwq8HyuuOQt/++zxuP9SfyI00LOGWjjnXkkg57yLMTYs6gBiYFLtrKFKnD3Utz4iRpCNaDHhjCecOSTvr7OQnBiBv7SgiBEMa8pq00hbchmMHtZkvB/VZ61DjRGMaMmiJZuJdLPomrvJl7EshqasFYiNlEJzNhNy06ifUWtTBowxTzEDQFtzNnScTiGaZsH69Ey9C0nsa4EFitjkrqmMscgCtyRklBhBLmMFhHdUsFf97oj3Sq5HGOhtqPcwxuaIJ4yxuQDMTkxiwBKVLVMO4gtcCUWTV4KN6phlYSB+6KY0StNwhHJRrwU4s9amrG8RFG0nwNqctbQZJxnGIt0NqrtEhyqoZeFmol+zEpisNHvzNizmKOdyPhddgFa9rrhHWdA2Z61EisAk/PQ++JjBIvh+V9rVonYNDW9Lfz35mHqtWZzUIrgMwF8ZYxvgfK8mA/hw1UZFVI2qVRa7399KxCBM6+0K5B+OEDSFQGVxMKagQyiXvCZfv1C0PYsgwxiKto2+go3mnGX0C0cpgmwCRaCSYfGKQOcakp/39BfAwMC5fn2ApOjcHerpRGFff6CqOBN6v3SKV3c+57q6wUQOFUDw/a70DFsn9Jnh9aTIWUwD0iJgjB3NGNuLc/4igEMA/BlAAcADAN6swfiICiPkdLW+cNWwCAKzfQTHnnHbExSNriE9Ql7pUh0LRR5YbLzoxgiasxltnxrLYmjOmf3OTSW4I6wEFoHWNSTdfHd/EZYlVs9KPQR/LJphqJ+zCNDKBWQcPPR+meopdFlIOusr0ZrFmfJm6FGoriEgmFJayuXKtSgqQdxU5VcA+t3HxwH4Opw2EzsB3FTFcRFVotpZQ+UIHIGaQx9VWWxZLJQeaerRL+O7hsID7pezhiynl385rqFSKlkzEYu1yOOMKgTryRe9yuiyXEOacajnE/coxzr68nbo/epP0N7Cu26KXj6BsVTVIgg/DlgEJVSEyYcM1GBxhnO+w338YQA3cc7vAHAHY2xhdYdGVIOo9XzLQfg2K+IairAIAP+HEyj60rShBszN3goRikDOGvIsgoKN5qxltggiqpTUVauSkMkwY+97QV4pKAOAXVI9RHd/0UsdLSdrSCd8Q64h9x4DiqBgh+5d5xoyKalSW0wEg68VjhHoeg0lDBYbzxmIEdSHuKlKhjEmlMVpAB6VXksaXyAGEOI3V84P5LdPv4k/v/hWYFuaGEFc5kbYIgi+rprkLMI1pOu0CQB3LFiHrZ19nmASP2Cx7KIIOAol48QIMsYioqiURHW93yRkGItthcA5cOU9iwLbRI0B4Cg0izFwlJckoBvGJX9cEHie06Tx2jy8hm9c9lTwurqU0gSuoawcLE58uUTIn4kYS7muHfmQgVpZfDuAxxljd8PJEnoSABhjBwLYXeWxEVWgEhbB1f9Ygq/d8Zr2tSQCJy6LRlUEagql+LFkpB+irukcEN3++WePLvdaOovZnQge5zzXkHNPe/oKGJbLaGfpGQtozvkvvGN6sKbgqGmjQ8e0RsQUxHgyCSwJuVgLcPr9AMDkkS346YdngzHR5TP2VB7/9c7pgec64bZme3fguZj5f/XMQ/De2ZPx/jlT8LHj9g3FOYwTBc1mnSJMIidFARmQPAvncycfgH/+x9tj99N9d+UrlBsjGJDBYs75NQC+DGch+rdzPwXDAvAf1R0aUQ3ED7FqweJEFkG0IlBbSsizXMCf5Xm54yx4jOxailJMnPuuCiEwxLHCvZBxlUxHTx4jW3MRFoFzT2cdthe+fvaMwOvnHjE5dMwtn5iLr555sHFsVgKL4PQZ4ZVexftw5bmHYtq44Y5FoLiGpk9sM86UL3zbPpg+cYQylshhAPCzhsa1NeMnHzkSP/rQLIxo0b9fKmKBGgC44qxDpOuWFiM4cEKbtH/s7gCAL50+HYftHb8ar6zwBcE6gnLTR+tDrHuHc/6cZtsb1RkOUW28rKEqfeWSuYaiFYE6i+8IKQJfAYjnpr78prV/BcL6EL9FoQh8i8D5v6snj5HDohSBP8NXZ8G6GAFDtKDPZlhs4FH3Pvr3479HNg/2JGJwMpJsjb/eYuHxJxFuprGWsyykvrI47TmSHZBUfuuSAsqvI9Cfq5aU0YePaESqvZRkEtdQXK8XNcVQXvAckILFriBkqkVgeKxD1CwIgSV82DkvWOzs11+w0d4SrpQVx8pCWV25yrSASZSQTGIR6OoTCl7MQxzLAu23AfPSkM7eYQVUjt86yULxnOu/l0kri3WIY5MOPel5dbGgQNZQCW+V/P4OSNcQMfiolmtIfJkrESwOWwR+Hx7Ow0E6sa6wwBQvULE5R38hKDiFYhBCVhaK7a05YyZLoLZBtQg06aOMsUgBm7HiZ9M5zeu+YvPHxjkPWEwsoljNYmHffDkB1yTplBxyEoN83dIvnFG+I3EkvZLONYSAIC/vh1WvymJSBEOMahWUidNVxDUkuXM45xqLIDjbU4PFRUO8QKW7v+gJTnGufEFxDUlvVHtLzhNsAXM+xhVksgiiZGSSymLd+1jwFL3/HqkFZQzmxVl0SqISAjkKrrit/OuG901rESRVYuW5hvSPS4EsAqImVNs1lKRwKTZrSLIICm6gVkZN31aDxSaloNLRkw+5UvysIeEaki2CrCfY5HvIMF98MY3LR+ceYSx67pekslhttwz4MQJLmhFzBAvKLMs8U7e0iiByGJEkqaUzfULaFcoSSizxnicVrEln8rpU4KDyKtMiIEVA1IKilz5anW9ccDZuB2bzYqGZnDKr6u4voFdKEZUtgr6C7a0HLLCUHzljwXWFiwljBLt78t61hLDzs4b8ymJBu5QFE9XYTHUFaRdYiXn7s1Z8ZbHO5bSt03mPZddIKEYAs5LRKbJy3B2JloXkemVQTozA/45U9nuur22Qrlu2IiDXEFEDquYacs8nC+FL/7QQR1z9IADggUWbMPs7D2H+6h2h3juHXjkPp/7vY95zOUbQmy+iU3ENyS4h538wOygYLDZnDXX0aiyCoogRhC2CtpasJ2Bk949lMUwe5SzXcdjeIxNZBHECI4lFoGuJvcxdIc07lAWX6ASc9++EA8eFjhXHqWMTz084cGzkeHQkqaqWlUBQqOrHl+i6QhEk270sKtkiotIFcImvW5/LEvWiei0mHGSBc+9rG73Hz61ylkJ8dd1urRDc4K5WBQQVQaHIA885pBiBuy2qoEz4/GXmXXYSTj54PDp6Cr4rxf0FioylFrfgS551O335nccB15AFzJo6Cvd+8e347EkHJMoakre8Z/Zk/OYTc73V0wB9ZfHdl5yAGz96lPc8KiMnEFDnwbbRjDH88ANH4OH/PAnPXH4qHrjsRIxszXn7m1xDN39sLt47O1gTccvH5+Kpr51iHIf8WU9s1y9qyDnX9hvSK8u0MYLqS1Z5Fl/+UpVkERA1wFaCiZVGlz7KpVYDurYDKn3S7D5ftDVtqJ3/srArGmoHdEtMHjB+OA4c34aO3ryfd6/sL4KC8oxWdtfIGTti28zJI7Uzea2vW9o2qjWHUw+Z6CkfQL8ewT5jhgWURVSsRY6j2GrWEBxFd+CEEZg8qhWH7NWOSSNbvLGGsobckw1rygaKtQBgzPAmTBltXqNK3MOkkS3GxXs49N1H9U3njJfSXrfWefkULCYaAr+grErn1/jkbe7/QGzOY7/ssgWQL9qh/jTqj5sxc4BY1/Y4m7HQ3ppDd3/Ri02Ic4r9hSKQg4NNWctXBNmoGEH8uxv3Hmhn5opy0LmGBHLWjNprSHdtcS5dfYO8v6rU4jqriveGc3M6LFesFf9Y8/niSFtHUA6VbCNNioCoCdVaqtKvIwi/VrBtb3aX5PJ5xd9fsG2tMBIyKJQ+yqMVAQC0tzhF9dv3uMFV4RrKC0XguoYUi0DsJwt7VcAlqaaNDRZrKoszFgsI3qSuIbUNtU5YyVk2UemjYeUUfR9if5tz43h11oBpnEkFbdqsoXIIfjfLPBe5hohaYFfJJBBpqUVNcNa2pWCywR8so1oEhSJXsnSc/+JHY7GgD1x2JfUZ1vxtd33iO1xFIMbnuYZyYulFSfBm/BXKdOMRJHG7xQaLGYMaZ1Vn61ErnwWCxTZCriEVr4COhdtfR+XJxym9rKcIoi2CpDGCpILWswhqIFjlK5RdUEYWAVELytUDpqZyYrMuRlCU4gKcx1slQUXA0V+0A4U8uoIymUKMawhwUkEBYHuXYhEoriF1IXRxLd1KVWmIO8SJEQR/nhZjAQslqkJb9ZEXA+6X8P5NXruOcNpqlOsjLj3Ut2q40Y0UcA0FrhveN+lbLa5ViyycSsYhqNcQURO8rKESv3CmvHzbswjCrxVt7gkM205gEciuIdciaNb0mPcmvRG3YhqvsAi27+kLnMtXBOGsoWyGaX3PpTRWi/vBZzQzczWAHB0sFjNihFxDus9ejilE+f3VY+NaSHife4RFEDy/dO4I11BsjEXcTw00QSVld50MAlIEAxHb5rjpiZXo6ivE7wxnndgbH1+ZqL2DOht/btV2PLV8G97Y3IlfPLYCf3lxbeD119btxrzFm7znpmuI7SJD5UcPLgvcjx8sTm8RFIp2QOj57ROCs940tLc6MQLfNcTQmy/ipw8vB2BwDRmKvMrtOKlDlzVkMRbws0cpAtla2ri7FwuldQu0riHJgojy+6tGSFxgXNwD59xYU2Cqdo9UBJFXrW0dQSWpl2uIVhkbgDy0dDO+d9/rWLO9G9ecf3js/tc9tAw3P/kmJrY34/wjp0TuK4Sw+P1+5KZQl3G8b87ens/43J8/BQBYfe05AMwFWr5FwLF6+x787NEV3msFm3tCO8n6ufIC6E7WEMfwZj+1UmS4iLGkEcTfcNcKGN7kfPXFWgc257jlqTexrcuxEFpci0B2vzCmbw2t23bopHYs2dgRMZJ4ASpm5mOHN7ktsKFYBAlcQ65QX7+rx7+y5rCMJDijZu5NSvfNWMsmSYwAemWgU0iB1OGI71E1s4a+cMqBWL6lUxpT5S5ClcWEhxCEnb3JLIKuPmd/eb1YE0KOR33hotoymCwCcd6izT3XzokHOdWrdiBGYMoR8dnV0y+NxUbBtgNpnMK/3+3ed5rfzqdP2h+AP5sWS1kW7WD1rRCyakDWtHi9yt8/f3zkOOIsAsvyF7t535y98eiXTwZjatZQvGtIp3B0gstLH7Wi218LS0oQ5+5JlDUktZgIxgh0QYLwftHXjdmxBP7rXQfjV/821x9SJV1DFCwmBEJUJnVvii9Pki99kvRR3YLuApOSKEoWgcjaEYKsKLmGigmWTZQXYHdcQ9xv/8u5J4w6XddZKTMyIehFHYFtcwxr8me7QlGqs+5y2h7IxM38Mox592xK34xez8A8tuhgcbRfXSjhJGNwru++HmkR6OMXUZ9rkmA7EN19tlJUUnZTjIDwSDJrl/H2SiDkk7SYyEf8eOJiBEXuWwTNbqVs0eaecLF5fAfU3T15T4mIgjKdRSBiCUk7UsqImb4IDhc5x/DmsKdU9cMndQ3Fp4dGjy9j+QFr+fxR9Qsychvq0GuaTz8npY9G+f1FkF2Q3DVkzhoCDOmjEZ9rXFqouIckcbNyqWRAmrKGCA+dmRyF53ZJsG+SpnNRyzuaLAIh3G2NRSBXEyeJEezuyWPs8CZ3LOFg8ciUwkiHOtMv2ly7oHySvkGldBeNGzNjvmtIdtXIC9pHdfOOWphFd2nR0pohziIIKsvYYLH03UxWWew/1o49oW9IXCvJinnlUlGLYCi5hhhjqxljrzHGFjLG5tdjDAOZtCmenpBNlTVkPnc+KkZgsBbED65gc0+RiN45BZt7P2BnIfXoMe7uyWNsm9OgrK9QhM2D7RRU90QpATZ1pl+0ubbCNWQRJHQNld98zLcI5HHJgjfqGlakItDECDIZ97Xo2oCQRRCnCDLxMQLT1y1KicW9u75FEPNlqwSVDBbXyTlUz6yhUzjn2+p4/YFLylbRYrdEFkGCpSrzhiIswJw1JIwIWwoWi8wb2/atANuODxZ39xcxxrUIROBcriNoU2alpVjmqlCSYxsyYUWQzA0UN6Qk7gQxS5fHlTRGEPX56l7yrmXzyPOOaEkZLE5YR6BtOhd1DwMoRlDJUoUhZREQ0YgfRdLvhJjhJbGCk1QWR/XwN/lcvRYTXHINucHOotQP32kxET9Q4RrqdoO5wiLQuRhKq+wN+sJtrr83Nffd0rzXpbiGGHzFbXo3RFxEjtnIfvYowRrVfTMqWJwvRHeHVRdvj68sdh/wqLUJ/CJDeY8oSy9u5pypYYygkrP4oaYIOIAHGWMvMcYurtMYKsp3/rEER1/zcEXOJb67iRfelvzvgr+8uBbTLr8Xp/7oMVz6p5elc8f/ME6/7gl09RXw6d+FvXZn/PgJ7THiB1ewuZd1JGbxDy/ZjOseegMA8H9Pr8brmzq155ARFsEPHnAK03RLBAqSpM3qkGf7XX0FXP731yL3AVQftvifzP0iYzHmtWUe1xbu0y8rKjmLS5b9UUJYV4Er4iKi5bRMzlM6dqLuqd514prOuQOYOqbVqLhs7r8HquvJNM59xphbXwPApJHOQkGtTeG4T6WpbGXx0HINncA538AYmwDgIcbY65zzgIRxFcTFALDPPvvUY4yp+M3Tb1bsXN7sKLFrKLzjD+Y5AnTV1j1YtXUPfvqRIwH4Ajvu3Ot39uChJZtjxukXiokYQXdfwYsxiBjBzU+mf2/UDB5VIP/p4mM94bFxdw9U/v3t++GtHd14MOIeshkG5MPb/+8TR0vXDb5R4r1mzMk86i/YJVcWnzdrMvqLNs4/cm8A4Z47IrNJttBkBZMkfVTe44QDx+G8WZNx1mGTQvuL97e/aMe6rW786Bx89vcLAMS3oc5mLNz8sbmYNWUkfvHYysBr//72/fDrp5zvxiWnHIgpo1tx3qzJuPRPC43nG9maw40fnYMjpozC8dc+atzv8rMOwWF7j8Q7po+PHN+jX34HNkmLIpVCJV1DteiNpL1uPS7KOd/g/t8C4E4Ax2j2uYlzPpdzPnf8+OgPc7DhuYaSKgJ3P1mQmNw7vgkeZ1rHX1e2usV5O3oLfrBYyhpKgvwjaFEyeNTCrmP3H+stkqITxEfuMxqzpo6KvJ6pRcOcfUf711UEnfhsOPcXpyml1xCYEyf40NypxnHoXEMyUQJbZ5G0ZDN435wp2lmycL31R8SHBPIyl0lu/YxDJ2JCe0vofXr3LGe1M845mrIWPjh3aqLA/5mHTdJaUTItuQw+cNSU2PPtP74NxxuW7UxKZV1D9dEENVcEjLHhjLER4jGAdwJYVOtxDGTSriLm9/HxBYYp4OtnJEWfU722fsEZf5uwNDp6/FW/RB1B0gw+Wei25IJfzah2CrpXMhaLdCdFnTPYz8d83Zy0mEtakvjuhU/dVOAX5cLRpU8258zvR1PMtWTksacRXOp45f5TaanXzFlLRV1D9aEerqGJAO50v0BZAH/knD9Qh3EMWMQMMHWwWD5HTLvoONQgW5FzWMqIijaHmLh7iqA3741fCPMkwWHAEXzC3Z/NWMhazKtbiGqwphNeGYvFLpxucmvIM1dV0AVcQ6LVcQlSKe4I4XoCzJkv0emjzn/5rW+OWNFMjhHEUZIFpDmunBTbes2cdVS211DFTpWKmisCzvkqALNqfd1GQrh1UqePyq4hww/abzoXfXJVIDhtIoICSbYIxOOOHtk15FYWJ1UE8kzcFeRCEUQJH12RW9ZikQu3AMksgiTHx2XOlIpwPZmEc5LKYvkzUjN+AteKcUPJlCr4whaByMBKbxIMJIugkkMZMq4hIh7xY0yeNSQsAlkw6/dNUnQmj0FQtHnIfyyfSgicnnzRS/kUfv6kVogsuDMWC1gBvYaVxpyxhgWlZTHP3WHCZGUknfGK40tJUIw7hrH4WXqS9FHZsou0CFLECEq1CFTFLAyyUt6/gWQReEVuA2dIqSFFMAARP/zkriHnf5o6gjhUi6LIubeMo7ctsGC8v32Hu+qXJ3gSxwj8O5YXgQH8wjLdPcpKy+saarFEGS06kgoZ4XqKaslhIu6zYvBdW6a2HklcQ/Kh0TECP2sojlJn46G1naWGdI1M0jUSBjK0HoGG51Ztx6wpo2qSg6xD+ISfXbUdPf1FMAa8/NYuHHfA2NC+fYUinl+1HUC0ib1+Vw/29HOauGMAAByESURBVBUSu2lUi8C2eWjZxyeXb0UuY6G9JQebc7Q1Z9HVV8CdL68H4AuepFlDuYBFYAUUTU+ERaCeI18sJooRRAWCE13LSu5OUUniDomzCKJcWEI42QGLwPx9bsomDxaXOhtXx6uLbTUijWwJCMgiUFi7oxsfuek5XPH3V+s2BvFjfGNzF7551yJ8665FuODm5/Dmtj2hfb/9jyVY4K4+FSVbTrj2Ubzzx0+kcA2FYwR9+eC2L/zxZXzmtpdwwc3PoWhzjBomln90LIK0rhNZcKtC45j9xiQ6h7hmsqyh5F9/kY++92inUOlDc6fiomOd+pa9NAVaKqfPmBB4PqIlunDqoIltOMhNjxV1BirRdQThGEGUYjxkr3YAwLlHTI4cVzmEg8XOf1UpvmvmxKqNoRqkafo4UCGLQEEsD7l0Y3z1qw65yKpU5BnmG5s7Pd9/Z2+4+mnJBn8VrCQyPu8GouNmpGodQtH2XUNv228Mnn9zR+D1rr4Cpo0dhnU7/eKutGX+ah+d5qyFTgDfO/9wHLZ3e6JziHz4TIJgcdKgsFidDXAqnsVzzjk+dty0ROe4+WNzsd8V9wEAXvj6aZHW5u/+3zGYMtqpnH3z+2cbv09R2Uo611BUUHvyqNbAtVZfew6+848lFS2UNHVyVb8d8qIvjYBwgdag0WnVIItAwdIEXtNQid4mshC2LOZ9wXQ+Yfl6SVwwwu0UN8xQsJj7riFd3/6izb2OoYK0QcVcYPUt5rky1JqC6HP4mTxxrp+miOBpEtIofHnfJO0nkuybpMWE/J2I+zyqHYDNGBr4NbIABfx6mUaGFIEC08yk0lCJHldqb5mocxYCAdv4i4tzxyk6rWvItQhMs1nRH0iQNq0yZBG4CkD2bceNOyu5huJcP2l66lSSuMsmfduSdB+VFUG9M23CMQLnf6mTroFCVDZWo9D4d1Bh/Jz80r6cSQOjUciz8Qxj3lh0v2O533rUWsPqueNWClOLmGwbXoxAt4AL4HcMFaQttArGCCxPAaT5oQlh4yz+HlNQliJGUEniF6VJdp4kdQTy5KDM2HjZmArKGt4iIEUw+BC/m1K/m5VwDQUsghhhFrQI4jM+PIuAR49VTSOUXUPDDBaB2jkyrUWgrs0rfmBRaY8qQpkw5ufGm4gLJleLWEWQMBExyfsb1y67liSNETQaUdlYjQIpAoWC7QvKUqiERSDPxh3XkDuL18h5WZgnsQi6+51gOOc8MpVUtQiKth3rGlIFa1LBIwSEvHs2Iy3VmEKAZdw4Q9HmgZiD9rp1miKzmF9d0ttNslbzQHINqZ+jN5wG1wRpJioDlca/gwojBGvJrqEKrIwnWwQZKVis6ygqC+wk1khXn1uYheixhrOG/IXeTa4hVbAmFeBiP7XFsgjCJal29cYgrUyVtNdQrU372IXtU75vUe9zwDVUd4vAECxucE1ArqFBiJhVl+rhiZplv7ZuNz5z2/zYStRgsJh5szrxo77vtY048ydP4Ft3LUptEXS5KahxFsH1jywPPH/+ze1en3iTa0gNzqZt1RCwCCzmLYtYKPqrZsUFgMXYipzH7iuKqGqvCKJfTyquxXti+jyA4Pc4bXDc66VUIQViqixu/BhB47uGqI5AwbMISpylRLmGvvinl/Hmtj14a0c39h/fZtxPXnHLYswbiRD0n/+DsyjI65s6A33ZxcLyUdbMHtcisLl5IXoA2Oa2iRBcefdi73Frk/5rk8sw/OyCI/HEG1tx7P5jI2e+s6aOwitrnUI4368ftAi+fd5MTBzRgnccPB4ZxvC5kw/ARW8LL1J0+6ePxV0vr8eBE9rw7lmT8NunV2P2lFHY1OEvOPLVMw8OLVIiXFktuQw6egvGsVaa+GBx9Ou/+rej0Few0ZLL4EunT8dZh++Fl9bsxL5jhuHCXz8f2Fd8H/cZMwznzU5XLPaFUw9EX8HGBcdUZmGoOfuOwruPmISmjIW3HzRuUBRiAYPDNUSKQKHgCdPSjo+q3NWl9OmQhVLGYt4vRef66Zf6/whFEeUiEgVzskUgt3tuzWVi2zkYXUOWhXNnTca57oIjO/b0a/cDgBl7jcCU0a2499WNfqaPJACzloVxbc248txDvW1fO/MQ7bmOO2BsoP3GFWfPABAMXn/+5ANDxwnXk7oITrWJXwsi+vV3zdzLe3zp6QcBAKZPHKHdV3zXvn3ezNQz1xEtOVx93sxUx0QxYUQLfn7hHO95h2SdNjLkGhqE+DGCEo+PONAv8ok+R0ePX0EsB4t1rh9ZaIusoajeN74i8O9VuGCAZAFUo2som9w1xCWLRLhwZAFYCXfE8JheUV5W0kCLEVQwqCu+MgOxH85gsQjqlX1WSRr/DipMIWELBhNRQl63kpiOjl5ZEfiuIV16qCz0haLIR0SB9whFII1D7nvDEO9LNgnOnOoDjjiPLVskGZE1JFsE5UuuOBdLKVlJlSBeEVTuWsJCraRyqRSDJUZQ72ysSkCKQMGPEZRGlGvI7wgZfbyYtQu8rKGYLpdi7FH7+cFw7u3f3hr0EMZZBaasFrVAKyrP3eb+e6Vb6asWwrleQb74YHElLQLnPa53xpCOQSA/Bw2kCBRkQVkKUf553apRKp19BWUReh7KGjLhWQQJWgkHXEPNwUKwuGwbk4BXe/tE5blzcG+8pjqCaiOCfLWe0cVdr5LDEZ/xQBS6A9FKGaqQIlAoN0YQJeSFfI1K85TjA4Aj1P06AueByTVTTKEIZItAjhEA8W4Z0+xSVSCRC8Nw/73KeumjtbYIXEVQ9Sulo5LyUXx3qrWcZjkMRCtlqEKKQKEguYZsm2Pxht3e6lg6uvoKnt8dcHrxywVQWzqdFMZ80cZuV8irdQT5oo0de/rBOcfKrV3B8Uhunp3d/Vi4dldogRh57L35Il5Zuzv2Pjn8wLbaGz8umG2SKaoiiPqdc/iKK5fRWARJymbLZKDmf1c2WOzGCAag0B2AQxqykCJQKErB4n++thHnXP8UvnnXIuP+h101D4dfPc97/sEbn8Wlf3oZAPDSmh045ppHcPfC9bjszwuxdofTq1/t4/ONO1/DnP9+CL9/bg0+8X8vBl7LF23vx3zl3Yvx3hueNo4lX7DxvfuW4pI/Loi9T86556Pfb9yw0GtRqK4kgerOiXKB2DzsGpLn5pWaLe49qtX4mmcRSJeqtHBqk1p2vy3h4jqVVARzpznXnDCiOWbP2iO+H2p9RxomJVgUqFaU87FNrvN9UB2BglxHsHGXI7jFrN6EOoO+f9EmAMBid9GY51btwL2vbgxdQ/BP97W7F27wtv32k0fj2vtfR97mid1UnX35wMIwUXDuWwQHjG/Dbz95tKeE4vr0H7Z3O/762ePQ3pJDe2sWx33/UQDQ9va59f8dg4//5gUAwOdOPgC/fGwlgGCwWAg+2aCoVIvoBy47EZ2GYjFdIdCCb51RkesCwPNfPy1Qo/DbTx6DbV19scdV0ovzX++cjg/OnYJ9xw6v3EkBvPiN0yvyGT351VMwvkQl9dwVp2FY88Cw6l74xmmxva2ieOBLJ6G7L9lyrNWAFIGCnDUk0jgr7ctU+/g0Zy109xexVRIS40c0Y8roVmzc3autcp46ptWzMAQdPQW0tyT7MtmcewrJshgOkgqS4lwmjDEcPS08u81lw+/TgRP8CurRw3xLQi5os7xgsZQ+WqFg8YiWnHFZSHGfsuAdNaxJu28pTGwPzvJamzKYOmaYYW+fSn7dshkLB0RUsZdKqcJbJcn7YSLJEqG1YsKI8sbS3pJDe8zypdWEXEMKctZQR48zk0zT9EzGNJPvLwRfEAJpa2dfYJuzELut9dkP17R56OjNh1JPjWODlFrIWCCYmLbASggunV9fPpecFinHCMS1Wc1jBCJYPNCc1QNtPMRghxSBgpw1JCyC3piWCybEcaqpr1oEwhUj9xhqzlrIZiwUinrXUJtmuciOnrwXkI5DTh/NWCyQ6pl2CUdhMemWhgwoAvll5frO6/4OtQgkCgVciTUkKgkFUYlaQ4pAIWgROELVlKUTh1AkapaQGiPQzcCbcxZyGYa8bQeCt6LPT1tLWBHYHNi0OzqeEdzfd80ELIKUvXeiOoPKbiZZ0MvpqzrXUC1y+4UrS9feu55Qfj1Ra0gRKHhtHLjf/K1kReC6ltRgpZo1pGt61pzNIGdZyBeCEQLhm9UtIJ92rKI9RdZigThIeteQc6zOry9bCfKrsiIQu9R6JizcT3EV27WG9ABRa0gRKAjhyAHJIijNNSQsArl3EJDMImjJWchmGAqKRSAUQXMZja5EkywvWMxYwE+eVhF4riGNX58FZvn+djlrycsaqrEEFFkvUb2Z6sHAi1kQg50hoQg27e7Fr59cFZkf35sv4icPv+Et5Whz7glwsWi74PfPrcGqrV2RC8y88OYOT5E8vWJ74DXhirBtjh888Drmr9kZOr4pYyGXsbCtqx87u31FMt5dfyCuDUQUIgbw3XuXAAhnRaUttBLyO06Oyy4POVhsaWIEtUBYMGQREEOdIaEIPvv7l/Dde5di9fZu4z4vvLkDP3l4OZ5ftQNAsBJYdrcUbY5v3rUI7/n50yEXj8yfX1xrXOxEZCGt29mDX7h59SqMMcydNjq0XVgElgUcM20MfnGR3999+sQ2TBurT8eTZ/lCEby+qRMAMG3sMLS1ZHHopHb86EOz8Z9nTA/s35rLYMpopzDrf95/eOjcP/nwbBy2d7sxTfO0Qybgv98zE2cd7vfR55zju+89DAdPHIGvnXkwJo9swXuPTLdwSrlMbG/BjEnt+N77DscZh07EVdLaB43KJaccgI8cPbXewyAajCFRRyAEelR2iJj973Ezd/JF7rmJZNeQeNzZVwhZCjKdvflQ3yCBCEirLiOVdx8xGV/566uBNQfGtjV59/KXzx4X2P/BL70DADDt8nsD2+/43HE4at8x3nbZb/+Ro6digpvvft+lJ3rbl333rMixyZw2YyJOmzHR+Potnzjae/zrj83Fv/9uPjgHTjxoPOZ9yakqfeaK07BiS5fpFFUhl7Fwv3vPpxw8oabXrhZfeZd+8R6CiGJIWAQ+Uc3enNm7cA0JRrbm0Ffw/fSy8DcFZke25tDRmzcK+rx7nElRyKhugtFuwVOajEc1C0U2ZORVvGpB1Cpt5BIhiPowpBSBWsglI6yGPUqZ9/gRzeAcknUgKwJ9EHn8iGZ09BSMOf15V4onzfmXEauDRa17oKLGAPokC6Ndk4ZaTYSw142e0iYJoj4MKUUQlf0jZu+qRSCCs+LYoJtIbxGMb2vGtq4+9BpcRyLIHOca0iGEetSSmCqqgO2V7qHmFgHMq1KRGiCI+lAXRcAYO5MxtowxtoIxdnmtrhuVYy/cNN39YYtAPjZgERgE/fgRzdjitosYOzzcu0asFyDcUWlIuu6xjJrfLy9vWev+JlGuIXFvrTVeTJ4ghjo1VwSMsQyAGwCcBeBQABcwxmqSrhGpCAwZPiFFkE/mGtI9FghBXIpFIFItU7mGIlwupoXoq0WU+0e8pKuaJgiietTDIjgGwArO+SrOeT+APwF4TzUuVCjagdW6+iJ6BsmBW7m9rhDkom9QItdQjCLoK9joL9iJYgTqxDnjWQQpXEMDqHlNlEUg3lt1xTSCIKpLPX5xewNYKz1fB+Bt1bjQVfcsxh+ef8t7Pn/NTlx820t44LITceeC9fjVE6u81w4Y7/drb2/NYceefgD+when/ejx0Pkv+vXz2uuOa/OFv2hPazHfnXP7C2/h9heccY0elgsUjKlM32sEXlm7y3suWjlHteCdPLIFG6SeQ6pF0Jy1PCVW6xiBcEVNHmleMGbGXu21Gs6AYtrY4Vi3syd10z+CKJd6KALd9DQ0PWSMXQzgYgDYZ599SrrQ6YdODCiCvy9YDwB4ZOmWgBIAgJVb93iP3z9nb8yaOgo2B06fkT6//J0zJ+Ibe2agKWuhJWfhjgXrkM1YXiHZV951MO5ZuAHLNndi8qhW/OhDs/DVv72KbV39+Mw79sf750zxzvWbj8/FK+t2YWRrDmOGN2Pa2GH42QVH4oxD/bz9v3/+eIyUBPpdl5yA1zd14mPugjBqX6KffmS2555KumpWpZg1dRR+cdEcnHxweFWqAyeMwC8vmoOTylixqpG54cI5eGH1jtA6BgRRbeqhCNYBkEsfpwDYoO7EOb8JwE0AMHfu3JJ6AJxy8AS0t2Sl5nHJegaddfgkzNnHr+o9+/C9cN9rmyKPOXraaLy42mkV0d6Sw6dP2h8A8I9XnFvLWgz97r6XnHIgVm/bg2WbO9HeksOph0zEmOFN2NbVj7n7jsF0aZGYsW3NOPWQYLHWubOCFbjyWAFgQnuLVyQGAO2twY95bFuzdmGZWnH24ZOMr50V8dpgZ+SwXEDBE0StqIcN+iKAgxhj+zHGmgB8BMA91bqY7PpI2plTzaQZmcB9YlpIRbRqUHP5xbiEkBZplS2a5RPLRe0dVKllIAmCGBzU3CLgnBcYY18AMA9ABsBvOOeLq3U9R6i7i8YX/IXpI49RZtBJUixzBr+u6O2vCl9xTnWlsbQN30qhnIZ1BEEMPuqSnsE5vw/AfbW4lirUAadPUOQxiuDXrQamkjPMskVfn4xiMXjjUg5L2wK6FEgREAQhM+glgm42H1fIpS4UkyT90rTYunD5qBaBKX+/FhkjlVoYniCIwcHgVwQa//7unn7NnuWRdWfZ6rq9InNTjRGYFh+pRbsd3QIyBEEMXQa9RNAFehdv6Kj4dUTFrMmNFGcx1BKxVi9BEAQwBBTBgRPaQtvWGBaomTK61Ssgk3lHRF77Xm6a5vvm7A0A+PzJBwZe398tVLvwmGAtxFHuojPnznLSJT92/L4AgEnt5kKrUjhvVnixF1OGE0EQQxMWl0EzEJg7dy6fP39+ycd39RXQkrXQ1VdAe4uzVkDGYli7owdnX/8kAGDpd85Ea4K+O735IlZv34Mzf+Ict/racxKNYXtXH4767sOpjqk0YmGahVeegVHDws3wCIIYXDDGXuKcz43bb0g0dRHuGiH8xP/mXJ+3TxIlADiB5CRZRCoDaRZOWUMEQcgMaYlQaqpmKbn+mQGUqUNZQwRByAxpRaCmiSaluYTq34FUzUtZQwRByAxpiVC6RZD+ODV9tJ4MpLbUBEHUnyGuCEqzCJpK8LFHLQ5DEARRT4a0IlCLv5LCShDqNAsnCGKgMqQVQSkCnSAIYrAxJNJHo/j2eTNx1L6j43dU+O57D8PMyelW0rrq3EPxtv3Gpr5Wpbjz88djycbKV1UTBNHYDImCMoIgiKFI0oKyIe0aIgiCIEgREARBDHlIERAEQQxxSBEQBEEMcUgREARBDHFIERAEQQxxSBEQBEEMcUgREARBDHEaoqCMMbYVwJoSDx8HYFsFhzNQoPtqLOi+GovBcl/7cs7Na+26NIQiKAfG2PwklXWNBt1XY0H31VgM1vsyQa4hgiCIIQ4pAoIgiCHOUFAEN9V7AFWC7quxoPtqLAbrfWkZ9DECgiAIIpqhYBEQBEEQEQxqRcAYO5MxtowxtoIxdnm9x5MGxthvGGNbGGOLpG1jGGMPMcaWu/9Hu9sZY+x69z5fZYzNqd/IzTDGpjLG/sUYW8oYW8wYu9Td3uj31cIYe4Ex9op7X992t+/HGHveva8/M8aa3O3N7vMV7uvT6jn+OBhjGcbYy4yxf7rPG/6+GGOrGWOvMcYWMsbmu9sa+ntYDoNWETDGMgBuAHAWgEMBXMAYO7S+o0rFbwGcqWy7HMAjnPODADziPgecezzI/bsYwC9rNMa0FAB8mXM+A8CxAC5xP5NGv68+AKdyzmcBmA3gTMbYsQD+B8CP3fvaCeBT7v6fArCTc34ggB+7+w1kLgWwVHo+WO7rFM75bClNtNG/h6XDOR+UfwCOAzBPen4FgCvqPa6U9zANwCLp+TIAk9zHkwAscx//CsAFuv0G8h+AuwGcMZjuC8AwAAsAvA1OQVLW3e59HwHMA3Cc+zjr7sfqPXbD/UyBIxRPBfBPAGyQ3NdqAOOUbYPme5j2b9BaBAD2BrBWer7O3dbITOScbwQA9/8Ed3vD3avrNjgSwPMYBPfluk8WAtgC4CEAKwHs4pwX3F3ksXv35b6+G0D9FrOO5icAvgrAdp+PxeC4Lw7gQcbYS4yxi91tDf89LJXBvHg902wbrClSDXWvjLE2AHcAuIxz3sGYbvjOrpptA/K+OOdFALMZY6MA3Alghm43939D3Bdj7N0AtnDOX2KMnSw2a3ZtqPtyOYFzvoExNgHAQ4yx1yP2baT7KonBbBGsAzBVej4FwIY6jaVSbGaMTQIA9/8Wd3vD3CtjLAdHCfyBc/53d3PD35eAc74LwGNwYiCjGGNisiWP3bsv9/WRAHbUdqSJOAHAeYyx1QD+BMc99BM0/n2Bc77B/b8FjuI+BoPoe5iWwawIXgRwkJvh0ATgIwDuqfOYyuUeAB93H38cjo9dbP+Ym91wLIDdwsQdSDBn6n8LgKWc8+uklxr9vsa7lgAYY60ATocTXP0XgA+4u6n3Je73AwAe5a7zeSDBOb+Ccz6Fcz4Nzu/nUc75RWjw+2KMDWeMjRCPAbwTwCI0+PewLOodpKjmH4CzAbwBx1/7jXqPJ+XYbwewEUAezozkU3D8rY8AWO7+H+Puy+BkSK0E8BqAufUev+Ge3g7HpH4VwEL37+xBcF9HAHjZva9FAK50t+8P4AUAKwD8FUCzu73Ffb7CfX3/et9Dgns8GcA/B8N9ueN/xf1bLGRDo38Py/mjymKCIIghzmB2DREEQRAJIEVAEAQxxCFFQBAEMcQhRUAQBDHEIUVAEAQxxCFFQAxqGGNFt8Ok+IvsQssY+yxj7GMVuO5qxti4Eo57F2PsasbYaMbYfeWOgyCSMJhbTBAEAPRwzmcn3ZlzfmM1B5OAE+EUbJ0E4Ok6j4UYIpAiIIYkbtuEPwM4xd10Ied8BWPsagBdnPP/ZYx9EcBn4bTPXsI5/whjbAyA38ApSuoGcDHn/FXG2Fg4RYDj4RRTMelaHwXwRQBNcJrsfZ47vYnk8XwYTofc/QG8B8BEAB2Msbdxzs+rxntAEAJyDRGDnVbFNfRh6bUOzvkxAH4Op4eOyuUAjuScHwFHIQDAtwG87G77OoDfuduvAvAU5/xIOC0J9gEAxtgMAB+G0+RsNoAigIvUC3HO/wxgDpy244fDqVA+kpQAUQvIIiAGO1Guodul/z/WvP4qgD8wxu4CcJe77e0A3g8AnPNHGWNjGWMj4bhy3uduv5cxttPd/zQARwF40e2y2gq/mZnKQXDaGADAMM55Z4L7I4iyIUVADGW44bHgHDgC/jwA32KMzUR0S2LdORiAWznnV0QNxF0ucRyALGNsCYBJ7voG/8E5fzL6NgiiPMg1RAxlPiz9f1Z+gTFmAZjKOf8XnIVZRgFoA/AEXNeO26N/G+e8Q9l+FoDR7qkeAfABt++9WBd3X3Ug3Fku8V448YEfwGmENpuUAFELyCIgBjut7sxa8ADnXKSQNjPGnoczIbpAOS4D4Peu24fBWaN3lxtM/j/G2KtwgsWibfG3AdzOGFsA4HEAbwEA53wJY+ybcFbDsuB0k70EwBrNWOfACSp/HsB1mtcJoipQ91FiSOJmDc3lnG+r91gIot6Qa4ggCGKIQxYBQRDEEIcsAoIgiCEOKQKCIIghDikCgiCIIQ4pAoIgiCEOKQKCIIghDikCgiCIIc7/B6KzxOK0ZHwsAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from datetime import datetime\n",
    "agent = Agent(state_size=state_size, action_size=action_size, seed=0)\n",
    "start=datetime.now()\n",
    "scores = dqn()\n",
    "end=datetime.now()\n",
    "print(\"Duration: {}\".format(end-start))\n",
    "plot(scores)\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The neural network\n",
    "\n",
    "The neural network used is a [duelling network](https://arxiv.org/abs/1511.06581)\n",
    "\n",
    "The dueling network architecture represents two separate estimators: one for the state value function and one for the state-dependent action advantage function. According to the paper, the main benefit of this factoring is to generalize learning across actions without imposing any change to the underlying reinforcement learning algorithm. The results show that this architecture leads to better policy evaluation in the presence of many similar-valued actions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The network consists of two fully connected layers (512 and 128 neurons), followed by two heads called advantage (128 and 128 neurons) and value (128 and 1 neuron).\n",
    "The output is a q-value.\n",
    "\n",
    "\n",
    "\n",
    "from [dqn_agent.py](./dqn_agent.py)\n",
    "```python\n",
    "   # Q-Network\n",
    "   self.qnetwork_local = QNetwork(state_size, action_size, seed, 512, 128).to(device)\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "from [model.py](./model.py)\n",
    "\n",
    "```python\n",
    "def __init__():\n",
    "  self.feature =nn.Sequential(\n",
    "                nn.Linear(state_size, fc1_units),\n",
    "                nn.SELU(), \n",
    "                nn.Linear(fc1_units, fc2_units),\n",
    "                nn.SELU() \n",
    "  )\n",
    "  \n",
    "  self.advantage = nn.Sequential(\n",
    "            nn.Linear(fc2_units, 128),\n",
    "            nn.SELU(),\n",
    "            nn.Linear(128, action_size)\n",
    "  )\n",
    "        \n",
    "  self.value = nn.Sequential(\n",
    "            nn.Linear(fc2_units, 128),\n",
    "            nn.SELU(),\n",
    "            nn.Linear(128, 1)\n",
    "  )\n",
    "\n",
    "```\n",
    "\n",
    "This is how the two heads create one q-value from an advantage A(s,a) and a state value V(s).\n",
    "\n",
    "\n",
    "```python\n",
    "def forward(self, state):\n",
    "    x = self.feature(state)\n",
    "    advantage = self.advantage(x)\n",
    "    value     = self.value(x)\n",
    "    return value + advantage  - advantage.mean()\n",
    "```\n",
    "\n",
    "From the paper\n",
    "\n",
    "> The proposed network architecture, which we name the dueling architecture, <i>explicitly separates the representation of\n",
    "state values and (state-dependent) action advantages </i>. The\n",
    "dueling architecture consists of two streams that represent\n",
    "the value and advantage functions, while sharing a common convolutional feature learning module. The two streams\n",
    "are combined via a special aggregating layer to produce an\n",
    "estimate of the state-action value function Q \\[...\\]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Hyperparamters\n",
    "\n",
    "You can find a set of hyperparameter that can be used to tune aspects of the training algorithm in the file [dqn_agent](./dqn_agent.py)\n",
    "\n",
    "```python\n",
    "BUFFER_SIZE = int(1e5)  # replay buffer size\n",
    "BATCH_SIZE = 64         # minibatch size\n",
    "GAMMA = 0.99            # discount factor\n",
    "TAU = 1e-3              # for soft update of target parameters\n",
    "LR = 5e-4               # learning rate \n",
    "UPDATE_EVERY = 4        # how often to update the network\n",
    "```\n",
    "\n",
    "I found that tuning these hyperparameters did not improve performance. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Further improvements\n",
    "\n",
    "As you can see, I played around with different activation functions in the neural network, ending up with [SELU](https://arxiv.org/abs/1706.02515), which sometimes helped, solving the environment quicker.\n",
    "Also using Huber loss improved performance on solving the environment.\n",
    "\n",
    "As next steps I intend to implement [Prioritized experience replay](https://arxiv.org/abs/1511.05952).\n",
    "\n",
    "Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. Experience transitions are uniformly sampled from a replay memory.\n",
    "\n",
    "However, this approach simply replays transitions at the same frequency that they were originally experienced, regardless of their significance. With prioritizing experience, important transitions are replayed more frequently, and therefore the agent can learn more efficiently.\n",
    "\n",
    "I'd also be interested in how [Noisy nets](https://arxiv.org/abs/1706.10295) could improve performance, as the authors claim that <i>the induced stochasticity of the agent's policy can be used to aid efficient exploration</i>. This can be of help, because the trained agent gets stuck a lot of times within 100 episodes or less.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Misc\n",
    "\n",
    "You can run a non interactive version of the training from the commandline by running \n",
    "\n",
    "```python\n",
    "python main.py\n",
    "```\n",
    "\n",
    "from the root of this repo (after [activating the environment](./readme.md)...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "drlnd-pytorch",
   "language": "python",
   "name": "pytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}