Deep_Reinforcement_Learning_Part_11_.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Deep_Reinforcement_Learning_Part_11_.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9ibtEV6SY3Pl"
      },
      "source": [
        "This is the Part-11 of the Deep Reinforcement Learning Notebook series. In this Notebook I have introduced Noisy Nets.\n",
        "\n",
        "\n",
        "\n",
        "The Notebook series is about Deep RL algorithms so it excludes all other techniques that can be used to learn functions in reinforcement learning and also the Notebook Series is not exhaustive i.e. it contains the most widely used Deep RL algorithms only.."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "04bvh0woY_Tf"
      },
      "source": [
        "##What are Noisy Nets?\n",
        "\n",
        "One of the major problems in reinforcement learning is the exploration. There are various exploration strategies  like Epsilon-greedy , Boltzmann exploration. What these strategies do is they add noise to actions. But the exploration can also be improved more by adding noise to weights of the network . Noisy Nets is concept of a deep reinforcement learning agent with parametric noise added to its weights, and show that the induced stochasticity of the agent's policy can be used to aid efficient exploration."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kLO6PrSaaRt3"
      },
      "source": [
        "The key insight is that a single change to the weight vector can induce a consistent, and potentially very complex, state-dependent change in policy over multiple time steps – unlike dithering approaches where decorrelated (and, in the case of ε-greedy, state-independent) noise is added to the policy at every step"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VTq8VgfPavxu"
      },
      "source": [
        "For Understanding the details of intuition you can read the paper Noisy Networks for Exploration (https://arxiv.org/abs/1706.10295)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5K2vgZ3Qa7fZ"
      },
      "source": [
        "Here I have implemented the noisy nets with Double DQN with Prioritized Experience Replay(PER) and target networks \n",
        "\n",
        "(Refer https://github.com/Rahul-Choudhary-3614/Deep-Reinforcement-Learning-Notebooks/blob/master/Deep_Reinforcement_Learning_Part_5_.ipynb for better understanding of Double DQN with Prioritized Experience Replay(PER) and target networks)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vrzTuzLUbxRa"
      },
      "source": [
        "##The Algorithm Implementation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wkEZs-jocDQG"
      },
      "source": [
        "Below code setups the environment required to run and record the game and also loads the required library."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rXotgfUqb-A2"
      },
      "source": [
        "!apt-get update > /dev/null 2>&1\n",
        "!apt-get install cmake > /dev/null 2>&1\n",
        "!pip install --upgrade setuptools 2>&1\n",
        "!pip install ez_setup > /dev/null 2>&1\n",
        "!pip install gym[atari] > /dev/null 2>&1\n",
        "!pip install gym pyvirtualdisplay > /dev/null 2>&1\n",
        "!apt-get install -y xvfb python-opengl ffmpeg > /dev/null 2>&1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-X_UED1Cb96T"
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras.layers import Conv2D,Flatten,Input\n",
        "from tensorflow.keras.models import Model\n",
        "import gym\n",
        "import numpy as np\n",
        "import pickle\n",
        "import random\n",
        "from collections import deque\n",
        "import cv2\n",
        "from skimage.transform import resize\n",
        "from gym import logger as gymlogger\n",
        "from gym.wrappers import Monitor\n",
        "gymlogger.set_level(40) #error only\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "import math\n",
        "import glob\n",
        "import io\n",
        "import base64\n",
        "from IPython.display import HTML\n",
        "from IPython.display import clear_output\n",
        "from IPython import display as ipythondisplay"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Lz2PQ-Ab9ue"
      },
      "source": [
        "from pyvirtualdisplay import Display\n",
        "display = Display(visible=0, size=(1400, 900))\n",
        "display.start()\n",
        "\n",
        "\"\"\"\n",
        "Utility functions to enable video recording of gym environment and displaying it\n",
        "To enable video, just do \"env = wrap_env(env)\"\"\n",
        "\"\"\"\n",
        "\n",
        "def show_video():\n",
        "  mp4list = glob.glob('video/*.mp4')\n",
        "  if len(mp4list) > 0:\n",
        "    mp4 = mp4list[0]\n",
        "    video = io.open(mp4, 'r+b').read()\n",
        "    encoded = base64.b64encode(video)\n",
        "    ipythondisplay.display(HTML(data='''<video alt=\"test\" autoplay \n",
        "                loop controls style=\"height: 400px;\">\n",
        "                <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n",
        "             </video>'''.format(encoded.decode('ascii'))))\n",
        "  else: \n",
        "    print(\"Could not find video\")\n",
        "    \n",
        "\n",
        "def wrap_env(env):\n",
        "  env = Monitor(env, './video', force=True)\n",
        "  return env"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wByZB83ucth6"
      },
      "source": [
        "This part ensures the reproducibility of the code below by using a random seed and setups the environment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6A0HbJsub9po"
      },
      "source": [
        "RANDOM_SEED=1\n",
        "\n",
        "# random seed (reproduciblity)\n",
        "np.random.seed(RANDOM_SEED)\n",
        "tf.random.set_seed(RANDOM_SEED)\n",
        "\n",
        "# set the env\n",
        "env = (gym.make(\"UpNDown-v0\")) # env to import\n",
        "env.seed(RANDOM_SEED)\n",
        "env.reset(); # reset to env "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xtok-xboc2aq"
      },
      "source": [
        "Next we construct a SumTree and Memory object that will contain our sumtree and data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UQeSF5cmc2xu"
      },
      "source": [
        "class SumTree(object):\n",
        "\n",
        "  data_pointer = 0\n",
        "  \n",
        "  \"\"\"\n",
        "  Here we initialize the tree with all nodes = 0, and initialize the data with all values = 0\n",
        "  \"\"\"\n",
        "  def __init__(self, capacity):\n",
        "      self.capacity = capacity # Number of leaf nodes (final nodes) that contains experiences\n",
        "      \n",
        "      # Generate the tree with all nodes values = 0\n",
        "      # Remember we are in a binary node (each node has max 2 children) so 2x size of leaf (capacity) - 1 (root node)\n",
        "      # Parent nodes = capacity - 1\n",
        "      # Leaf nodes = capacity\n",
        "      self.tree = np.zeros(2 * capacity - 1)\n",
        "      \n",
        "      \"\"\" tree:\n",
        "          0\n",
        "          / \\\n",
        "        0   0\n",
        "        / \\ / \\\n",
        "      0  0 0  0  [Size: capacity] it's at this line that there is the priorities score (aka pi)\n",
        "      \"\"\"\n",
        "      \n",
        "      # Contains the experiences (so the size of data is capacity)\n",
        "      self.data = np.zeros(capacity, dtype=object)\n",
        "  \n",
        "  \n",
        "  \"\"\"\n",
        "  Here we add our priority score in the sumtree leaf and add the experience in data\n",
        "  \"\"\"\n",
        "  def add(self, priority, data):\n",
        "      # Look at what index we want to put the experience\n",
        "      tree_index = self.data_pointer + self.capacity - 1\n",
        "      \n",
        "      \"\"\" tree:\n",
        "          0\n",
        "          / \\\n",
        "        0   0\n",
        "        / \\ / \\\n",
        "tree_index  0 0  0  We fill the leaves from left to right\n",
        "      \"\"\"\n",
        "      \n",
        "      # Update data frame\n",
        "      self.data[self.data_pointer] = data\n",
        "      \n",
        "      # Update the leaf\n",
        "      self.update (tree_index, priority)\n",
        "      \n",
        "      # Add 1 to data_pointer\n",
        "      self.data_pointer += 1\n",
        "      \n",
        "      if self.data_pointer >= self.capacity:  # If we're above the capacity, you go back to first index (we overwrite)\n",
        "          self.data_pointer = 0\n",
        "          \n",
        "  \n",
        "  \"\"\"\n",
        "  Update the leaf priority score and propagate the change through tree\n",
        "  \"\"\"\n",
        "  def update(self, tree_index, priority):\n",
        "      # Change = new priority score - former priority score\n",
        "      change = priority - self.tree[tree_index]\n",
        "      self.tree[tree_index] = priority\n",
        "      \n",
        "      # then propagate the change through tree\n",
        "      while tree_index != 0:    # this method is faster than the recursive loop in the reference code\n",
        "          \n",
        "          \"\"\"\n",
        "          Here we want to access the line above\n",
        "          THE NUMBERS IN THIS TREE ARE THE INDEXES NOT THE PRIORITY VALUES\n",
        "          \n",
        "              0\n",
        "              / \\\n",
        "            1   2\n",
        "            / \\ / \\\n",
        "          3  4 5  [6] \n",
        "          \n",
        "          If we are in leaf at index 6, we updated the priority score\n",
        "          We need then to update index 2 node\n",
        "          So tree_index = (tree_index - 1) // 2\n",
        "          tree_index = (6-1)//2\n",
        "          tree_index = 2 (because // round the result)\n",
        "          \"\"\"\n",
        "          tree_index = (tree_index - 1) // 2\n",
        "          self.tree[tree_index] += change\n",
        "  \n",
        "  \n",
        "  \"\"\"\n",
        "  Here we get the leaf_index, priority value of that leaf and experience associated with that index\n",
        "  \"\"\"\n",
        "  def get_leaf(self, v):\n",
        "      \"\"\"\n",
        "      Tree structure and array storage:\n",
        "      Tree index:\n",
        "            0         -> storing priority sum\n",
        "          / \\\n",
        "        1     2\n",
        "        / \\   / \\\n",
        "      3   4 5   6    -> storing priority for experiences\n",
        "      Array type for storing:\n",
        "      [0,1,2,3,4,5,6]\n",
        "      \"\"\"\n",
        "      parent_index = 0\n",
        "      \n",
        "      while True: # the while loop is faster than the method in the reference code\n",
        "          left_child_index = 2 * parent_index + 1\n",
        "          right_child_index = left_child_index + 1\n",
        "          \n",
        "          # If we reach bottom, end the search\n",
        "          if left_child_index >= len(self.tree):\n",
        "              leaf_index = parent_index\n",
        "              break\n",
        "          \n",
        "          else: # downward search, always search for a higher priority node\n",
        "              \n",
        "              if v <= self.tree[left_child_index]:\n",
        "                  parent_index = left_child_index\n",
        "                  \n",
        "              else:\n",
        "                  v -= self.tree[left_child_index]\n",
        "                  parent_index = right_child_index\n",
        "          \n",
        "      data_index = leaf_index - self.capacity + 1\n",
        "\n",
        "      return leaf_index, self.tree[leaf_index], self.data[data_index]\n",
        "    \n",
        "  @property\n",
        "  def total_priority(self):\n",
        "    return self.tree[0] # Returns the root node"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tfnYAL51c679"
      },
      "source": [
        "class Memory(object):  # stored as ( s, a, r, s_ ) in SumTree\n",
        "\n",
        "    PER_e = 0.01  # Hyperparameter that we use to avoid some experiences to have 0 probability of being taken\n",
        "    PER_a = 0.6  # Hyperparameter that we use to make a tradeoff between taking only exp with high priority and sampling randomly\n",
        "    PER_b = 0.4  # importance-sampling, from initial value increasing to 1\n",
        "    \n",
        "    PER_b_increment_per_sampling = 0.001\n",
        "    \n",
        "    absolute_error_upper = 1.  # clipped abs error\n",
        "\n",
        "    def __init__(self, capacity):\n",
        "        # Making the tree \n",
        "        \"\"\"\n",
        "        Remember that our tree is composed of a sum tree that contains the priority scores at his leaf\n",
        "        And also a data array\n",
        "        We don't use deque because it means that at each timestep our experiences change index by one.\n",
        "        We prefer to use a simple array and to overwrite when the memory is full.\n",
        "        \"\"\"\n",
        "        self.tree = SumTree(capacity)\n",
        "        \n",
        "    \"\"\"\n",
        "    Store a new experience in our tree\n",
        "    Each new experience have a score of max_prority (it will be then improved when we use this exp to train our DDQN)\n",
        "    \"\"\"\n",
        "    def store(self, experience):\n",
        "        # Find the max priority\n",
        "        max_priority = np.max(self.tree.tree[-self.tree.capacity:])\n",
        "        \n",
        "        # If the max priority = 0 we can't put priority = 0 since this exp will never have a chance to be selected\n",
        "        # So we use a minimum priority\n",
        "        if max_priority == 0:\n",
        "            max_priority = self.absolute_error_upper\n",
        "        \n",
        "        self.tree.add(max_priority, experience)   # set the max p for new p\n",
        "\n",
        "        \n",
        "    \"\"\"\n",
        "    - First, to sample a minibatch of k size, the range [0, priority_total] is / into k ranges.\n",
        "    - Then a value is uniformly sampled from each range\n",
        "    - We search in the sumtree, the experience where priority score correspond to sample values are retrieved from.\n",
        "    - Then, we calculate IS weights for each minibatch element\n",
        "    \"\"\"\n",
        "    def sample(self, n):\n",
        "        # Create a sample array that will contains the minibatch\n",
        "        memory_b = []\n",
        "        b_idx, b_ISWeights = np.empty((n,), dtype=np.int32), np.empty((n, 1), dtype=np.float32)\n",
        "      \n",
        "        # Calculate the priority segment\n",
        "        # Here, as explained in the paper, we divide the Range[0, ptotal] into n ranges\n",
        "        priority_segment = self.tree.total_priority / n       # priority segment\n",
        "\n",
        "        # Here we increasing the PER_b each time we sample a new minibatch\n",
        "        self.PER_b = np.min([1., self.PER_b + self.PER_b_increment_per_sampling])  # max = 1\n",
        "        \n",
        "        for i in range(n):\n",
        "            \"\"\"\n",
        "            A value is uniformly sample from each range\n",
        "            \"\"\"\n",
        "            a, b = priority_segment * i, priority_segment * (i + 1)\n",
        "            value = np.random.uniform(a, b)\n",
        "            \"\"\"\n",
        "            Experience that correspond to each value is retrieved\n",
        "            \"\"\"\n",
        "            index, priority, data = self.tree.get_leaf(value)\n",
        "            #P(j)\n",
        "            sampling_probabilities = priority / self.tree.total_priority\n",
        "            #  IS = (1/N * 1/P(i))**b /max wi == (N*P(i))**-b  /max wi\n",
        "            b_ISWeights[i] = np.power(n * sampling_probabilities, -self.PER_b)\n",
        "                                   \n",
        "            b_idx[i]= index\n",
        "            \n",
        "            experience = [data]\n",
        "            \n",
        "            memory_b.append(experience)\n",
        "\n",
        "        \n",
        "        b_ISWeights=b_ISWeights/tf.math.reduce_max(b_ISWeights)\n",
        "        \n",
        "        return b_idx,memory_b, b_ISWeights\n",
        "    \n",
        "    \"\"\"\n",
        "    Update the priorities on the tree\n",
        "    \"\"\"\n",
        "    def batch_update(self, tree_idx, abs_errors):\n",
        "        abs_errors += self.PER_e  # convert to abs and avoid 0\n",
        "        clipped_errors = np.minimum(abs_errors, self.absolute_error_upper)\n",
        "        ps = np.power(clipped_errors, self.PER_a)\n",
        "\n",
        "        for ti, p in zip(tree_idx, ps):\n",
        "            self.tree.update(ti, p)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FTOXhHN9c_rA"
      },
      "source": [
        "Next we implemented Noisy Dense Layer . This is the main implementation of this notebook. This is the layer that improves exploration "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zV2rS8fhc__h"
      },
      "source": [
        "class noisy_dense(tf.keras.Model):\n",
        "  def __init__(self,units=32,activation_fn=None):\n",
        "    super(noisy_dense,self).__init__()\n",
        "    self.units = units\n",
        "    self.activation_fn = activation_fn\n",
        "\n",
        "  def f(self,x):\n",
        "    return tf.multiply(tf.sign(x), tf.pow(tf.abs(x), 0.5))\n",
        "  \n",
        "  def build(self, input_shape):\n",
        "    # Initializer of \\mu and \\sigma\n",
        "    input_shape = tensor_shape.TensorShape(input_shape)\n",
        "    last_dim = tensor_shape.dimension_value(input_shape[-1])\n",
        "\n",
        "    self.w_mu = self.add_weight(\n",
        "        shape=(last_dim, self.units),\n",
        "        initializer=\"random_normal\",\n",
        "        trainable=True,\n",
        "        )\n",
        "    self.w_sigma = self.add_weight(\n",
        "        shape=(last_dim, self.units), initializer=\"random_normal\", trainable=True\n",
        "        )\n",
        "    \n",
        "    self.b_mu = self.add_weight(\n",
        "        shape=(self.units,),\n",
        "        initializer=\"random_normal\",\n",
        "        trainable=True,\n",
        "        )\n",
        "    self.b_sigma = self.add_weight(\n",
        "        shape=(self.units,), initializer=\"random_normal\", trainable=True\n",
        "        )\n",
        "\n",
        "  def call(self,inputs):\n",
        "    p = tf.random.normal((tf.shape(inputs)[1],1))\n",
        "    q = tf.random.normal((1, self.units))\n",
        "    f_p = self.f(p)\n",
        "    f_q = self.f(q)\n",
        "    w_epsilon = f_p*f_q\n",
        "    b_epsilon = tf.squeeze(f_q)\n",
        "     # w = w_mu + w_sigma*w_epsilon\n",
        "    self.w = self.w_mu + tf.multiply(self.w_sigma, w_epsilon)\n",
        "    ret = tf.matmul(inputs, self.w)\n",
        "\n",
        "    # b = b_mu + b_sigma*b_epsilon\n",
        "    self.b = self.b_mu + tf.multiply(self.b_sigma, b_epsilon)\n",
        "    if self.activation_fn is None:\n",
        "      return ret + self.b\n",
        "    else:\n",
        "      return self.activation_fn(ret + self.b)\n",
        "\n",
        "  def get_config(self):\n",
        "    config = super(noisy_dense, self).get_config()\n",
        "    config.update({\n",
        "        'units':\n",
        "            self.units,\n",
        "        'activation':\n",
        "            activations.serialize(self.activation)\n",
        "    })\n",
        "    return config\n",
        "\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    input_shape = tensor_shape.TensorShape(input_shape)\n",
        "    input_shape = input_shape.with_rank_at_least(2)\n",
        "    if tensor_shape.dimension_value(input_shape[-1]) is None:\n",
        "      raise ValueError(\n",
        "          'The innermost dimension of input_shape must be defined, but saw: %s'\n",
        "          % input_shape)\n",
        "    return input_shape[:-1].concatenate(self.units)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oJmMixzOXjaB"
      },
      "source": [
        "Creating a Noisy Net Model Class. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ximhl61pXjH7"
      },
      "source": [
        "class _noisy_dense_model(tf.keras.Model):\n",
        "  \n",
        "  def __init__(self,state_shape,action_shape,N_atoms):\n",
        "    super(_noisy_dense_model,self).__init__()\n",
        "\n",
        "    self.layer_1 = Conv2D(32,kernel_size=8,strides=4,activation='relu',input_shape=(60,60,4,))\n",
        "    self.layer_2 = MaxPooling2D(pool_size=(2,2))\n",
        "    self.layer_3 = Activation('relu')\n",
        "\n",
        "    self.layer_4 = Conv2D(64,kernel_size=4,strides=2,activation='relu')\n",
        "    self.layer_5 = MaxPooling2D(pool_size=(2,2))\n",
        "    self.layer_6 = Activation('relu')\n",
        "\n",
        "    self.layer_7 = Conv2D(64,kernel_size=3,strides=1,activation='relu')\n",
        "    self.layer_8 = MaxPooling2D(pool_size=(2,2))\n",
        "    self.layer_9 = Activation('relu')\n",
        "\n",
        "    self.layer_10 = Flatten()\n",
        "    self.layer_11 = noisy_dense(512,tf.keras.activations.relu)\n",
        "    self.layer_12 = noisy_dense(self.action_shape,tf.keras.activations.softmax)\n",
        "  \n",
        "  def call(self,x):\n",
        "    x = self.layer_3(self.layer_2(self.layer_1(x)))\n",
        "    x = self.layer_6(self.layer_5(self.layer_4(x)))\n",
        "    x = self.layer_9(self.layer_8(self.layer_7(x)))\n",
        "    x = self.layer_12(self.layer_11(self.layer_10(x)))\n",
        "    return x"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zX66WYgBdn5R"
      },
      "source": [
        "\n",
        "Defining the Noisy DDQN Class"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RPojIGcud3Vd"
      },
      "source": [
        "class Noisy_DDQN:\n",
        "\n",
        "  def __init__(self,memory_size,path_1=None,path_2=None):\n",
        "    self.memory=Memory(memory_size)\n",
        "    self.state_shape= (60, 60, 4) # the state space\n",
        "    self.action_shape=env.action_space.n # the action space\n",
        "    self.gamma=[0.99] # decay rate of past observations\n",
        "    self.learning_rate= 0.001 # learning rate in deep learning\n",
        "    self.epsilon_initial_value=1.0 # initial value of epsilon\n",
        "    self.epsilon_current_value=1.0# current value of epsilon\n",
        "    self.epsilon_final_value=0.001 # final value of epsilon\n",
        "    self.observing_episodes=10    #No of observations before training the training model\n",
        "    self.observing_episodes_target_model=200\n",
        "    self.batch_size=64\n",
        "    if not path_1:\n",
        "      self.target_model=_noisy_dense_model()    #Target Model is model used to calculate target values\n",
        "      self.training_model=_noisy_dense_model()  #Training Model is model to predict q-values to be used.\n",
        "    else:\n",
        "      self.training_model = _noisy_dense_model()\n",
        "      self.training_model.load_weights(path_1)\n",
        "      self.target_model = _noisy_dense_model()\n",
        "      self.target_model.load_weights(path_2)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nPNz73G9eCcY"
      },
      "source": [
        "Action Selection: The get_action method guides out action choice. Initially, when training begins we use exploration policy but later we do exploitation."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J4Z2xHjJeJrG"
      },
      "source": [
        "  def get_action(self, state,status='Training'):\n",
        "    '''samples the next action based on the E-greedy policy'''\n",
        "    if status=='Testing':\n",
        "      q_values=(self.training_model(state))[0]   #Exploitation\n",
        "      return np.argmax(q_values)\n",
        "    if random.random() < self.epsilon_current_value:                                    #Exlporation\n",
        "      action=random.choice(list(range((self.action_shape))))\n",
        "    else:\n",
        "      q_values=(self.training_model(state))[0]   #Exploitation\n",
        "      max_Q = np.argmax(q_values)\n",
        "      action = max_Q\n",
        "    return action"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5dYgQxc5eKQg"
      },
      "source": [
        "This is the preprocessing we do to the image we obtained by interacting with the environment. Here I have done grayscaling and also cropped the image to remove game scores and area which I found was not necessary to train the agent. Then I have downscaled the image.This speeds up the training process."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fJVXADJXeLpH"
      },
      "source": [
        "  def get_frame(self,frame):\n",
        "    frame=frame[25:-15,10:]\n",
        "    frame=cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY)\n",
        "    frame=(resize(frame,(60,60)))/255.\n",
        "    return frame"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pv-Pt7rNeZst"
      },
      "source": [
        "Updating the training_model\n",
        "\n",
        "The update_training_model method updates the training model weights.\n",
        "\n",
        "This is the same as the updating method used in DDQN With PER notebook"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "blXt9vQBeZSL"
      },
      "source": [
        " def update_training_model(self):\n",
        "\n",
        "    '''\n",
        "    Updates the policy network using the NN model.\n",
        "    '''\n",
        "\n",
        "\n",
        "    tree_idx, batch, ISWeights_mb = self.memory.sample(self.batch_size)\n",
        "    \n",
        "    states_mb=np.zeros((self.batch_size,*self.state_shape))\n",
        "    targets = np.zeros((self.batch_size,self.action_shape))\n",
        "    absolute_errors=np.zeros((self.batch_size,1))\n",
        "\n",
        "    for i in range(self.batch_size):\n",
        "      state=batch[i][0][0]\n",
        "      states_mb[i]=state\n",
        "      preds_=self.training_model(state)\n",
        "      targets[i]=preds_\n",
        "\n",
        "      reward=(batch[i][0][2])\n",
        "      \n",
        "      terminal=batch[i][0][4]\n",
        "\n",
        "     \n",
        "      action=batch[i][0][1]\n",
        "      if terminal:  # If we are in a terminal state, only equals reward\n",
        "        targets[i, action] = np.asarray(reward)\n",
        "      else:\n",
        "        next_state=batch[i][0][3]\n",
        "        preds_next_state_target_model=self.target_model(next_state)[0,action]\n",
        "        targets[i, action] =  np.asarray(reward) + np.asarray(self.gamma)*np.asarray(preds_next_state_target_model)   # Take the Qtarget for action a'\n",
        "      \n",
        "      absolute_errors[i]=np.abs(np.sum(targets[i]-preds_,axis=1))\n",
        "  \n",
        "      # Update priority\n",
        "      absolute_errors=absolute_errors/np.amax(absolute_errors)\n",
        "      self.memory.batch_update(tree_idx, absolute_errors)\n",
        "\n",
        "      optimizer = tf.keras.optimizers.Adam(learning_rate=self.learning_rate)\n",
        "      def train_step(states, targets,ISWeights):\n",
        "        with tf.GradientTape() as tape:\n",
        "          preds= (self.training_model)(states,training=True)  #This is the 𝑄@(𝑠,𝑎)\n",
        "          loss= ISWeights*(targets-preds)                 \n",
        "          \n",
        "        grads = tape.gradient(loss,self.training_model.trainable_variables)\n",
        "        optimizer.apply_gradients(zip(grads, self.training_model.trainable_variables))\n",
        "      train_step(states_mb,targets,ISWeights_mb)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G4Y8M9Dneqo7"
      },
      "source": [
        "Updating the target_model\n",
        "The update_target_model method sets the target model weights to training model weights."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3fXrFYajesOp"
      },
      "source": [
        "def update_target_model(self):\n",
        "    self.target_model.set_weights(self.training_model.get_weights())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0RpeC5yneuwE"
      },
      "source": [
        "Training the model\n",
        "This method creates a training environment for the model. Iterating through a set number of episodes, it uses the model to sample actions and play them. When such a timestep ends, the model is using the observations to update the policy.\n",
        "We know that in a dynamic game we cannot predict action based on 1 observation(which is 1 frame of the game in this case) so we will use a stack of 4 frames to predict the output.\n",
        "\n",
        "We can also downscale the rewards to help model learn faster."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BAvptUPudnY6"
      },
      "source": [
        "  def train(self, episodes):\n",
        "    '''\n",
        "    train the model\n",
        "    episodes - number of training iterations\n",
        "    ''' \n",
        "    for episode in range(episodes):\n",
        "      # each episode is a new game env\n",
        "      state=env.reset()\n",
        "      done=False\n",
        "      state= self.get_frame(state)\n",
        "      stacked_frame=np.stack((state,state,state,state),axis=2)\n",
        "      stacked_frame=stacked_frame.reshape(1,stacked_frame.shape[0],stacked_frame.shape[1],stacked_frame.shape[2])\n",
        "      episode_reward=0 #record episode reward\n",
        "      lives = 5\n",
        "      while not done:\n",
        "        # play an action and record the game state & reward per episode\n",
        "        action=self.get_action(stacked_frame)\n",
        "        next_state, reward, done, info=env.step(action)\n",
        "        reward = reward/10.0\n",
        "        next_state=self.get_frame(next_state)\n",
        "        next_state_ = next_state.reshape(1,next_state.shape[0],next_state.shape[1],1)\n",
        "        stacked_frames_1 = np.append(next_state_, stacked_frame[:, :, :, :3], axis=3)\n",
        "        experience = stacked_frame, action, reward, stacked_frames_1, 1*done\n",
        "        self.memory.store(experience)\n",
        "        stacked_frame=stacked_frames_1\n",
        "        episode_reward+=reward\n",
        "      print(\"Episode:{}  reward:{}\".format(episode,episode_reward))\n",
        "      if episode%50==0:\n",
        "        self.evaluate(episode)\n",
        "      if episode%self.observing_episodes==0 and episode!=0:\n",
        "        self.update_training_model()\n",
        "      if episode%self.observing_episodes_target_model==0 and episode!=0:\n",
        "        self.update_target_model()\n",
        "      if episode%500==0 and episode!=0:\n",
        "        weights = self.training_model.get_weights()\n",
        "        with open(\"training_model_{}.txt\".format(episode), \"wb\") as fp:\n",
        "          pickle.dump(weights, fp)\n",
        "        self.target_model.save_weights(\"target_model_{}\".format(episode))\n",
        "      if self.epsilon_current_value > self.epsilon_final_value:\n",
        "        self.epsilon_current_value=self.epsilon_current_value-(self.epsilon_initial_value-self.epsilon_final_value)/1000\n",
        "        print('Current Epsilon Value:',self.epsilon_current_value)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_JUs_VZbfF7F"
      },
      "source": [
        "memory_size=1000\n",
        "no_of_episodes=1000\n",
        "\n",
        "Agent=DDQN(memory_size)\n",
        "Agent.train(no_of_episodes)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rt3Ogo3YfCGP"
      },
      "source": [
        "With the help of below code we run our algorithm and see the success of it"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J3Anr8tdfCY_"
      },
      "source": [
        "class tester:\n",
        "\n",
        "  def __init__(self,path):\n",
        "    self.model = _noisy_dense_model()\n",
        "    with open(path, \"rb\") as fp:\n",
        "      weights = pickle.load(fp)\n",
        "    self.model(np.zeros(1,60,60,4));\n",
        "    self.model.set_weights(weights)\n",
        "      \n",
        "  def get_action(self, state):\n",
        "        '''samples the next action based on the E-greedy policy'''\n",
        "        q_values=(self.model(state))[0]    #Exploitation\n",
        "        action = np.argmax(q_values)\n",
        "        return action\n",
        "  \n",
        "  def get_frame(self,frame):\n",
        "    frame=frame[25:-15,10:]\n",
        "    frame=cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY)\n",
        "    frame=(resize(frame,(60,60)))/255.\n",
        "    return frame"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fnozsfCyfeQH"
      },
      "source": [
        "env=(wrap_env(gym.make(\"UpNDown-v0\")))\n",
        "state=env.reset()\n",
        "test=tester(\"Actor.h5\")\n",
        "state=test.get_frame(state)\n",
        "stacked_frames = np.stack((state,state,state,state),axis=2)\n",
        "stacked_frames = stacked_frames.reshape(1,stacked_frames.shape[0],stacked_frames.shape[1],stacked_frames.shape[2]) \n",
        "while True:\n",
        "  env.render('ipython')\n",
        "  action = test.get_action(stacked_frames)\n",
        "  next_state, reward, done, _=env.step(action)\n",
        "  print(action,reward)\n",
        "  next_state=test.get_frame(next_state)\n",
        "  next_state_ = next_state.reshape(1,next_state.shape[0],next_state.shape[1],1)\n",
        "  stacked_frames = np.append(next_state_, stacked_frames[:, :, :, :3], axis=3)\n",
        "  if done:\n",
        "    break\n",
        "env.close()\n",
        "show_video()"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}