Deep_Reinforcement_Learning_Part_7_.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "DDPG.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HGPNwtYfEBW8",
        "colab_type": "text"
      },
      "source": [
        "This is the Part-7 of the Deep Reinforcement Learning Notebook series. In this Notebook I have introduced Deep Determinsitic Policy Gradients(DDPG).\n",
        "\n",
        "The Notebook series is about Deep RL algorithms so it excludes all other techniques that can be used to learn functions in reinforcement learning and also the Notebook Series is not exhaustive i.e. it contains the most widely used Deep RL algorithms only."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rHR_l8HmF36E",
        "colab_type": "text"
      },
      "source": [
        "##What is DDPG Algorithm?\n",
        "\n",
        "\n",
        "DDPG is extension of Deep Q-Learning(DQN) for continous tasks. DDPN is introduced in paper CONTINUOUS CONTROL WITH DEEP REINFORCEMENT LEARNING\n",
        "(Lillicrap et al., 2016). DDPG is an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces.\n",
        "\n",
        "In past notebook we read that DQNs can be imporoved by using Double Dqn and target networks. If you want to read more about Doble Dqn and target networks checkout this link: https://github.com/Rahul-Choudhary-3614/Deep-Reinforcement-Learning-Notebooks/blob/master/Deep_Reinforcement_Learning_Part_5_.ipynb\n",
        "\n",
        "DDQN uses all these techniques to make training more stable\n",
        "\n",
        "An obvious approach to adapting deep reinforcement learning methods such as DQN to continuous domains is to to simply discretize the action space. However, this has many limitations, most notably the curse of dimensionality: the number of actions increases exponentially with the number of degrees of freedom. For example, a 7 degree of freedom system (as in the human arm) with the coarsest discretization ai ∈ {−k, 0, k} for each joint leads to an action space with dimensionality: $3^7$ = 2187. The situation is even worse for tasks that require fine control of actions as they require a correspondingly finer grained discretization, leading to an explosion of the number of discrete actions. Such large action spaces are difficult to explore efficiently, and thus successfully training DQN-like networks in this context is likely intractable. Additionally, naive discretization of action spaces needlessly throws away information about the structure of the action domain, which may be essential for solving many problems.\n",
        "\n",
        "DDPG is  a model-free, off-policy actor-critic algorithm using deep function approximators that can learn policies in high-dimensional, continuous action spaces.\n",
        "\n",
        "DDPG  combines DQN with policy gradient methods using the actor-critic framework to learn a deterministic policy $\\mu_\\theta(s)$ that acts to approximate Q-learning with guidance from a DQN-like critic $Q_\\omega(s,a)$.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l5ENcd93QIdZ",
        "colab_type": "text"
      },
      "source": [
        "## Deterministic Policy Gradients (DPG)\n",
        "DPG were introduced in paper Deterministic Policy Gradient Algorithms (Silver et al., 2014).A policy can be either deterministic or stochastic. \n",
        "\n",
        "A policy can be either deterministic or stochastic. A deterministic policy is policy that maps state to actions. You give it a state and the function returns an action to take.On the other hand, a stochastic policy outputs a probability distribution over actions.\n",
        "\n",
        "Deterministic policy gradient methods have better relative sample efficiency since they don't integrate over action space while stochastic policy methods integrate over both state and action space. The DPG paper (Silver et al., 2014) showed deterministic policy gradients are the expectation of the action value gradient and introduced a deterministic version of the policy gradient theorem to provide an expression for $\\nabla_\\theta J(\\theta)$ that doesn't require the derivative of the state distribution $\\rho^\\beta$.\n",
        "\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-08-14%20at%201.05.52%20AM.png?raw=true)\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iboPyLD5VnmH",
        "colab_type": "text"
      },
      "source": [
        "##Batch Normalization\n",
        "When learning from low dimensional feature vector observations, the different components of the observation may have different physical units (for example, positions versus velocities) and the ranges may vary across environments. This can make it difficult for the network to learn effectively and may make it difficult to find hyper-parameters which generalise across environments with different scales of state values. This issue is address by adapting technique called batch normalization.\n",
        "\n",
        "This technique normalizes each dimension across the samples in a minibatch to have unit mean and variance. In addition, it maintains a run- ning average of the mean and variance to use for normalization during testing (in our case, during exploration or evaluation). In deep networks, it is used to minimize covariance shift during training, by ensuring that each layer receives whitened input. In the low-dimensional case, batch normalization is used on the state input and all layers of the μ network and all layers of the Q network prior to the action input.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zUZ81TpNWrA6",
        "colab_type": "text"
      },
      "source": [
        "##Exploration in Continuous action space\n",
        "Deterministic policies have a harder time attaining sufficient exploration compared to stochastic polices. Continuous action spaces also make exploration important. An advantage of off- policies algorithms such as DDPG is that we can treat the problem of exploration independently from the learning algorithm. We constructed an exploration policy μ′ by adding noise sampled from a noise process N to our actor policy\n",
        "\n",
        "$μ'(s_t)$=$μ(s_t/θ_t^μ)$ + $\\mathcal{N}$\n",
        "\n",
        "$\\mathcal{N}$ can be chosen to suit the environment. $\\mathcal{N}$ used in DDPG paper is an Ornstein-Uhlenbeck process so the exploration is temporally correlated. Other noises such as plain Gaussian noise works just as well."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i3LThsEMYOtc",
        "colab_type": "text"
      },
      "source": [
        "##DDPG \n",
        "\n",
        "![alt text](https://github.com/Machine-Learning-rc/Unimportant/blob/master/Screenshot%202020-08-14%20at%201.22.35%20AM.png?raw=true)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DPzcadXQY0OH",
        "colab_type": "text"
      },
      "source": [
        "#IMPLEMENTING DDPG"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h0ucWoIn2W5w",
        "colab_type": "text"
      },
      "source": [
        "Below code setups the environment required to run and record the game and also loads the required library."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IaQTUrD2z5zu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!apt-get update > /dev/null 2>&1\n",
        "!apt-get install cmake > /dev/null 2>&1\n",
        "!pip install --upgrade setuptools 2>&1\n",
        "!pip install ez_setup > /dev/null 2>&1\n",
        "!pip install gym[atari] > /dev/null 2>&1\n",
        "!pip install gym pyvirtualdisplay > /dev/null 2>&1\n",
        "!apt-get install -y xvfb python-opengl ffmpeg > /dev/null 2>&1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2JqZMpBf2eOY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras.layers import Dense,Dropout,Conv2D, Flatten,MaxPooling2D ,Activation,Input\n",
        "from tensorflow.keras.models import Sequential,load_model,Model\n",
        "import gym\n",
        "import numpy as np\n",
        "import random\n",
        "from collections import deque\n",
        "from tensorflow.keras.utils import normalize as normal_values\n",
        "import cv2\n",
        "from gym import logger as gymlogger\n",
        "from gym.wrappers import Monitor\n",
        "gymlogger.set_level(40) #error only\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "import math\n",
        "import glob\n",
        "import io\n",
        "import base64\n",
        "from IPython.display import HTML\n",
        "from IPython.display import clear_output\n",
        "from IPython import display as ipythondisplay"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wQ1lAMJw2eu0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from pyvirtualdisplay import Display\n",
        "display = Display(visible=0, size=(1400, 900))\n",
        "display.start()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lpVRvNTO2kZY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\"\"\"\n",
        "Utility functions to enable video recording of gym environment and displaying it\n",
        "To enable video, just do \"env = wrap_env(env)\"\"\n",
        "\"\"\"\n",
        "\n",
        "def show_video():\n",
        "  mp4list = glob.glob('video/*.mp4')\n",
        "  if len(mp4list) > 0:\n",
        "    mp4 = mp4list[0]\n",
        "    video = io.open(mp4, 'r+b').read()\n",
        "    encoded = base64.b64encode(video)\n",
        "    ipythondisplay.display(HTML(data='''<video alt=\"test\" autoplay \n",
        "                loop controls style=\"height: 400px;\">\n",
        "                <source src=\"data:video/mp4;base64,{0}\" type=\"video/mp4\" />\n",
        "             </video>'''.format(encoded.decode('ascii'))))\n",
        "  else: \n",
        "    print(\"Could not find video\")\n",
        "    \n",
        "\n",
        "def wrap_env(env):\n",
        "  env = Monitor(env, './video', force=True)\n",
        "  return env"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k08o59jX2p8p",
        "colab_type": "text"
      },
      "source": [
        "This part ensures the reproducibility of the code below by using a random seed and setups the environment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WvQaf4YB2oHI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "RANDOM_SEED=1\n",
        "# random seed (reproduciblity)\n",
        "np.random.seed(RANDOM_SEED)\n",
        "tf.random.set_seed(RANDOM_SEED)\n",
        "\n",
        "# set the env\n",
        "env = (gym.make(\"Pendulum-v0\")) # env to import\n",
        "#env=wrap_env(env)  #use this when you want to record a video of episodes\n",
        "env.seed(RANDOM_SEED)\n",
        "env.reset() # reset to env\n",
        "\n",
        "State_Space = env.observation_space.shape[0]\n",
        "actions_Space = env.action_space.shape[0]\n",
        "\n",
        "upper_bound = env.action_space.high[0]\n",
        "lower_bound = env.action_space.low[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "scZYvvpw207b",
        "colab_type": "text"
      },
      "source": [
        "Defining the OUNoise Class."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-I-uIAZA3L5f",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# --------------------------------------\n",
        "# Ornstein-Uhlenbeck Noise\n",
        "# Author: Flood Sung\n",
        "# Date: 2016.5.4\n",
        "# Reference: https://github.com/rllab/rllab/blob/master/rllab/exploration_strategies/ou_strategy.py\n",
        "# --------------------------------------\n",
        "\n",
        "import numpy as np\n",
        "import numpy.random as nr\n",
        "\n",
        "class OUNoise:\n",
        "    \"\"\"docstring for OUNoise\"\"\"\n",
        "    def __init__(self,action_dimension,mu=0, theta=0.15, sigma=0.2):\n",
        "        self.action_dimension = action_dimension\n",
        "        self.mu = mu\n",
        "        self.theta = theta\n",
        "        self.sigma = sigma\n",
        "        self.state = np.ones(self.action_dimension) * self.mu\n",
        "        self.reset()\n",
        "\n",
        "    def reset(self):\n",
        "        self.state = np.ones(self.action_dimension) * self.mu\n",
        "\n",
        "    def noise(self):\n",
        "        x = self.state\n",
        "        dx = self.theta * (self.mu - x) + self.sigma * nr.randn(len(x))\n",
        "        self.state = x + dx\n",
        "        return self.state\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    ou = OUNoise(3)\n",
        "    states = []\n",
        "    for i in range(1000):\n",
        "        states.append(ou.noise())\n",
        "    import matplotlib.pyplot as plt\n",
        "\n",
        "    plt.plot(states)\n",
        "    plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6IOe7pM6-6kt",
        "colab_type": "text"
      },
      "source": [
        "Defining the DDPG Class. At initiation, the DDPG object sets a few parameters like environment, action bounds,exploration noise,create Actor,target Actor,Critic and target Critic models and remember(that records the observations of each step.)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "63-PDhR66EnA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class DDPG:\n",
        " \n",
        "  def __init__(self,env,upper_bound,lower_bound,buffer_capacity=50000,p_1=None,p_2=None,p_3=None,p_4=None):\n",
        "    self.env=env #import env\n",
        "    self.upper_bound=upper_bound\n",
        "    self.lower_bound=lower_bound\n",
        "    self.state_shape=env.observation_space.shape[0] # the state space\n",
        "    self.action_shape=env.action_space.shape[0] # the action space\n",
        "    self.gamma=0.99 # decay rate of past observations\n",
        "    self.learning_rate= 1e-4 # learning rate in deep learning\n",
        "    self.alpha=0.005\n",
        "    self.batch_size=64\n",
        "    self.exploration_noise=OUNoise(self.action_shape)\n",
        "    if not p_1:\n",
        "      self.Actor_model=self._create_model('Actor')\n",
        "      self.target_Actor_model=self._create_model('Actor') \n",
        "      self.Critic_model=self._create_model('Critic')      \n",
        "      self.target_Critic_model=self._create_model('Critic')  \n",
        "    else:\n",
        "      self.Actor_model=load_model(p_1) \n",
        "      self.Critic_model=load_model(p_2) \n",
        "      self.target_Actor_model=load_model(p_3)\n",
        "      self.target_Critic_model=load_model(p_4) \n",
        "    \n",
        "    self.buffer_capacity = buffer_capacity\n",
        "    self.buffer_counter = 0\n",
        "        # record observations\n",
        "    self.states=np.zeros((self.buffer_capacity, self.state_shape))\n",
        "    self.rewards=np.zeros((self.buffer_capacity,1))\n",
        "    self.dones= np.zeros((self.buffer_capacity, 1))\n",
        "    self.actions=np.zeros((self.buffer_capacity, 1))\n",
        "    self.next_states=np.zeros((self.buffer_capacity, self.state_shape))\n",
        "  \n",
        "  def remember(self,state, action, reward,next_state,done):\n",
        "    '''stores observations'''\n",
        "    index = self.buffer_counter % self.buffer_capacity\n",
        "    self.states[index] = state\n",
        "    self.rewards[index]=reward\n",
        "    self.dones[index]=done\n",
        "    self.actions[index]=action\n",
        "    self.next_states[index]=next_state\n",
        "    self.buffer_counter += 1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mBBxfL5UCMMo",
        "colab_type": "text"
      },
      "source": [
        "Creating a Neural Network Model (Actor and Critic)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8GMU722wCKid",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def _create_model(self,model_type):\n",
        " \n",
        "    ''' builds the model using keras'''\n",
        " \n",
        "    state_input = Input(shape=(3,))\n",
        "    layer_1=Dense(512,activation=\"relu\")(state_input)\n",
        "    layer_2=BatchNormalization()(layer_1)\n",
        "    layer_3=Dense(512,activation=\"relu\")(layer_2)\n",
        "    layer_4=BatchNormalization()(layer_3)\n",
        "    layer_5=Dense(64,activation=\"relu\")(layer_4)\n",
        "    layer_6=BatchNormalization()(layer_5)\n",
        "\n",
        "    if model_type=='Actor':\n",
        "      output = Dense(self.action_shape, activation='tanh',kernel_initializer=tf.keras.initializers.RandomUniform(minval=-0.003, maxval=0.003,seed=1))(layer_6)\n",
        "      output=output * self.upper_bound\n",
        "      model = Model(inputs=[state_input],outputs=[output])\n",
        "      model.compile(optimizer=Adam(learning_rate=self.learning_rate), loss=\"mse\")\n",
        "    else:\n",
        "      action_input = Input(shape=(self.action_shape))\n",
        "      action_layer_1 = Dense(128, activation=\"relu\")(action_input)\n",
        "      action_layer_2 = BatchNormalization()(action_layer_1 )\n",
        "      action_layer_3 = Dense(64, activation=\"relu\")(action_layer_2)\n",
        "      action_layer_4 = BatchNormalization()(action_layer_3)\n",
        "      concat = Concatenate()([layer_6,action_layer_4])\n",
        "\n",
        "      concat_layer_1=Dense(256,activation=\"relu\")(concat)\n",
        "      concat_layer_2=BatchNormalization()(concat_layer_1)\n",
        "\n",
        "      output = Dense(1)(concat_layer_2)\n",
        "\n",
        "      model = Model(inputs=[state_input, action_input],outputs=[output])\n",
        "      model.compile(optimizer=Adam(learning_rate=self.learning_rate), loss=\"mse\")\n",
        "    return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y_3mDvozCbUx",
        "colab_type": "text"
      },
      "source": [
        "Action Selection\n",
        "\n",
        "The get_action method guides the action choice. It uses the Actor network to get a action for a given state. Then, we add noise during training for\n",
        "exploration. Then we clip the action so that action doesn't got out of legal actions range."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dC0Bd-aCDQJN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_action(self, state,status=\"training\"):\n",
        "    '''samples the next action based on the policy probabilty distribution \n",
        "      of the actions'''\n",
        "    action = self.Actor_model.predict(state)\n",
        "    \n",
        "    if status==\"training\":\n",
        "        action = action + self.exploration_noise.noise()\n",
        "\n",
        "\n",
        "    # We make sure action is within bounds\n",
        "    legal_action = np.clip(action, self.lower_bound, self.upper_bound)\n",
        "\n",
        "    \n",
        "    return legal_action"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z052-mQ2DCCT",
        "colab_type": "text"
      },
      "source": [
        "Updating networks\n",
        "\n",
        "We update critic and actor according to loss that we discussed earlier and do soft updates on target networks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Inr6-X7oDiYE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        " def update_models(self):\n",
        "    '''\n",
        "    Updates the network.\n",
        "    '''\n",
        "    record_range = min(self.buffer_counter, self.buffer_capacity)\n",
        "    batch_indices = np.random.choice(record_range,self.batch_size)\n",
        "\n",
        "    states_mb=self.states[batch_indices]\n",
        "    actions_mb=self.actions[batch_indices]\n",
        "    next_states_mb=self.next_states[batch_indices]\n",
        "    rewards_mb=self.rewards[batch_indices]\n",
        "    dones_mb=self.dones[batch_indices]\n",
        "\n",
        "    optimizer = tf.keras.optimizers.Adam(learning_rate=self.learning_rate)\n",
        "  \n",
        "\n",
        "    def train_step_critic(next_states,rewards,dones):\n",
        "      with tf.GradientTape() as tape:\n",
        "        target_actions=self.target_Actor_model(next_states)\n",
        "        critic_value = self.Critic_model([next_states, target_actions])\n",
        "        next_state_critic_value=self.target_Critic_model([next_states,target_actions])\n",
        "        targets=rewards+(1-dones)*self.gamma*next_state_critic_value\n",
        "        critic_loss = tf.reduce_mean((targets-critic_value)**2)\n",
        "        print(\"Critic Loss:\",critic_loss)\n",
        "      critic_grad = tape.gradient(critic_loss, self.Critic_model.trainable_variables)\n",
        "      optimizer.apply_gradients(zip(critic_grad, self.Critic_model.trainable_variables))\n",
        "    train_step_critic(next_states_mb,rewards_mb,dones_mb)\n",
        "\n",
        "    \n",
        "    def train_step_actor(states):\n",
        "      with tf.GradientTape() as tape:\n",
        "        actions=self.Actor_model(states)\n",
        "        critic_value = self.Critic_model([states, actions])\n",
        "        actor_loss = -tf.reduce_mean(critic_value)\n",
        "        print(\"Actor Loss:\",actor_loss)\n",
        "      actor_grad = tape.gradient(actor_loss, self.Actor_model.trainable_variables)\n",
        "      optimizer.apply_gradients(zip(actor_grad, self.Actor_model.trainable_variables))\n",
        "    train_step_actor(states_mb)\n",
        "  \n",
        "  def update_target_models(self):\n",
        "    actor_weights = np.array(self.Actor_model.get_weights())\n",
        "    actor_tartget_weights = np.array(self.target_Actor_model.get_weights())\n",
        "    new_weights = self.alpha*actor_weights + (1-self.alpha)*actor_tartget_weights\n",
        "    self.target_Actor_model.set_weights(new_weights)\n",
        "\n",
        "    critic_weights = np.array(self.Critic_model.get_weights())\n",
        "    critic_tartget_weights = np.array(self.target_Critic_model.get_weights())\n",
        "    new_weights = self.alpha*critic_weights + (1-self.alpha)*critic_tartget_weights\n",
        "    self.target_Critic_model.set_weights(new_weights)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "acLSM1alDqXq",
        "colab_type": "text"
      },
      "source": [
        "Training and evaluating the model\n",
        "\n",
        "This method creates a training environment for the model. Iterating through a set number of episodes, it uses the model to sample actions and play them. When such a sequence ends, the model is using the recorded observations to update the policy.\n",
        "\n",
        "The evaluate method helps us keep an eye on the model's performance\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c4pxtTWQCaWQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "  def evaluate(self):\n",
        "    Average_Reward=0.0\n",
        "    for episode in range(20):\n",
        "      env = (gym.make(\"Pendulum-v0\"))\n",
        "      state_=(env.reset()).reshape((1,3))\n",
        "      done=False\n",
        "      episode_reward=0  \n",
        "      print(\"Episode Started\")\n",
        "      while not done:\n",
        "        action=self.get_action(state_,\"testing\")\n",
        "        next_state, reward, done, info=env.step(action) \n",
        "        next_state.reshape((1,3))\n",
        "        episode_reward+=reward\n",
        "      Average_Reward+=episode_reward\n",
        "      print(\"Testing_Episode:{}  Reward:{} Average_Reward:{} \\n\\n\".format(episode,episode_reward,Average_Reward/20.0))\n",
        "      print(\"Episode Ended\")\n",
        " \n",
        " \n",
        "  def train(self,episodes):\n",
        "    env=self.env\n",
        "    for episode in range(episodes):\n",
        "      state_=(env.reset()).reshape((1,3))\n",
        "      done=False\n",
        "      episode_reward=0  \n",
        "      print(\"Episode Started\")\n",
        "      while not done:\n",
        "        action=self.get_action(state_)\n",
        "        next_state, reward, done, info=env.step(action)\n",
        "        next_state=next_state.reshape((1,3)) \n",
        "        self.remember(state_, action, reward,next_state,done)\n",
        "        self.update_models()\n",
        "        self.update_target_models()\n",
        "        state_ = next_state.reshape((1,3))\n",
        "        episode_reward+=reward\n",
        "      print(\"Episode:{}  Reward:{}\\n\".format(episode,episode_reward))\n",
        "      print(\"Episode Ended\")\n",
        "      if episode%100==0 and episode!=0:\n",
        "        self.Actor_model.save('Actor_{}.h5'.format(episode+900))\n",
        "        self.target_Actor_model.save('target_Actor_{}.h5'.format(episode+900))\n",
        "        self.Critic_model.save('Critic_{}.h5'.format(episode))\n",
        "        self.target_Critic_model.save('target_Critic_{}.h5'.format(episode+900))\n",
        "        print(\"\\n\\n\",\"Evaluating\")\n",
        "        self.evaluate()\n",
        "      "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bEK_AU81E5Zn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "Agent=DDPG(env,upper_bound,lower_bound)\n",
        "Agent.train(episodes=300)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}