Leave_No_Trace.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Leave_No_Trace.ipynb",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HkUm4luQtVac",
        "colab_type": "text"
      },
      "source": [
        "# Leave no trace"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lAXw3be4oH4p",
        "colab_type": "text"
      },
      "source": [
        "# **INTRODUCTION**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JF1BvoFKoLQc",
        "colab_type": "text"
      },
      "source": [
        "In this work, we are going to study a paper : \"*Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning*\". For this matter we are going to train a bipedal robot how to walk using the framework introduced in the paper. We will build our work upon a benchmark : Soft-Actor Critic Method, and use it as a baseline."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m7fYk7D3t_Kh",
        "colab_type": "text"
      },
      "source": [
        "# **Environment presentation : BipedalWalker-v2**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ljP0olHgx-kE",
        "colab_type": "text"
      },
      "source": [
        "![Texte alternatif…](https://miro.medium.com/max/1348/1*1hZSWhdg1BKy9iXlPA_luA.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NXmyW7-5uJmj",
        "colab_type": "text"
      },
      "source": [
        "For the purpose of our experiments, we will be using **Gym**,  a toolkit for developing and comparing reinforcement learning algorithms. Gym offers many environments in which we can train different types of agents to accomplish some tasks. The environment we will use for our experiments is the **Bipedal walker envronment**. <br><br>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eD59szFiumVe",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "In this environment a 2D bipedal walker has to learn a policy to walk without falling over.\n",
        "\n",
        "The total reward calculation is based on the total distance achieved by the agent. The episode ends when the robot body touches ground or the robot reaches far right side of the environment. BipedalWalker-v2 defines \"solving\" as getting average reward of 300 over 100 consecutive trials\n",
        "\n",
        "https://gym.openai.com/envs/BipedalWalker-v2/\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Owx77MdYwCad",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "<br><br>\n",
        "**How is the agent interacting with his environment?**<br>\n",
        "Source: [OpenAI](https://openai.com/)\n",
        "\n",
        "Each timestep, the agent chooses an action, and the environment returns an observation and a reward.\n",
        "\n",
        "- observation (object): an environment-specific object representing your observation of the environment. For example, pixel data from a camera, joint angles and joint velocities of a robot, or the board state in a board game line Taxi.\n",
        "- reward (float): amount of reward achieved by the previous action. The scale varies between environments, but the goal is always to increase your total reward.\n",
        "- done (boolean): whether it’s time to reset the environment again. Most (but not all) tasks are divided up into well-defined episodes, and done being True indicates the episode has terminated. (For example, perhaps the pole tipped too far, or you - lost your last life.)\n",
        "info (dict): ignore, diagnostic information useful for debugging. Official evaluations of your agent are not allowed to use this for learning.\n",
        "<br><br>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N9xFuidExeWV",
        "colab_type": "text"
      },
      "source": [
        "For example, for our walker :<br>\n",
        "- **Observations** : are made of a collection of 24 numeric values representing the state of our agent at a given step. Those include the angle of its \"right\" or \"left\" hips, same for the knees, the inclination angle of the hull (0 meaning that the hull is horizontal, thus stable), lidar data collected by the agent, giving an idea about the way the agent stands, its velocity etc.\n",
        "\n",
        "\n",
        "- **Rewards** : The total reward calculation is based on the total distance achieved by the agent, total 300+ points up to the far end. If the robot falls, it gets -100 and the game ends. The episode ends when the robot body touches ground or the robot reaches far right side of the environment.\n",
        "\n",
        "- **Actions** : the input the agent provides to the environment. Given a state, our agent can take 4 actions at time (making one action at a step) : Hip_1 Torque, Knee_1 Torque, Hip_2 Torque, Knee_2 Torque. Meaning, we control the 2 legs of the robot, and each leg has 2 motors (hip and knee). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zM0efsthoOrQ",
        "colab_type": "text"
      },
      "source": [
        "# **Benchmark Framework**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kQdYSCNUolw9",
        "colab_type": "text"
      },
      "source": [
        "As we previously said, our goal is to do some experiments emphasizing some results of the paper. For this, we will be building our work upon an existing benchmark.\n",
        "<br><br>\n",
        "For this purpose, we will use a Pytorch implementation of continuous action actor-critic algorithm. The algorithm uses DeepMind's Deep Deterministic Policy Gradient (DDPG) method for updating the actor and critic networks along with Ornstein–Uhlenbeck process for exploring in continuous action space while using a Deterministic policy.\n",
        "DDPG is a policy gradient alogrithm, that uses stochastic behaviour policy for exploration (Ornstein-Uhlenbeck in this case) and outputs a deterministic target policy, which is easier to learn.\n",
        "<br><br>\n",
        "In this framework, we use a Soft Actor Critic (SAC) for the agent. In our SAC implementation, we have:\n",
        "- **The actor** : a network made of 3-layer neural network that takes as an input the current state and outputs an action (denoting the policy). This network is updated by maximizing the critic score : $\\sum Q(s,a)$\n",
        "- **The target actor** : which is a neural network, the target policy, updated with a soft update rule (weighted sum) using the current target actor parameters and the actor parameters. This is the policy used when we wish to exploit our agent policy.\n",
        "- **The critic** : this network consists of a 3-layer neural network taking into input the state (s) and correspoding action (a) and outputs the state-action value function denoted by Q(s,a). This network is updated by minimizing the loss between the prediction of the network and what hould this prediction be (the current gained reward added to the cumulated Q-value gained up to now estimated using the current policy applied to the previous state - temporal difference learning) : $L_2(r+\\gamma*Q(s1,a1) - Q(s,a))$\n",
        "- **The target critic** : same principle as the target actor but with the critic.\n",
        "<br><br>\n",
        "\n",
        "**PS**:\n",
        "- We helped ourselves with this [project](https://github.com/vy007vikas/PyTorch-ActorCriticRL).\n",
        "- The purpose of this study is not to experiment the SAC on itself, but use it as a benchmark and a tool to experiment the framework introduced in the paper we study : the Leave-No-Trace framework. Therefore, from now on, the SAC will be treated as a black box."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-Kle0V1F1dHE",
        "colab_type": "text"
      },
      "source": [
        "![Texte alternatif…](https://raw.githubusercontent.com/steph1793/Leave-No-Trace/master/images/Algo1.PNG)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AESpscqyomf1",
        "colab_type": "text"
      },
      "source": [
        "# **Leave-No-Trace Framework**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-yD9--Feo8l8",
        "colab_type": "text"
      },
      "source": [
        "The idea of the authors is to introduce a framework which allows the agent to detect actions that would lead to a failure avoid them, and do only reversible actions. This framework is said to have many advantages that we will try to check in the next experiments:\n",
        "\n",
        "- The agent can make longer steps before failing (less resets)\n",
        "- The agent learns more safe actions (reversible actions, allowing him to go back easily to it's initial state)\n",
        "- The performance of the agent is not altered by the framework.\n",
        "\n",
        "All those are some advantages claimed in the paper.\n",
        "\n",
        "To achieve this, the authors introduced a second agent, the reset agent, which alternatly learns from a given state how to reach an equilibirum state (generally the inital state), in which the agent can be considered to be safe and will not fail. The main agent becomes what we call a forward agent. And while training this forward agent, we regularly use the critic of the reset agent in order to evaluate how good is each action the agent is about to take. If the action is considered to be bad, we switch to the reset policy and then take an action advised by this reset policy. For our reset agent, we will also use a SAC since it is an off-policy method with a Q-learning network.\n",
        "<br><br>\n",
        "\n",
        "**PS : For the reset agent, we will hav to define  rewatrd function other than the reward function of the main agent. We will talk much more about it in the experiments.**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uieCPchcEY6b",
        "colab_type": "text"
      },
      "source": [
        "![Texte alternatif…](https://raw.githubusercontent.com/steph1793/Leave-No-Trace/master/images/algo2.PNG)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xbLNSQkpX7se",
        "colab_type": "text"
      },
      "source": [
        "# **Install some dependencies and import them**\n",
        "Pay attention to restart the notebook after installing the dependencies."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pV3o6O5OkhSV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install folium==0.2.1 > /dev/null 2>&1\n",
        "!apt-get update > /dev/null 2>&1\n",
        "!apt-get install cmake > /dev/null 2>&1\n",
        "!pip install --upgrade setuptools 2>&1\n",
        "!pip install ez_setup > /dev/null 2>&1\n",
        "!pip install gym[atari] > /dev/null 2>&1\n",
        "!pip install 'gym[box2d]' > /dev/null 2>&1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PnuLa6vaqo2J",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!git clone https://github.com/steph1793/Leave-No-Trace"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AQxth2s3qwCr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "cd Leave-No-Trace"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bv5aMNE72VKg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!mkdir results\n",
        "!mkdir results/Models\n",
        "!mkdir results/Models2\n",
        "!mkdir results/Models4"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "go1flRLHeerQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import gym\n",
        "from gym import logger as gymlogger\n",
        "gymlogger.set_level(40) #error only\n",
        "import tensorflow as tf\n",
        "import torch.nn.functional as F\n",
        "import torch\n",
        "from torch.autograd import Variable\n",
        "import numpy as np\n",
        "import random\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "import math\n",
        "\n",
        "import buffer\n",
        "from helper import show_video, wrap_env\n",
        "from utils import create_agent\n",
        "from train import Trainer\n",
        "\n",
        "import gc"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PkA16NBGketi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#remove \" > /dev/null 2>&1\" to see what is going on under the hood\n",
        "!pip install gym pyvirtualdisplay > /dev/null 2>&1\n",
        "!apt-get install -y xvfb python-opengl ffmpeg > /dev/null 2>&1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NUpAXhvBq4pv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from pyvirtualdisplay import Display\n",
        "display = Display(visible=0, size=(1400, 900))\n",
        "display.start()\n",
        "\n",
        "\n",
        "def video_callable(_ep):\n",
        "  if _ep<50:\n",
        "    return (_ep)%10==0\n",
        "  else:\n",
        "    return (_ep)%50==0 "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YCfx0zCLEwsB",
        "colab_type": "text"
      },
      "source": [
        "# **Visualize the environement**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FRwzePusFCnE",
        "colab_type": "code",
        "cellView": "both",
        "colab": {}
      },
      "source": [
        "env = gym.make('BipedalWalker-v3')\n",
        "obs= env.reset()\n",
        "for _ in range(15):\n",
        "    action = [1,1,-1,-1]\n",
        "    obs, rew, done, info = env.step(action)\n",
        "    plt.imshow(env.render('rgb_array'))\n",
        "    plt.grid(False)\n",
        "env.close()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-aBkf4ObFv0t",
        "colab_type": "text"
      },
      "source": [
        "# **Some Hyperparameters**\n",
        "\n",
        "Here we define some hyperparameters like the maximum number of episodes to run, the maximum number of steps per episode. We also intialize a \"MAX_BUFFER\" variable which represents the size of the buffer in which we store the different (state, action, reward, state) tuples and from which we sample the training batch."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "--suB3Z8FuoH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "MAX_EPISODES = 1500\n",
        "MAX_STEPS = 800\n",
        "MAX_BUFFER = 1000000\n",
        "MAX_TOTAL_REWARD = 300\n",
        "\n",
        "\n",
        "S_DIM = env.observation_space.shape[0]\n",
        "A_DIM = env.action_space.shape[0]\n",
        "A_MAX = env.action_space.high[0]\n",
        "\n",
        "print(' State Dimensions : ', S_DIM)\n",
        "print(' Action Dimensions : ', A_DIM)\n",
        "print(' Action Max : ', A_MAX)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GDRsYqayLv7h",
        "colab_type": "text"
      },
      "source": [
        "# **EXPERIMENTS**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9BuAyuGbiEeJ",
        "colab_type": "text"
      },
      "source": [
        "**Our experiments**:\n",
        "\n",
        "- First, we will run our benchmark model and collect some informations that we will study later in the notebook.\n",
        "- Secondly, we will run our Leave-No-Trace Framework\n",
        "<br><br>\n",
        "\n",
        "**Our goal** is to study:\n",
        "\n",
        "- how good is the SAC only method.\n",
        "- the performance of the main agent when helped by the reset agent\n",
        "- Comparative study of both methods\n",
        "- the impact of the reset agent \n",
        "- the impact of the reset reward function\n",
        "<br><br>\n",
        "\n",
        "For the experiments, we will look essentially at two **performance \"metrics\"**:\n",
        "\n",
        "- The maximum number of steps achieved by the agent per episode\n",
        "- The total number of rewards collected throughout each episode\n",
        "\n",
        "We will also visualize and comment the training using some videos.\n",
        "We run a series of four experimentations. Each experiment took at **least 6 hours** for training the agent(s). \n",
        "<br><br><br>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c72iP_6rL2vW",
        "colab_type": "text"
      },
      "source": [
        "## **SAC only *experiment***"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "am6BovLGnXNz",
        "colab_type": "text"
      },
      "source": [
        "First, we creat our environement. We  wrap this environment in a wrapper, allowing us to limit the maximum number of steps per episode (we want our agent to reach the maximum number of points within a given number of steps)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D6L8SBkM-0Bi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "env = wrap_env(gym.make('BipedalWalker-v3'), MAX_STEPS, video_folder='./results/video', video_callable=video_callable )\n",
        "ram = buffer.MemoryBuffer(MAX_BUFFER)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2k3GhygEn1sj",
        "colab_type": "text"
      },
      "source": [
        "Then, we create the agent (Soft Actor Critic)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZtifrHhSX9qT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "agent = create_agent(S_DIM, A_DIM, A_MAX)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J8iT1yDqn-z0",
        "colab_type": "text"
      },
      "source": [
        "We create a trainer object which will be a helper for exploring/exploiting the agent policy and optimizing this agent ( by updating its parameters)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3iroHabHf5l7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "trainer = Trainer(agent, ram)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "68QY5jX0oeyq",
        "colab_type": "text"
      },
      "source": [
        "Finally, we write our algorithm described in the pseudo-code above, \"Benchmark Framework\" part."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NZs6doEQjurY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "total_rewards_per_ep = []\n",
        "num_steps_per_ep = []\n",
        "for _ep in range(MAX_EPISODES):\n",
        "  observation = env.reset()\n",
        "  print()\n",
        "  r = 0\n",
        "  for i in range(MAX_STEPS):\n",
        "    env.render()\n",
        "    state = np.float32(observation)\n",
        "\n",
        "    action = trainer.get_exploration_action(state)\n",
        "\n",
        "    new_observation, reward, done, info = env.step(action)\n",
        "    r += reward\n",
        "    if done:\n",
        "      new_state = None\n",
        "    else:\n",
        "      new_state = np.float32(new_observation)\n",
        "      # push this exp in ram\n",
        "      ram.add(state, action, reward, new_state)\n",
        "\n",
        "    observation = new_observation\n",
        "    print( end='\\rEpisode : {}, reward : {}'.format(_ep, r))\n",
        "\n",
        "    # perform optimization\n",
        "    trainer.optimize()\n",
        "    if done:\n",
        "      break\n",
        "\n",
        "  gc.collect()\n",
        "\n",
        "  total_rewards_per_ep.append(r)\n",
        "  num_steps_per_ep.append(i+1)\n",
        "\n",
        "  if _ep==0 or (_ep+1)%100 == 0:\n",
        "    trainer.save_models(_ep, './results/Models')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hB_q2b8f9ixN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gc.collect()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IGOLOVCVeDo5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "np.savetxt(\"results/1_steps_per_ep.txt\", num_steps_per_ep)\n",
        "np.savetxt(\"results/1_rewards.txt\", total_rewards_per_ep)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FNz9BBgRL8O-",
        "colab_type": "text"
      },
      "source": [
        "## **Leave-No-Trace experiment with SAC**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IG8jIbIv9Ftt",
        "colab_type": "text"
      },
      "source": [
        "First, we create the method described in the pseudo-code above, \"Leave-No-Trace Framework\" part. We coded the method with a variant. We do not program hard reset after N failures of the reset policy as described in the paper. We just use the hard reset programmed in the environment (reset when falling, etc). So alternatly, we train the forward policy and the reset policy. And, during the forward agent train, we use early aborts to avoid taking actions that will lead to failures. The ealy aborts are programed in that way:\n",
        "\n",
        "- At each iteration we use the reset critic to evaluate the Q value of the action. If this Q-value is higher than a given threshold (Q_min), the action is considered to be safe, otherwise, we use the reset actor policy to take an action and avoid bad actions.\n",
        "<br><br>\n",
        "\n",
        "\n",
        "Two \"parameters\" must then be taken into account in our experiments : the Q_min threshold, and the reset reward."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5psHo0XTrrRD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def train_with_reset_agent(reset_reward_fn, q_min, save_model_folder):\n",
        "  global total_rewards_per_ep\n",
        "  global num_steps_per_ep\n",
        "  for _ep in range(MAX_EPISODES):\n",
        "    observation = env.reset()\n",
        "    print()\n",
        "    r = 0\n",
        "    i=0\n",
        "    while i < MAX_STEPS:\n",
        "      i += 1\n",
        "      \n",
        "      state = np.float32(observation)\n",
        "\n",
        "      action = forward_trainer.get_exploration_action(state)\n",
        "\n",
        "      q_value = reset_trainer.critic.forward(Variable(torch.from_numpy(np.float32([state]))), Variable(torch.from_numpy(np.float32([action]))))\n",
        "      if q_value < q_min: #### early abort, we switch to reset policy (safe mode)\n",
        "        action = reset_trainer.get_exploitation_action(state)\n",
        "        i -= 1\n",
        "      else:\n",
        "        env.render()\n",
        "\n",
        "      new_observation, reward, done, info = env.step(action)\n",
        "      r += reward\n",
        "      if done:\n",
        "        new_state = None\n",
        "      else:\n",
        "        new_state = np.float32(new_observation)\n",
        "        # push this exp in ram\n",
        "        forward_ram.add(state, action, reward, new_state)\n",
        "\n",
        "      observation = new_observation\n",
        "      print( end='\\rEpisode : {}, reward : {}'.format(_ep, r))\n",
        "\n",
        "      # perform optimization\n",
        "      forward_trainer.optimize()\n",
        "      if done:\n",
        "        break\n",
        "    observation = env.reset()\n",
        "    for i in range(MAX_STEPS):\n",
        "      state = np.float32(observation)\n",
        "      action = reset_trainer.get_exploration_action(state)\n",
        "      new_observation, _, done, info = env.step(action)\n",
        "      reward = reset_reward_fn(new_observation)\n",
        "\n",
        "      if done :\n",
        "        new_state = None\n",
        "      else:\n",
        "        new_state = np.float32(new_observation)\n",
        "        reset_ram.add(state, action, reward, new_state)\n",
        "\n",
        "      observation = new_observation\n",
        "\n",
        "      reset_trainer.optimize()\n",
        "      if done :\n",
        "        break\n",
        "\n",
        "    gc.collect()\n",
        "\n",
        "    total_rewards_per_ep.append(r)\n",
        "    num_steps_per_ep.append(i+1)\n",
        "\n",
        "    if _ep==0 or (_ep+1)%100 == 0:\n",
        "      forward_trainer.save_models(_ep, save_model_folder)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AvEGLueMAOTh",
        "colab_type": "text"
      },
      "source": [
        "We define two reset reward functions. \n",
        "- In the first reward function, we penalize the agent when it takes actions that puts him in a state that is very far away from the sate where the hull is horizontal and one leg is straight and touches the ground. We consider that position to be an equilibrium state (not the best necessarily).\n",
        "- In the second reward function, we consider the equilibrium state to be the one where the hull is horizontal and the body is higher than a certain threshold.\n",
        "\n",
        "In the two cases, the maximum reward the reset agent can get is 0."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EXr4nZulNsn-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "def reset_reward_fn_1(state):\n",
        "  r0 = float(F.smooth_l1_loss(torch.Tensor(np.array(state)[[0]]), torch.Tensor(np.array(inital_state)[[0]])))\n",
        "  r1 = float(F.smooth_l1_loss(torch.Tensor(np.array(state)[[9,11,13]]), torch.Tensor(np.array(inital_state)[[9,11,13]])))\n",
        "  r2 = float(F.smooth_l1_loss(torch.Tensor(np.array(state)[[4,6,8]]), torch.Tensor(np.array(inital_state)[[4,6,8]])))\n",
        "  r = - (r0 + min(r1, r2))\n",
        "  return r\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "knpfSpDkTNuS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "def reset_reward_fn_2(state):\n",
        "  r = -5.0 * (env.hull.position[1]< 4.7)\n",
        "  r += -5.0 * np.abs(state[0])\n",
        "  return r"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x0H8g-wDBrCR",
        "colab_type": "text"
      },
      "source": [
        "We build a helper function, that creates the environment, the agents, the trainers, and the buffers."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nUdj2qOIzS8U",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def build_env_agents(video_folder):\n",
        "  env = wrap_env(gym.make('BipedalWalker-v3'), MAX_STEPS, video_folder=video_folder, video_callable=video_callable )\n",
        "  inital_state = env.reset()\n",
        "\n",
        "  forward_ram = buffer.MemoryBuffer(MAX_BUFFER)\n",
        "  reset_ram = buffer.MemoryBuffer(MAX_BUFFER)\n",
        "\n",
        "  forward_agent = create_agent(S_DIM, A_DIM, A_MAX)\n",
        "  reset_agent = create_agent(S_DIM, A_DIM, A_MAX)\n",
        "\n",
        "  forward_trainer = Trainer(forward_agent, forward_ram)\n",
        "  reset_trainer = Trainer(reset_agent, reset_ram)\n",
        "\n",
        "  return env, inital_state, forward_ram, reset_ram, forward_agent, reset_agent, forward_trainer, reset_trainer"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cShx3l0f0TEV",
        "colab_type": "text"
      },
      "source": [
        "### **Equilibrium state: horizontal hull and a least one straight leg;   *q*\\_min=-5.0**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5nbll32qDLZa",
        "colab_type": "text"
      },
      "source": [
        "In this section, we use the first reset reward function with Q_min threshold at -5.0"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dkmxEtzoz_hX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "env, inital_state, forward_ram, reset_ram, forward_agent, reset_agent, forward_trainer, reset_trainer = build_env_agents(\"results/video2\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i5nS5m1O0PXe",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "q_min = -5.0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LwlYrU2eyBen",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "total_rewards_per_ep = []\n",
        "num_steps_per_ep = []\n",
        "\n",
        "train_with_reset_agent(reset_reward_fn=reset_reward_fn_1, q_min = q_min , save_model_folder= \"results/Models2\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yqvQebSUyT2x",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "np.savetxt(\"results/with_reset_2_rewards.txt\", np.array(total_rewards_per_ep))\n",
        "np.savetxt(\"results/with_reset_2_steps_per_ep.txt\", np.array(num_steps_per_ep))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XSuMfRaO5ZH_",
        "colab_type": "text"
      },
      "source": [
        "### **Equilibrium state: horizontal hull and body higher than a certain level;   q\\_min=-50.0**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RAcloCmKDW1V",
        "colab_type": "text"
      },
      "source": [
        "In this section, we use the second reset reward function and a Q_min threshold at -50.0"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S_YVDx2pKmAV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "env, inital_state, forward_ram, reset_ram, forward_agent, reset_agent, forward_trainer, reset_trainer = build_env_agents(\"results/video4\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uvubxTLB37M-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "q_min = -50.0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QLLZQM5tLt7B",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "total_rewards_per_ep = []\n",
        "num_steps_per_ep = []\n",
        "\n",
        "train_with_reset_agent(reset_reward_fn=reset_reward_fn_2, q_min = q_min , save_model_folder= \"results/Models4\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0IMx1hZxFzfq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "np.savetxt(\"results/with_reset_4_rewards.txt\", np.array(total_rewards_per_ep))\n",
        "np.savetxt(\"results/with_reset_4_steps_per_ep.txt\", np.array(num_steps_per_ep))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jK2et1gtL0S2",
        "colab_type": "text"
      },
      "source": [
        "# **RESULTS**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ENuwaPhXRDjS",
        "colab_type": "text"
      },
      "source": [
        "We reload the different measures we have saved previsously."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JO4CV4p_EDGq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "## Results from the simple SAC Framework experiment (number of steps per episode and total rewards per episode)\n",
        "steps_per_ep_1 = np.loadtxt(\"results/1_steps_per_ep.txt\")\n",
        "rewards_1 = np.loadtxt(\"results/1_rewards.txt\")\n",
        "\n",
        "## results from the Leave-No-Trace Framework using Q_min =-5 and the first reset reward function\n",
        "with_reset_2_steps_per_ep = np.loadtxt(\"results/with_reset_2_steps_per_ep.txt\")\n",
        "with_reset_2_rewards = np.loadtxt(\"results/with_reset_2_rewards.txt\")\n",
        "\n",
        "## results from the Leave-No-Trace Framework using Q_min =-50 and the second reset reward function\n",
        "with_reset_4_steps_per_ep = np.loadtxt(\"results/with_reset_4_steps_per_ep.txt\")\n",
        "with_reset_4_rewards = np.loadtxt(\"results/with_reset_4_rewards.txt\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OLeOjiCORrzf",
        "colab_type": "text"
      },
      "source": [
        "#### Comparative study"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8mLuuXuCEkys",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "f, (ax1, ax2) = plt.subplots(1, 2,figsize=(12,4))\n",
        "ax1.plot(with_reset_2_steps_per_ep)\n",
        "ax1.plot(steps_per_ep_1)\n",
        "\n",
        "ax1.set_title('Number of steps per episode')\n",
        "ax1.set_xlabel(\"Episode\")\n",
        "ax1.set_ylabel(\"steps\")\n",
        "\n",
        "ax2.plot(with_reset_2_rewards)\n",
        "ax2.plot(rewards_1)\n",
        "ax2.set_xlabel(\"Episode\")\n",
        "ax2.set_ylabel(\"steps\")\n",
        "\n",
        "_=ax2.set_title('Total rewards per episode')\n",
        "\n",
        "_=f.legend([\"simple SAC\", \"SAC with reset agent\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7_JvxhqPSH5H",
        "colab_type": "text"
      },
      "source": [
        "**Comments**\n",
        "<br>\n",
        "We can clearly observe that the Leave-No-Trace framework allowed our agent to run much more steps without failing. It is even possible to get much more rewards than the simple agent alone. In this experiment, the safe action is when the agent is keeping its head horizontal and at least one leg straight and touching the ground."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Oo45h3hIE7G4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "f, (ax1, ax2) = plt.subplots(1, 2,figsize=(12,4))\n",
        "ax1.plot(with_reset_4_steps_per_ep)\n",
        "ax1.plot(steps_per_ep_1)\n",
        "\n",
        "ax1.set_title('Number of steps per episode')\n",
        "ax1.set_xlabel(\"Episode\")\n",
        "ax1.set_ylabel(\"steps\")\n",
        "\n",
        "ax2.plot(with_reset_4_rewards)\n",
        "ax2.plot(rewards_1)\n",
        "ax2.set_xlabel(\"Episode\")\n",
        "ax1.set_ylabel(\"steps\")\n",
        "\n",
        "_=ax2.set_title('Total rewards per episode')\n",
        "_=f.legend([\"simple SAC\", \"SAC with reset agent\"])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ueowW9tmUQ0R",
        "colab_type": "text"
      },
      "source": [
        "**Comments**\n",
        "<br>\n",
        "In this experiment, the equilibrium state is considered to be the state where the agent head is horizontal and hi body is higher than a certain level. The result are obtained with a Q_min=-50 but we tried with a Q_min=-5 and obtained similar results. The way we defined the \"good\" state doesn't work as in the previous case.\n",
        "<br>\n",
        "However, we can see that the training is much more longer per episode than it is with a simple SAC without reset agent."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gh90bUOsbzwN",
        "colab_type": "text"
      },
      "source": [
        "#### Visualisation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nm5pstMxNuAa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "show_video(\"results/video/openaigym.video.0.3678.video001099.mp4\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5DzjZkHvN5k1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "show_video(\"results/video2/openaigym.video.0.2266.video001600.mp4\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w3zskGkRb-un",
        "colab_type": "text"
      },
      "source": [
        "**Comments**\n",
        "<br>\n",
        "The first video shows, the agent after 1100 episodes in the SAC-Only framework. It is one of the longer sequences the agent has been able to hold when we look at all the videos monitored. It lasts only 3 seconds.\n",
        "<br>\n",
        "The second video shows the forward agent in the Leave-No-Trace Framework after 800 episodes (Be aware about the number of episodes; looking at the name of the video it seems like episode 1600 but it is episode 800 in fact; this is due to the fact that we train both agents in the same environment, making the environement step incrementation going twice faster than it should.)\n",
        "<br><br>\n",
        "When we look at the second video, we can observe that the agent tries to go forward but in a very safe way. It restricts itself in the actions it takes. It is important then to well define the equilibrium state (as we've seen it before), otherwise the agent could be too cautious and finally forget the inital goal which is here to go forward as fast as possible.\n",
        "<br><br>\n",
        "**Does the reset agent impact considerably the main agent?**<br>\n",
        "The answer to this question is rather yes. This is in part what we've explained in the previous paragraph. For example, if we pay attention to the way our agent moves, it tries to do it by keeping at least one leg very straight (corresponding exactly to what we defined in the reset reward function)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HSzrL2nlfMBq",
        "colab_type": "text"
      },
      "source": [
        "# **Conclusion**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "38cSEWN0fOtw",
        "colab_type": "text"
      },
      "source": [
        "In this work, we tried to prove the point of this [paper](https://arxiv.org/abs/1711.06782). We managed to prove that introducing a reset agent allows the main agent to run longer episodes and then reduce the number of hard resets and the cost that it implies. But more than that, we have seen that our agent learns to accomplish its task by taking \"safe\" actions. It is learning the main goal, while also learning how to be cautious about the actions it takes. For example, in our experiments, we can see that the agent while learning to move forward also learns how to stay in balance and not fall."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FgEzlBZIhK87",
        "colab_type": "text"
      },
      "source": [
        "**Experimental method**<br>\n",
        "- **Reproductibility** : In order to check our results, we run those experiments three (03) times and had about the same conclusions.\n",
        "- **HyperParameters Tuning**: Due to our lack of computation power, we were limited in the amount of experiments we could run. We thought about studying the impact of the Q_min threshold. But rather than that, we prefered to focus on proving the contribution of the reset agent. In further experiments, we could try to study the influence of the Q-value threshold."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UlMjOmLui8Fb",
        "colab_type": "text"
      },
      "source": [
        "# **References**\n",
        "\n",
        "[Leave No trace: Learning to Reset for Safe and Autonomous Reinforcement Learning](https://arxiv.org/abs/1711.06782!), *by Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, Sergey Levine* \n",
        "\n",
        "[Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://arxiv.org/abs/1801.01290), *by Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine*\n",
        "\n",
        "[GYM Framework](https://gym.openai.com/), *by OpenAI*\n",
        "\n",
        "[Soft-Actor-Critic Implementation](https://github.com/vy007vikas/PyTorch-ActorCriticRL), *Vikas Yadav*\n"
      ]
    }
  ]
}