Wind_gust_alleviation_code_and_explanation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Importing Libs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here most of the libs are imported, this helps to ensure that all the packages are corrected installed.\n",
    "\n",
    "_gametoy_ is our file containing **FooEnv** which is the simulator itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gametoy import FooEnv "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import gym\n",
    "import numpy as np\n",
    "from collections import deque\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.optimizers import Adam\n",
    "import time\n",
    "import os\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Initialize RL Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These here are the key parameters when working with this RL technique. It will define: the learning rate, how fast does it changes? Memory size, how much time steps will be in our memory? The exploration decay, how long we will priorize random solutions over the ones learned? Check the pdf of the work for more information about these parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#ENV_NAME = \"\"\n",
    "\n",
    "GAMMA = 0.95\n",
    "LEARNING_RATE = 0.001\n",
    "\n",
    "MEMORY_SIZE = 10000\n",
    "BATCH_SIZE = 20\n",
    "\n",
    "EXPLORATION_MAX = 1.0\n",
    "EXPLORATION_MIN = 0.01\n",
    "EXPLORATION_DECAY = 0.9999"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we will initializate our simulator. The first number is the reward used, the second how we are changing the main parameters of the aircraft and the third in this case is the velocity, check the function code for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env = FooEnv(6,5,100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next class defines really important functions.\n",
    "- **init**: defines the neural network that will be used, here it is recommended for the user to personalizate it by himself/herself. Try new possibilities, simpler or even far more complexes networks.\n",
    "- rembember: saves the the step and all relevant information for the technique called **Replay Memory**.\n",
    "- **act** : exploration x explotation, it in this point that the code decides, during training, if it should take a random action or the best learned action.\n",
    "- play : same as act, but ensures that the Neural Network will not take random actions, only the best one learned.\n",
    "- **experience replay**: applies the **Replay memory** memory to our neural network, it's this function that trains the neural network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class DQNSolver:\n",
    "\n",
    "    def __init__(self, observation_space, action_space):\n",
    "        self.exploration_rate = EXPLORATION_MAX\n",
    "\n",
    "        self.action_space = action_space\n",
    "        self.memory = deque(maxlen=MEMORY_SIZE)\n",
    "\n",
    "        self.model = Sequential()\n",
    "        self.model.add(Dense(9, input_shape=(observation_space,), activation=\"relu\"))\n",
    "        for i in range(14):\n",
    "            self.model.add(Dense(18, activation=\"relu\"))\n",
    "        self.model.add(Dense(9, activation=\"relu\"))\n",
    "        self.model.add(Dense(self.action_space, activation=\"linear\"))\n",
    "        self.model.compile(loss=\"mse\", optimizer=Adam(lr=LEARNING_RATE))\n",
    "\n",
    "\n",
    "    def remember(self, state, action, reward, next_state, done):\n",
    "        self.memory.append((state, action, reward, next_state, done))\n",
    "\n",
    "    def act(self, state):\n",
    "        if np.random.rand() < self.exploration_rate:\n",
    "            return random.randrange(self.action_space)\n",
    "        q_values = self.model.predict(state)\n",
    "        return np.argmax(q_values[0])\n",
    "\n",
    "    def play(self, state):\n",
    "        q_values = self.model.predict(state)\n",
    "        return np.argmax(q_values[0])\n",
    "    \n",
    "    def experience_replay(self):\n",
    "        if len(self.memory) < BATCH_SIZE:\n",
    "            return\n",
    "        batch = random.sample(self.memory, BATCH_SIZE)\n",
    "        for state, action, reward, state_next, terminal in batch:\n",
    "            q_update = reward\n",
    "            if not terminal:\n",
    "                q_update = (reward + GAMMA * np.amax(self.model.predict(state_next)[0]))\n",
    "            q_values = self.model.predict(state)\n",
    "            q_values[0][action] = q_update\n",
    "            self.model.fit(state, q_values, verbose=0)\n",
    "        self.exploration_rate *= EXPLORATION_DECAY\n",
    "        self.exploration_rate = max(EXPLORATION_MIN, self.exploration_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Score Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class scorelog:\n",
    "\n",
    "    def __init__(self, number_runs, logfile_path):\n",
    "        self.n = number_runs\n",
    "        self.i = 0\n",
    "        self.mean_scores = np.zeros(self.n)\n",
    "        self.mean_score = 0\n",
    "        self.action_space = action_space\n",
    "        self.scores = np.zeros(self.n)\n",
    "        self.run = 0\n",
    "        self.score = 0\n",
    "        self.duration = 0\n",
    "        self.logfile_path = logfile_path\n",
    "        if not os.path.exists(logfile_path):\n",
    "            self.logFile = open(logfile_path, 'w')\n",
    "            self.logFile.write(\"Step,Mean_Reward,score,Time \\n\")\n",
    "            #self.logFile.write(\"Step,Episode,Mean_Reward,score,Time \\n\")\n",
    "        else:\n",
    "            self.logFile = open(logfile_path,'a')\n",
    "            \n",
    "    def log(self, score, run,duration):\n",
    "        self.i += 1\n",
    "        self.run = run\n",
    "        self.score = score\n",
    "        self.duration = duration\n",
    "        self.mean_score = (1.0/self.i)*(score- self.mean_score)\n",
    "        self.mean_scores[self.i] = self.mean_score\n",
    "        self.scores[self.i] = score\n",
    "        \n",
    "    def logwrite(self):\n",
    "        self.logFile.write(\"%d,%.3f,%.3f,%.3f \\n\" % (self.run, self.mean_score, self.score, self.duration))\n",
    "        \n",
    "    def logclose(self):\n",
    "        self.logFile.close()\n",
    "        self.logFile = open(self.logfile_path,'a')\n",
    "        \n",
    "    def scoreplot(self):\n",
    "        plt.plot(self.mean_scores[:self.i])\n",
    "        plt.show()\n",
    "        plt.plot(self.scores[:self.i])\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Play and tests functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These functions were created to make it easier to play a single game and see its results or plot it into a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def play_one(Filename):\n",
    "    #The agent will play one game and plot the results in the specified file.\n",
    "        step = 0\n",
    "        state = env.reset()\n",
    "        env.zrefer()\n",
    "        beta = env.beta\n",
    "        state = np.reshape(np.append(beta,state), [1, observation_space])\n",
    "        while True:\n",
    "            step += 1\n",
    "            action = dqn_solver.play(state)\n",
    "            state_next, reward, terminal, info = env.step(action)\n",
    "            reward = reward# if not terminal else -reward\n",
    "            state_next = np.reshape(np.append(env.beta,state_next), [1, observation_space])\n",
    "            dqn_solver.remember(state, action, reward, state_next, terminal)\n",
    "            state = state_next\n",
    "            if step>10000:\n",
    "                break\n",
    "        env.plot(Filename)\n",
    "        print('Score:',env.score)        \n",
    "\n",
    "def play_render():\n",
    "        #The agent will play one game and plot the results inside the jupyter notebook.\n",
    "        step = 0\n",
    "        state = env.reset()\n",
    "        env.zrefer()\n",
    "        beta = env.beta\n",
    "        state = np.reshape(np.append(beta,state), [1, observation_space])\n",
    "        while True:\n",
    "            step += 1\n",
    "            action = dqn_solver.play(state)\n",
    "            state_next, reward, terminal, info = env.step(action)\n",
    "            reward = reward# if not terminal else -reward\n",
    "            state_next = np.reshape(np.append(env.beta,state_next), [1, observation_space])\n",
    "            dqn_solver.remember(state, action, reward, state_next, terminal)\n",
    "            state = state_next\n",
    "            if step>10000:\n",
    "                break\n",
    "        env.render()\n",
    "        print('Score:',env.score)\n",
    "\n",
    "def play_dumb():\n",
    "    #The game will be played with only \"0s\" actions.\n",
    "    step = 0\n",
    "    state = env.reset()\n",
    "    env.zrefer()\n",
    "    beta = env.beta\n",
    "    state = np.reshape(np.append(beta,state), [1, observation_space])\n",
    "    while True:\n",
    "        step += 1\n",
    "        action = 0\n",
    "        state_next, reward, terminal, info = env.step(action)\n",
    "        reward = reward# if not terminal else -reward\n",
    "        state_next = np.reshape(np.append(env.beta,state_next), [1, observation_space])\n",
    "        dqn_solver.remember(state, action, reward, state_next, terminal)\n",
    "        state = state_next\n",
    "        if step>10000:\n",
    "            break\n",
    "    env.render()\n",
    "    print('Score:',env.score)\n",
    "        \n",
    "def play_vel(Filename):\n",
    "        #The agent will play one game and plot the results in the specified file, containing the velocity on its name.\n",
    "        step = 0\n",
    "        state = env.reset()\n",
    "        env.zrefer()\n",
    "        beta = env.beta\n",
    "        state = np.reshape(np.append(beta,state), [1, observation_space])\n",
    "        while True:\n",
    "            step += 1\n",
    "            action = dqn_solver.play(state)\n",
    "            state_next, reward, terminal, info = env.step(action)\n",
    "            reward = reward# if not terminal else -reward\n",
    "            state_next = np.reshape(np.append(env.beta,state_next), [1, observation_space])\n",
    "            dqn_solver.remember(state, action, reward, state_next, terminal)\n",
    "            state = state_next\n",
    "            if step>10000:\n",
    "                break\n",
    "        env.plot(Filename+str(env.V))\n",
    "        print('Score:',env.score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preparing files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "observation_space = env.observation_space+1 #defines the observation space\n",
    "action_space  = env.action_space # receives the action space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "run = 0 #initialize the counting to 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dqn_solver = DQNSolver(observation_space, action_space) #generates the neural network\n",
    "dqn_solver.model.summary() #plots the neural network's structure generated"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initializes some values and states where the files will be saved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "DQN_SAVE = \"dqn_PCC_test_\"\n",
    "filepath = 'NN/'\n",
    "if not os.path.exists(filepath):\n",
    "    os.makedirs(filepath)\n",
    "DQN_SAVE_FILE = filepath + '/' + DQN_SAVE + '0.h5'\n",
    "counter_dqn = 0\n",
    "dqn_solver.model.save(DQN_SAVE_FILE)\n",
    "\n",
    "logfile = filepath+'/log.txt'\n",
    "scoreLog = scorelog(100000,logfile)\n",
    "\n",
    "duration = 0\n",
    "counter_play = 0\n",
    "\n",
    "filename_play = filepath + '/' + 'Gameplay/' \n",
    "if not os.path.exists(filename_play):\n",
    "    os.makedirs(filename_play)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we have the code responsible for training our Neural Network and keeping the log, it is important to state that it is an infinite loop which is broken by an _if_ condition, if you erases it, it will keep running foreverrrrrr!!!! So, pay attention to this.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "while True:\n",
    "    start_time = time.time()\n",
    "    scoreLog.log(env.score, run, duration)\n",
    "    scoreLog.logwrite()\n",
    "    scoreLog.logclose()\n",
    "    if (run%5 == 0):\n",
    "        #plots in the jupyter notebook the result each 5 games\n",
    "        env.render()\n",
    "    if (run%1 == 0):\n",
    "        dqn_solver.model.save(DQN_SAVE_FILE)\n",
    "        counter_dqn +=1 \n",
    "        DQN_SAVE_FILE = filepath + '/' + DQN_SAVE + str(counter_dqn) +'.h5'\n",
    "        FILENAME = filepath + '/' + DQN_SAVE + str(counter_dqn)\n",
    "        env.plot(FILENAME)\n",
    "        filename_play = filepath + '/' + 'Gameplay/' + DQN_SAVE + 'gameplay_'+ str(counter_play)\n",
    "        play_one(filename_play) #saves plots of playing mode\n",
    "        counter_play +=1\n",
    "        \n",
    "    run += 1\n",
    "    state = env.reset()\n",
    "    beta = env.beta\n",
    "    state = np.reshape(np.append(beta,state), [1, observation_space])\n",
    "    step = 0\n",
    "    if run > 100000:\n",
    "        break\n",
    "        \n",
    "   \n",
    "    while True:\n",
    "        step += 1\n",
    "        #env.render()\n",
    "        action = dqn_solver.act(state)\n",
    "        #action = 0\n",
    "        state_next, reward, terminal, info = env.step(action)\n",
    "        reward = reward# if not terminal else -reward\n",
    "        state_next = np.reshape(np.append(env.beta,state_next), [1, observation_space])\n",
    "        dqn_solver.remember(state, action, reward, state_next, terminal)\n",
    "        state = state_next\n",
    "        if terminal or step>1500:\n",
    "            end_time = time.time()\n",
    "            duration = end_time-start_time\n",
    "            print (\"Run: \" + str(run) + \", exploration: \" + str(dqn_solver.exploration_rate) + \", steps: \" + str(step) + \", score: \" + str(env.score), \", time:\" + str(duration))\n",
    "            #score_logger.add_score(step, run)\n",
    "            break\n",
    "        #if step%10 == 0:\n",
    "        dqn_solver.experience_replay()\n",
    "print('End')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code hereon presented is responsible for loading an already saved neural network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import load_model\n",
    "\n",
    "#DQN_SAVE = \"dqn_PCC_toy_R-06_\"\n",
    "DQN_SAVE = \"dqn_PCC_toy_R6-1_\"\n",
    "\n",
    "filepath = 'NeuralNetworks'\n",
    "\n",
    "DQN_SAVE_FILE = filepath + '/' + DQN_SAVE + '65' + '.h5'\n",
    "dqn_solver.model = load_model(DQN_SAVE_FILE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dqn_solver.model.summary() #show the structure of the loaded neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Tests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Notebook tests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These codes are responsible for testing the neural networks obtained in the wanted conditions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "env = FooEnv(6,5,80) #loads the simulator with the wanted parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "play_render() #play and plots the result in this notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Prints the values for the criteria used in this work to evaluate the AI agent performance.\n",
    "\n",
    "print('Overshoot%',100*np.max(np.abs(env.Z))/np.max(np.abs(env.zref)))\n",
    "\n",
    "margin = np.max(np.abs(env.Z))*.03\n",
    "final = env.Z[-2]\n",
    "t1 = np.nonzero( (env.Z < final - np.abs(margin) ) | ( env.Z > final + np.abs(margin) ) )[0][-2]\n",
    "print('Time to stabilize', t1, 'ms' )\n",
    "\n",
    "margin = np.max(np.abs(env.zref))*.03\n",
    "final = env.zref[-2]\n",
    "t2 = np.nonzero( (env.zref < final - np.abs(margin) ) | ( env.zref > final + np.abs(margin) ) )[0][-2]\n",
    "print('No control time to stabilize',t2, 'ms' )\n",
    "\n",
    "print('Relative time to stabilize',t1/t2*100,)\n",
    "\n",
    "\n",
    "print('Delta Max%', 100*(np.max(env.Z)-np.min(env.Z)) / (np.max(env.zref)-np.min(env.zref))  )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "play_dumb() #plays without taking any action."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Log Loading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "logfile_path = filepath + '/' + 'log.txt'\n",
    "\n",
    "Log = pd.read_csv(logfile_path)\n",
    "Log.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(Log.Episode[5:])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# File saving tests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Play the tests that plots the result in a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "filename_play = filepath + '/' + 'Gameplay/' + DQN_SAVE + 'gameplay_'+ 'test_'\n",
    "play_one(filename_play)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "filename_play = filepath + '/' + 'Test_vel/' \n",
    "if not os.path.exists(filename_play):\n",
    "    os.makedirs(filename_play)\n",
    "filename_test = filepath + '/' + 'Test_vel/' + DQN_SAVE + 'Vel_'\n",
    "\n",
    "env = FooEnv(6,1)\n",
    "for i in range(10):\n",
    "    play_vel(filename_test)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}