This repository contains implementations of deep reinforcement learning algorithms on various environments.
To install the required packages, run the following commands in the terminal:
git clone https://github.com/olibridge01/DeepRL.git
cd DeepRL
pip install -r requirements.txt
DeepRL/
├── agents/
│ └── (RL agents)
├── figs/
│ ├── gifs/
│ │ └── (gifs of agent behaviour)
│ └── plots/
│ └── (plots e.g. scores/episode durations)
├── misc/
│ └── (miscellaneous resources e.g. fonts)
├── networks/
│ └── (neural networks used for deep RL algorithms)
├── saves/
│ └── (saved weights for trained agent networks)
├── utils/
│ └── (utility functions e.g. plotters, agent config etc.)
├── README.md
├── requirements.txt
└── run.py (script to train agents/plot gifs etc.)
Below are some examples of the RL algorithms implemented.
- Random agent: Agent selects moves randomly from the action space. This is not a very useful agent but is used to show how the agent behaves in the following environments by acting at random.
- Q-learning agent: Agent learns with an epsilon-greedy policy via a table of action value function values - Watkins, 1989.
- DQN agent: Agent learns with an epsilon-greedy policy via a feedforward neural network with an Experience Replay buffer - Mnih et al, 2013.
- DDQN agent: Agent learns with Deep Double Q-learning - van Hasselt et al, 2015.
Below is a brief overview of the background theory for the algorithms used.
The action value function
where the return
The action value function satisfies the Bellman equation:
The optimal action value function
For a deterministic environment - where each state/action pair
where
In Q-Learning, the agent learns an estimate of the optimal action value function
where
In Deep Q-Learning (DQN), the agent learns an estimate of the optimal action value function torch.nn.SmoothL1Loss()
(Huber) loss function was used to train the network.
In both the cases above, performing these updates alone is known as a greedy policy - the agent chooses the action which maximises the action value function at each step. This typically explores the state/action space poorly, so instead an epsilon-greedy policy is used - the agent chooses a random action with probability
The RL algorithms described above were implemented on a series of OpenAI's gymnasium
environments (see below). All results are shown alongside a random agent for comparison.
In the following examples, the agent receives physical properties of the system as the current state e.g. player position/velocity etc. This typically makes it simpler for the agent to learn an optimal policy.
For the deep RL algorithms, a multi-layer perceptron (MLP) neural network was used to approximate the action value function Cartpole-v1
environment is shown in the corresponding section below.
The aim is to keep the pole balanced by only moving the cart left or right. The agent is given a reward of +1
for every timestep it keeps the pole upright. An episode ends if the pole deviates by more than ±12°
from the vertical, or the cart leaves the frame. An episode is a success if it keeps the pole upright for 500
timesteps.
The agent receives the following observations:
p
: The position of the cart on the track.v
: The velocity of the cart.a
: The angle of the pole from the vertical.av
: The pole's angular velocity.
The neural network takes these observations as a 4-dimensional input, and outputs the action value function
![](https://private-user-images.githubusercontent.com/86416298/297044015-3f51fcc7-9010-4f5a-9787-52daea2c4118.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYxNTExOTMsIm5iZiI6MTcyNjE1MDg5MywicGF0aCI6Ii84NjQxNjI5OC8yOTcwNDQwMTUtM2Y1MWZjYzctOTAxMC00ZjVhLTk3ODctNTJkYWVhMmM0MTE4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEyVDE0MjEzM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTJhMDZjOTI1ZTkzNDI5Y2U5Zjk5ZjdjZjYwMzk4ZDk5Yjg0YmZiMDVjZmJmYjg1OTBmMjkyNTZjNGE3ZGE2OGYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.l5HhJduCigph5O-A7uA5IZrv2T01Q1ksH7wlzAT8Q_Y)
A plot of the scores as a function of training episodes is shown below for DQN, DDQN and Q-learning. The agents were each run 6
times and the mean rolling score over the last 10
episodes is shown.
In general DQN and DDQN were able to solve Cartpole-v1
in slightly fewer episodes than Q-Learning. Q-Learning performs reasonably well here due to the relatively small state/action space, but will struggle to scale to larger environments.
The aim is to land the Lunar Lander between the flags. The agent is rewarded based on how close the lander is to the pad, how fast it is moving, and how little it has to use its engines. It is also given a reward of -100
(+100
) for crashing (landing safely) on the moon. An episode terminates if the lander crashes, leaves the frame or comes to rest on the ground. An episode is a success if the agent's cumulative reward is over 200
. Gifs of the agent's behaviour are shown below.
The aim is to accelerate the car up the hill to the flag. The agent is given a reward of 200
). An episode is a success if the car reaches the flag in under 200
timesteps. Gifs of the agent's behaviour are shown below.
Here, rather than receiving physical properties of the current state (e.g. position, velocity etc.), the agent receives only a screen image (a 2D array of coloured pixels) as the current state. In order to handle this, a few modifications were made to the DQN/DDQN algorithms implemented above. These are listed below, and are described in more detail in Mnih et al, 2013 where the agent was trained to play Atari games.
- The action value function was approximated using a convolutional neural network (CNN) (with some extra fully-connected layers on the end) rather than an MLP. This allows the agent to learn features from the screen image (a diagram of the architecture is shown in the following section).
- The screen image was preprocessed before being passed into the network. The images were converted to a single channel (grayscale) and cropped to
84x84
. Each observation was stacked with the previous3
frames to give an input tensor of shape(4,84,84)
. - At each step,
4
frames were skipped and the action was repeated for each of these frames.
The aim is to achieve the highest score possible by racing a car around a racetrack. The car accumulates a reward of -0.1
per frame, and a reward of +1000/N
for every racetrack tile visited, where N
is the number of tiles on the whole track.
![](https://private-user-images.githubusercontent.com/86416298/297042124-692103b6-7200-4a33-8ad7-fcb9f245bf08.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYxNTExOTMsIm5iZiI6MTcyNjE1MDg5MywicGF0aCI6Ii84NjQxNjI5OC8yOTcwNDIxMjQtNjkyMTAzYjYtNzIwMC00YTMzLThhZDctZmNiOWYyNDViZjA4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEyVDE0MjEzM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWVjMWUzNjBlZDdhZTQ0Nzk5ODQ0NDNlMWM5M2RlNzkyN2I2NGVkYzIyMWIyNjg0Zjg3MmFlNjZlMDhlZGNkZTImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.d_knU3isqi0TP5QftM8aOe1xgYkj06tBMYA57XpwS8Q)
A plot of the rolling scores (100
-episode window) as a function of training episodes for the DQN algorithm is shown below. The score clearly improves when the agent learns how to drive the car round the track effectively.
Oli Bridge ([email protected])
Feel free to explore the different directories, and please reach out if you have any questions or suggestions. 🚀