CartPole

Buidling neural network to solve OpenAI's cart pole balancing environment.

CartPole.mov

Description

This environment corresponds to the version of the cart-pole problem described by Barto, Sutton, and Anderson in "Neuronlike Adaptive Elements That Can Solve Difficult Learning Control Problem". A pole is attached by an un-actuated joint to a cart, which moves along a frictionless track. The pendulum is placed upright on the cart and the goal is to balance the pole by applying forces in the left and right direction on the cart¹

Rewards

Since the goal is to keep the pole upright for as long as possible, a reward of +1 for every step taken, including the termination step, is allotted. The threshold for rewards is 50.

Starting State

All observations are assigned a uniformly random value in (0.0, 2.0)

Computational Graph for the Deep Reinforcement learning model.

• It has 6 fully connected layers followed by dropout layers with categorical cross entropy loss and Adam optimizer with a learning rate of 1e-3.

Accuracy

(The accuracy here isnt that great since the model was only run for 30 epochs, if trained for longer the accuracy will get better.)

Loss

Getting Started

To run CartPole locally perform the following steps (requires Python 3.7+):

1. git clone https://github.com/Suraj-Bhor/CartPole.git
2. cd Cartpole
3. pip install -r requirements.txt
4. python cartpole.py

Visualize graphs

To visualize graphs the model saves the tensorboard checkpoints to /tmp/tflearn_logs/. Once the model is trained you can visualize the results buy issuing the following command.

• tensorboard --logdir='/tmp/tflearn_logs'

Acknowledgments

1. Sentdex
2. Gym Library
3. Tensorboard
4. Spinning up OpenAI

Footnotes

1. https://github.com/openai/gym/blob/master/gym/envs/classic_control/cartpole.py ↩