Skip to content

Reinforcement Learning using Policy Gradient to solve OpenAI Gym games

Notifications You must be signed in to change notification settings

gabrielgarza/openai-gym-policy-gradient

Repository files navigation

Reinforcement Learning with Policy Gradient

Uses a 3 layer neural network as the policy network

The idea is to create a deep policy network that is intelligent enough to generalize to most games in OpenAI's Gym.

  1. To run this code first install OpenAI's Gym: https://github.com/openai/gym

  2. Download this repo and run python run_carpole.py to run the agent (or any other game in this repo, like python run_lunarlander.py) and see it improve over time.

  3. To run a Box2D game like LunarLander you have to install the Box2D Physics engine: pip install -e '.[box2d]'

Lunar Lander

Reward for moving from the top of the screen to landing pad and zero speed is about 100..140 points. If lander moves away from landing pad it loses reward back. Episode finishes if the lander crashes or comes to rest, receiving additional -100 or +100 points. Each leg ground contact is +10. Firing main engine is -0.3 points each frame. Solved is 200 points. Landing outside landing pad is possible and fuel is infinite.

Initially, the agent is as good as randomly picking the next action: pg-ll-initial

After several hundred episodes, the agent starts learning how to fly and hover around: pg-ll-learning_to_fly

Finally after about 3K episodes the agent can land pretty well: pg-ll-landings

Cartpole

A pole is attached by an un-actuated joint to a cart, which moves along a frictionless track. The system is controlled by applying a force of +1 or -1 to the cart. The pendulum starts upright, and the goal is to prevent it from falling over. A reward of +1 is provided for every timestep that the pole remains upright. The episode ends when the pole is more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center.

Initially, the agent is quite dumb, but it's exploring the state/action/reward space: Untrained

As more episodes go by, it starts to get better by learning from experience (using reward guided loss): pg-cartpole-2

Eventually, the agent masters the game (trained on my Macbook Pro for ~10 minutes): pg-cartpole-trained

After 297 episodes the agent scored 617,332!

record

About

Reinforcement Learning using Policy Gradient to solve OpenAI Gym games

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages