Reinforcement Learning

SSIM-Sim (Youtrack)   •  SNOW (Website)   •  Reinforcement-Learning-Research-Project (GitHub)   •  Reinforcement-Learning-Research-Project (Wiki)   •  isabelleysseric (Docker)   •  User: isabelleeysseric (Fuel Ignition Gazebo)   •  Models: NORLAB (Fuel Ignition Gazebo)   •  NORLAB (Laboratoire de recherche)   •  GLO-4001-introduction-a-la-robotique-mobile (Cours)

isabelleysseric (GitHub)   •  isabelleysseric.com (Portfolio)   •  isabelle-eysseric (LinkedIn)

Hello, during my research on simulators I found some articles and books that helped me in my work. I share my sources with you in the hope that it helps you too. If you have other interesting resources to share on the subject, do not hesitate.

And below you will find a section for troubleshooting with the installation of Gym-Ignition.

Figure: PacMan game and Reinforcement Learning problem

Articles on RL

• Learning to walk in minutes using massively parallel Deep Reinforcement learning (30 Oct 2021) by Nikita Rudin, David Hoeller, Philipp Reist, Marco Hutter
• Sim-to-Real transfer in Deep Reinforcement Learning for robotics: a survey (8 Jul 2021) by Wenshuai Zhao, Jorge Peña Queralta, Tomi Westerlund
• Benchmarking Deep Reinforcement Learning for continous control (27 May 2016) by Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel
• Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning (25 Aug 2021) by Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, Gavriel State
• High-Fidelity Yet Fast Dynamic Models of Wheeled Mobile Robots (June 2016) by Neal Seegmiller, Alonzo Kelly
• End-to-End Reinforcement Learning for Multi-agent continous control (Dec. 2019) by Zilong Jiao, Jae Oh
• Sim2Real learning of obstacle avoidance for robotic manipulators in uncertain environments (30 September 2021) by Kefang Zhang, Jiatao Lin, Lv Bi, Tan Zhang
• Reinforcement Learning Control of a Forestry Crane Manipulator (4 March 2021) by Jennifer Andersson, Kenneth Bodin, Daniel Lindmark, Martin Servin, Erik Wallin
• Comparing Popular Simulation Environments in the Scope of Robotics and Reinforcement Learning (8 March 2021) by Marian Körber, Johann Lange, Stephan Rediske, Simon Steinmann, Roland Glück
• OpenAI Gym (5 June 2016) by Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba
• Gym-Ignition reproducible robotic simulations for Reinforcement Learning (5 June 2016) by Diego Ferigo, Silvio Traversaro, Giorgio Metta, Daniele Pucci
• Multi-task Learning for Continuous Control (3 Feb 2018) by Himani Arora, Rajath Kumar, Jason Krone, Chong Li ...

Articles on AV and SIM

*AV: Autonomous Vehicles
*SIM: Simulators

• Kilometer-scale autonomous navigation in subarctic forests: challenges and lessons learned (27 Nov 2021) by Dominic Baril, Simon-Pierre Deschênes, Olivier Gamache, Maxime Vaidis, Damien LaRocque, Johann Laconte, Vladimír Kubelka, Philippe Giguère, François Pomerleau
• A review of physics simulators for robotic applications (25 March 2021) by Jack Collins, Shelvin Chand, Anthony Vanderkop, David Howard
• CARLA an open urban driving simulator
• Simulator AirSim - High fidelity visual and physical simulation for autonomous vehicles
  ...

Articles on Physics

• Fundamentals of Applied Dynamics (Springer, 2004) by Roberto A. Tenenbaum
• Physics-based virtual environments for autonomous earthmoving and mining machinery (January 2018) by Martin Servin, Michael Brandl
• Springer Handbook of Robotics (Springer, 2016) by Bruno Siciliano, Oussama Khatib ...

Figure: OpenAI environment

Environments OpenAi Gym:

OpenAI Gym is a toolkit for developing and compar ing reinforcement learning algorithms. Here, a list of environments based on OpenAi Gym for vehicles.

• Gym-Ignition is designed for repeatable results and allows parallel or headless mode simulation while providing commo n rigid body dynami cs utilities.
• Wheeled Mobile Robot Dynamics Engine (WMRDE) is for simulation and model ing of wheeled mobi le robots (WMR) and t racked vehicles.
• Highway-Env : A collection of envi ronments for autonomous driving and tactical decision-making tasks.
• Gym Electric Motor (GEM) is an OpenAI Gym environment for simulating electric motor control and RL experiments.
• MuJoCo (Multi-Joint dynamics with Contact) is intended for cont inuous control tasks, performed in a multi-body dynami cs simulator with contact.
• Gym-Chrono is a set of cont inuous state and action space DRL environm ents based on the C hrono physics eng ine.

Ressources

Web Sites:

OpenAI Gymm library
OpenAI Gymmnasium
Environments: Atari, MuJoCo, Toy text, Classic control, Box2D, Third Party environments
Note: Gymnasium is a maintained fork of OpenAI’s Gym library.

ScenarIO and gym-ignition
Gym-Ignition environment
Stable Baselines

OpenAI Spinning Up
Environments: single-agent, Multi-Agent, External Agents and Applications like external simulators (Unity3D, Gazebo, ...)

Tutorials:

Solving Blackjack with Q-Learning on Gymnasium Documentation
Handling Time Limits on Gymnasium Documentation
Implementing Custom Wrappers on Gymnasium Documentation
Training using REINFORCE for Mujoco on Gymnasium Documentation
Make your own custom environment on Gymnasium Documentation

Make your own custom environment on Gymn Documentation
Vectorising your environments on Gym Documentation

Reinforcement Learning (DQN) tutorial on PyTorch tutorials.
Train a mario-Playing RL agent on PyTorch tutorials.

Playing CartPole with the Actor-Critic Method on TensorFlow Tutorials.
Agents is a library for reinforcement learning in TensorFlow on TensorFlow Tutorials.
0. Introduction to RL and Deep Q Networks on TensorFlow Tutorials.
1. Train a Deep Q Network with TF-Agents on TensorFlow Tutorials.
2. Environment on TensorFlow Tutorials.
3. Policies on TensorFlow Tutorials.
4. Drivers on TensorFlow Tutorials.
5. Replay Buffers on TensorFlow Tutorials.
6. REINFORCE agent on TensorFlow Tutorials.
7. SAC minitaur with the Actor-Learner API on TensorFlow Tutorials.
8. Networks on TensorFlow Tutorials.
9. DQN C51/Rainbow on TensorFlow Tutorials.
10. Checkpointer and PolicySaver on TensorFlow Tutorials.
Bandits tutorials on TensorFlow Tutorials.
Introduction to Multi-Armed Bandits on TensorFlow Tutorials.
A Tutorial on Multi-Armed Bandits with Per-Arm Features on TensorFlow Tutorials.
Tutorial on Ranking in TF-Agents on TensorFlow Tutorials.

Panda pick and place the cube in the bucket with OpenAI Gym robotics environments simulated in Ignition Gazebo.

Serving reinforcement learning policy models on Ray RLlib Tutorials.
Hands-on RL with Ray’s RLlib on Ray RLlib Tutorials.
Cartpole LSTM on Ray RLlib Tutorials.

Examples folder on Ray RLlib Tutorials.

Courses: Markov Decision Process (MDP) and Dynamic Programming

CS 188 | Introduction to Artificial Intelligence (Fall 2018, Week 4)

• RL | Video 1 - Slides
• RL | Video 2 - Slides

Courses: Reinforcement Learning

CS 188 | Introduction to Artificial Intelligence (Fall 2018, Week 5)

• RL | Video 1 - Slides
• RL | Video 2 - Slides

Courses: Deep Reinforcement Learning:

CS 285 | Deep Reinforcement Learning
OpenAI Spinning Up
Stable Baselines
OpenAI Gym

Courses: Advanced Topics in Learning and Decision Making:

CS 294-190 | Advanced topics in learning and decision making (Fall 2021)

• DRL | Videos
  
• RL and Language (Week 10)
  Learning to Win by Reading Manuals in a Monte-Carlo Framework (30 April 2012) by S.R.K. Branavan, David Silver, Regina Barzilay
  

Courses: Advanced Robotics:

CS 287: Advanced Robotics by Pieter Abbeel, Fall 2019

• Videos and Website

Books:

Reinforcement Learning an Introduction (Bradford Books, 2018) by Richard S. Sutton and Barto

Figure: troubleshooting-it-admin from Tenor site

Troubleshooting with Gym-Ignition

During my project on reinforcement learning for the Gazebo Ignition simulator (Fortress version) I encountered some problems during the installation of Gym-Ignition and I found solutions. Some solutions solved my problems, some not completely, so if you have better solutions or other problems you solved that you would like to share, feel free to share them, I will add them to the list.

Thank you for your contribution and hoping that this section can help you too.

- PROBLEM 1 : ERROR: Could not find a version that satisfies the requirement …    

Problem:

ERROR: Could not find a version that satisfies the requirement gy_ignition_models (from versions: none)
ERROR: No matching distribution found for gy_ignition_models

Solution:

pip install gym-ignition-models

- PROBLEM 2 : WARNING: You are using pip version … ; however, version … is available.   

Problem:

WARNING: You are using pip version 21.3.1; however, version 22.0.4 is available.
You should consider upgrading via the '/home/norlab/PycharmProjects/Ignition/venv/bin/python -m pip install --upgrade pip' command.

Solution:

/home/norlab/PycharmProjects/Ignition/venv/bin/python -m pip install --upgrade pip  

- PROBLEM 3 : gym-ignition-models   

Solution:

# From PyPI (release versions)
pip3 install gym-ignition-models

# From PyPI (pre-release versions)
pip3 install --pre gym-ignition-models

# From the repository
pip3 install git+https://github.com/robotology/gym-ignition-models.git

- PROBLEM 4 : E: Unable to locate package colcon    

Problem:

sudo apt-get install colcon
E: Unable to locate package colcon

Solution:

sudo apt install python3-colcon-common-extensions  

- PROBLEM 5 : ImportError: cannot import name 'gazebo' from partially initialized module 'scenario'      

Problem:

from scenario import gazebo as scenario_gazebo
ImportError: cannot import name 'gazebo' from partially initialized module 'scenario' (most likely due to a circular import) (/home/norlab/PycharmProjects/Ignition/scenario.py)

Solution: I don't have the solution yet

- PROBLEM 6 : QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime'    

Link

Problem:

QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime'

Solution:

export XDG_RUNTIME_DIR=/some/directory/you/specify 
export RUNLEVEL=3
source .bashrc