Reinforcement Learning Flight Control: Hybrid Soft Actor-Critic and Incremental Dual Heuristic Programming for Fault-Tolerance

Authors: Casper Teirlinck  

Part of obtaining MSc Thesis at Control & Simulation Departement, Faculty of Aerospace Engineering at Delft University of Technology

Paper published by the American Institute of Aeronautics and Astronautics @ doi.org/10.2514/6.2024-2406

System Dependencies

• Python + pip (tested with Python 3.9.9)

Installation

1. Create and activate a python virtual environment in project root

   python -m venv venv

2. Install python module

   For use:

   python setup.py install

   For development:

   python setup.py develop

Usage

Evaluateing pre-trained agents:

• scripts/evaluate_sac_inner.py: evaluate SAC attitude controller
• scripts/evaluate_idhpsac_inner.py: evaluate hybrid SAC-IDHP attitude controller
• scripts/evaluate_idhpsac_inner_dc.py: evaluate hybrid SAC-IDHP attitude controller - decoupled version
• scripts/evaluate_sac_outer.py: evaluate SAC/SAC-IDHP cascaded altitude controller

Training new agents:

• scripts/train_sac_inner.py: train a new SAC attitude controller
• scripts/train_idhpsac_inner.py: train a new hybrid SAC-IDHP attitude controller
• scripts/train_idhpsac_inner_dc.py: train a new hybrid SAC-IDHP attitude controller - decoupled version
• scripts/train_sac_outer.py: train a new SAC/SAC-IDHP cascaded altitude controller

References

• K. Dally and E.-J. Van Kampen, “Soft Actor-Critic Deep Reinforcement Learning for Fault Tolerant Flight Control” [paper]

• T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor” [paper]

• T. Haarnoja et al., “Soft Actor-Critic Algorithms and Applications” [paper]

• S. Heyer, D. Kroezen, and E.-J. Van Kampen, “Online Adaptive Incremental Reinforcement Learning Flight Control for a CS-25 Class Aircraft” [paper]

• D. Kroezen, “Online Reinforcement Learning for Flight Control: An Adaptive Critic Design without prior model knowledge” [paper]

• J. Lee, “Longitudinal Flight Control by Reinforcement Learning: Online Adaptive Critic Design Approach to Altitude Control” [paper]