Safe-to-fly : An On-board Intelligent Fault Diagnosis System With AutoML for Unmanned Aerial Vehicles [Paper | Code]

✯ Abstract

Real-time fault diagnosis in Unmanned Aerial Vehicles (UAVs) is a challenging task. Data-driven intelligent diagnosis of faults ensures flight safety for UAVs. In this paper, realtime fault diagnosis on small scale fixed-wing UAVs has been shown by data of natural flight conditions with a wrapped wing structure that breaks the geometric symmetry. AutoML based approach was taken for multi-class fault classification. Two datasets were created from the combination of flight data of two days. The experimental results showed that the proposed Deep Learning AutoML model significantly improves performance over conventional Machine Learning methods such as the Decision Tree, KNN, and the Random Forest. The test accuracy of the proposed AutoML model was 74% and 100% on the first and second datasets, respectively. The AutoML model’s capability in classifying low fault severity and complex faults demonstrated the method’s usefulness and excellence.

✯ Datset Acquisition & Preparation

• The dataset for the fault detection was acquired from M. Bronz et al. [1] It contains multi-class fault data for two days(21st of July 2020 and 23rd of July 2020) of identical flights.
• It comprised of faults related to left control surface efficiency and right control surface efficiency with eight fault types and normal flight data.
• Two datasets are created by the combination of flight data for two days. The first dataset consists of the flight data of 21st July, which has been divided into training and testing data with an 80:20 split.
• The second dataset is created by combining flight data of 21st July and 23rd July and dividing the data in training and testing set of split size 80:20.

✯ Fault Detection Method

• Several state-of-the-art Machine Learning algorithms such as Decision Tree Classifier, KNN, and Random Forest Classifier were applied on the datasets for comparison of the results with the AutoML model.
• Auto-Keras leverages Neural Architecture Search, but with the addition of network morphism and Bayesian optimisation to direct the network morphism for a more effective neural network search.
• The Safe-to-fly AutoML Model I was trained on flight data of 21st July and tested on 23rd July data.
• The Safe-to-fly AutoML Model II was trained on 80% of combined data of 21st July and 23rd July and evaluated on the rest 20% data.

✯ Model Evaluation on Dataset 1

• Table III shows a comparison of training and test accuracy for different ML Algorithms employed.
• It is evident that the AutoML model has comparable results with respect to other ML Algorithms.
• The Fig. 3 shows the comparison of different performance metrics, namely Precision, Recall and F-1 Score for the ML models.
• As the testing dataset was of an entirely different flight, the AutoML model performance demonstrates its usability in the real world.

✯ Model Evaluation on Dataset 2

• As the Dataset 2 was prepared by combining the flight data of 21st and 23rd July, which resulted in larger training data than the previous case.
• Hence an improvement in the performance of all the ML Algorithms can be seen.
• The Table IV shows the comparison of different performance metrics values for the various Machine Learning algorithms with the proposed AutoML model.
• The Fig. 4 shows the Confusion Metrics obtained when the AutoML model was evaluated on the test dataset.
• It is evident from the confusion matrix that AutoML nearly classified all the faults accurately.
• The Fig. 5 shows the comparison of performance metrics for different types of faults, which was obtained with the help of the Confusion Matrix.
• The high values of precision recall and F-1 score shows the efficiency of the AutoML model.
• This also proves the usability and generality of the AutoML model used.

✯ BibTeX

If you find this code or paper useful, please use the following reference:

@INPROCEEDINGS{9752852,  
author={Bompilwar, Ritik and Rathor, Surya Pratap Singh and Sinha, Aparna and Das, Debanjan},  
booktitle={2022 IEEE Delhi Section Conference (DELCON)},   
title={Safe-to-fly : An On-board Intelligent Fault Diagnosis System With AutoML for Unmanned Aerial Vehicles},   
year={2022},  
volume={},  
number={},  
pages={1-5},  
doi={10.1109/DELCON54057.2022.9752852}}

✯ References

1. M. Bronz, E. Baskaya, D. Delahaye, and S. Puechmore, “Real-time fault detection on small fixed-wing uavs using machine learning,” in 2020 AIAA/IEEE 39th Digital Avionics Systems Conference (DASC). IEEE, 2020, pp. 1–10
2. Orignal Dataset : https://github.com/mrtbrnz/fault_detection