Comprehensive Comparison of Various Machine Learning Algorithms for RF Fingerprints Classification

Abstract

In these days, the use of drones has become quite common. Remote controls can do the control of these drones with RF signals. It is important to prevent security vulnerabilities caused by using drones in our daily lives. A complex dataset was created by extracting the characteristics of the RF signals and preprocessing them. To solve this complex data set and problem, the application of models including Support Vector Machine (SVM), Random Forest, Decision Tree, Gradient Boosting, XGBoost and Neural Network (NN) models, including various ML models and comparison of optimization studies of these applied models are examined in this article. In addition, a wide range of studies was carried out to compare ML models, including comparison metrics such as Accuracy, Precision, Recall, Mean Squared Error (MSE), F1 Score, $R^{2}$ and Training Time. In line with these results, the highest score was obtained in the $\mathrm{R}^{2}$ comparison metric (97%) in the Neural Network (NN). Compared to the others, the results of Neural Network (NN) were more successful, but the Training Time (245 sec) in the Neural Network (NN) method is by far more than the other ML methods, which shows us that the NN method requires a very high computing process. As a result of the comparison, another outstanding Ensemble-based ML method is Decision Tree. This is because besides the very low Training Time $(5\sec)$, it has managed to be the 2nd ML algorithm with the highest $\mathrm{R}^{2}$ score (96%). Apart from these, among other ML methods, SVM performed slightly less well $(\mathrm{R}^{2}$ 91%) in solving this complex problem. The advanced Gradient Method (95%) and XGBoost (96%), which also have the Ensemble structure, showed a head-to-head performance regarding $\mathrm{R}^{2}$ scores. However, XGBoost (30 sec) has a very short Training Time compared to Gradient Boosting (180 sec). As a result, the approach of each ML method to solving the complex problem differed from each other, and the success rates and Training Time also differed equally. The most important work to be done here is to choose which ML method you want to achieve according to the limited system in hand and the performance-accuracy dilemma.