Machine Learning-Based Malware Detection and Classification in Encrypted TLS Traffic

Abstract

Malware has become a significant threat to Internet users in the modern digital era. Malware spreads quickly and poses a significant threat to cyber security. As a result, network security measures play an important role in countering these cyber threats. Existing malware detection techniques are unable to detect them effectively. A novel Ensemble Machine Learning (ML)-based malware detection technique from Transport Layer Security (TLS)-encrypted traffic without decryption is proposed in this paper. The features are extracted from TLS traffic. Based on the extracted features, malware detection is performed using Ensemble ML algorithms. The benign and malware file datasets are created using features extracted from TLS traffic. According to the experimental results, the 65 new extracted features perform well in detecting malware from encrypted traffic. The proposed method achieves an accuracy of 99.85% for random forest and 97.43% for multiclass classification for identifying malware families. The ensemble model achieved an accuracy of 99.74% for binary classification and 97.45% for multiclass classification. © 2023, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.