Enhanced Malicious Traffic Detection in Encrypted Communication Using TLS Features and a Multi-class Classifier Ensemble

Abstract

The use of encryption for network communication leads to a significant challenge in identifying malicious traffic. The existing malicious traffic detection techniques fail to identify malicious traffic from the encrypted traffic without decryption. The current research focuses on feature extraction and malicious traffic classification from the encrypted network traffic without decryption. In this paper, we propose an ensemble model using Deep Learning (DL), Machine Learning (ML), and self-attention-based methods. Also, we propose novel TLS features extracted from the network and perform experimentation on the ensemble model. The experimental results demonstrated that the ML-based (RF, LGBM, XGB) ensemble model achieved a significant accuracy of 94.85% whereas the other ensemble model using RF, LSTM, and Bi-LSTM with self-attention technique achieved an accuracy of 96.71%. To evaluate the efficacy of our proposed models, we curated datasets encompassing both phishing, legitimate and malware websites, leveraging features extracted from TLS 1.2 and 1.3 traffic without decryption. © The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2024.