Detection and Categorization of DNS over HTTPS Traffic Using Lightweight Feature Selection Methods and Ensemble Classification Model

Abstract

In recent times, the adoption of DNS-over-HTTPS (DoH) has been projected as a means to secure DNS queries through encryption and shielding these communications from potential eavesdroppers via HTTPS and TLS protocols. However, this advancement is a double-edged sword as it also offers a veil for cybercriminals to execute undetected data exfiltration and command-and-control (C2) attacks. This creates a substantial challenge for network administrators who must detect malicious activities without direct visibility into the content of DoH traffic. Our research addresses this critical issue by detecting and categorizing DoH using lightweight feature selection algorithms, Mutual Information (MI) and Fisher’s score to determine the most relevant features within DoH traffic. These features were then used to train Random Forest, CatBoost, and XGBoost classifiers. We adopted a layered approach: the first layer focuses on accurately detecting DoH traffic, while the second layer classifies the nature of the detected DoH traffic. A Voting-based ensemble classifier trained and tested on features identified by both algorithms achieved high accuracy rates of 99.7% in the initial layer and 100% in the second layer. Experimental results demonstrate that our model infrequently misclassifies HTTPS traffic as DoH, enhancing the reliability of our detection mechanism. In an effort to make these processes transparent, we have implemented an explainable AI framework that clarifies the decision-making pathways of our best base model. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.