Behavioral Fingerprinting for Detecting Covert DNS Tunneling Activities in Enterprise Networks

Abstract

DNS tunneling is a common covert communication technique used to bypass traditional security monitoring systems and exfiltrate data through seemingly legitimate domain name queries. Because DNS traffic is usually allowed across enterprise and institutional networks, detecting tunneling behavior requires models that can identify subtle statistical and behavioral deviations rather than rely only on blacklist matching. This study proposes an explainable ensemble learning framework for DNS tunneling detection based on query behavior and flow-level features. The proposed model combines LightGBM, CatBoost, and Extremely Randomized Trees through a rank-weighted voting strategy. SHAP analysis is applied to explain how query length, domain entropy, subdomain repetition, response size, query interval, and NXDOMAIN ratio contribute to detection decisions. Experiments are conducted on a DNS traffic dataset collected from a controlled enterprise network with 6 recursive resolvers and 3,850 client hosts over 35 days. The dataset contains 5.64 million DNS query-flow records, including normal browsing, software update traffic, cloud service access, malware-generated domains, iodine-based tunneling, dnscat2 tunneling, and low-rate data exfiltration. After feature engineering, 46 features are retained for training and interpretation. The proposed model achieves 98.73% accuracy, 97.88% F1-score, and 99.21% AUC in binary detection. In multi-class detection, it obtains a macro-F1 of 96.42% across normal DNS, malware DNS, high-throughput tunneling, and low-rate tunneling classes. Compared with standalone CatBoost, the ensemble model reduces the false positive rate from 2.31% to 1.44% and improves recall for low-rate tunneling by 4.27%. SHAP results show that subdomain entropy, repeated long-label queries, abnormal NXDOMAIN ratio, and short periodic query intervals are the strongest indicators of DNS tunneling behavior. The results demonstrate that explainable ensemble learning can provide accurate and interpretable detection of covert DNS-based network anomalies.