Automated multi-level malware detection system based on reconstructed semantic view of executables using machine learning techniques at VMM

Abstract

In order to fulfill the requirements like stringent timing restraints and demand on resources, Cyber–Physical System (CPS) must deploy on the virtualized environment such as cloud computing. To protect Virtual Machines (VMs) in which CPSs are functioning against malware-based attacks, malware detection and mitigation technique is emerging as a highly crucial concern. The traditional VM-based anti-malware software themselves a potential target for malware-based attack since they are easily subverted by sophisticated malware. Thus, a reliable and robust malware monitoring and detection systems are needed to detect and mitigate rapidly the malware based cyber-attacks in real time particularly for virtualized environment. The Virtual Machine Introspection (VMI) has emerged as a fine-grained out-of-VM security solution to detect malware by introspecting and reconstructing the volatile memory state of the live guest Operating System (OS) by functioning at the Virtual Machine Monitor (VMM) or hypervisor. However, the reconstructed semantic details by the VMI are available in a combination of benign and malicious states at the hypervisor. In order to distinguish between these two states, extensive manual analysis is required by the existing out-of-VM security solutions. To address the foremost issue, in this paper, we propose an advanced VMM-based guest-assisted Automated Multilevel Malware Detection System (AMMDS) that leverages both VMI and Memory Forensic Analysis (MFA) techniques to predict early symptoms of malware execution by detecting stealthy hidden processes on a live guest OS. More specifically, the AMMDS system detects and classifies the actual running malicious executables from the semantically reconstructed process view of the guest OS. The two sub-components of the AMMDS are: Online Malware Detector (OMD) and Offline Malware Classifier (OFMC). The OMD recognizes whether the running processes are benign or malicious using its Local Malware Signature Database (LMSD) and online malware scanner and the OFMC classify unknown malware by adopting machine learning techniques at the hypervisor. The AMMDS has been evaluated by executing large real-world malware and benign executables on to the live guest OSs. The evaluation results achieved 100% of accuracy and zero False Positive Rate (FPR) on the 10-fold cross-validation in classifying unknown malware with maximum performance overhead of 5.8%. © 2017 Elsevier B.V.