In this work, we aim to analyze and characterize malware persistence mechanisms using dynamic instrumentation. We are specifically interested in analyzing malware that is particularly stealthy; specifically, we aim to analyze malware that utilizes sophisticated rootkit techniques to deceive or hide from malware analyzers and other security solutions. We have developed Dione, a disk instrumentation infrastructure that interposes between a system and its hard disk and reconstructs high-level file system operations from raw intercepted disk access contents and metadata. Using Dione and other analysis tools, we plan to characterize malware by the mechanisms used to persist on disk--that is, to label malware by the mechanisms it uses to survive and restart after a system reboot.

Intrusion detection is one of the high priority and challenging tasks in many technologies, particularly, in virtualization technology. There is a need to safeguard these systems from known vulnerabilities and at the same time take steps to detect new and unseen, but possible, system abuses by developing more reliable and efficient intrusion detection systems. In this correspondence, we propose a machine learning based intrusion detection algorithm based on Enhanced Boosting with Decision Stumps algorithm to detect various categories of attacks utilizing information embedded within the virtual machine monitor (VMM) level. In the algorithm, decision stumps are used as weak classifiers. The decision rules are provided for different types of features. By combining the weak classifiers for the heterogeneous mixture features types into a strong classifier, the relations between these features are handled naturally, without any forced conversions between them. Moreover, adjustable initial weights based on the area under the ROC curve (AUC) are adopted to make a tradeoff between the false-alarm and detection rates. Experimental results show that our algorithm has low computational complexity and error rates as tested on real malwares.

A Virtual Machine Monitor (VMM) is becoming an increasingly popular service hosting platform. Recently, intrusion detection systems (IDSs) which utilize VMMs have been introduced. One particular challenge is considered here, learning from high dimensional imbalanced data, which is an intrinsic problem to intrusion detection. Feature selection methods are critical to achieve optimal performance with imbalanced data, resulting in features that are relevant for obtaining lower false alarms but at possibly lower detection accuracies. In this work I proposed a new Boosting based feature selection that evaluates the relative importance of individual features using the fractional absolute confidence that Boosting produces. My approach accounts for the sample distributions by optimizing for the area under the ROC curve (AUC). Empirical results on different commercial virtual appliances and malwares indicate that significant input feature selection is important to design a VMM IDS that is lightweight, efficient and effective.

This work focuses on design and implementation of a host based intrusion detection using machine learning techniques to increase the efficiency and accuracy in detection. We aim to model the behavior of processes by analyzing the system call traces. Each trace is the sequence of system calls which is collected by executing a process on the host computer. Extracting the relationships and the dynamic behavior between the system calls in a sequence is the main challenge of this work. We propose a multi-Hidden Markov Model (m-HMM) based approach for sequential behavior analysis. In information and computer security, immediate detection of any intrusions is one of the most important tasks to protect the system. In this project, instead of classifying a complete set of system call trace as an intrusion or not, we improve the intrusion detection system by estimating where the intrusion begins in a process trace. This step is also important to understand the behavior of intrusion by monitoring the following system calls after the detection.