High Order Non-stationary Markov Models and Anomaly Propagation Analysis in Intrusion Detection System (ids)

A new concept targeted to decrease false positive rates of anomaly based intrusion detection operating in the system call domain is proposed. To mitigate false positives, network based correlation of collected anomalies from different hosts is suggested, as well as a new means of host-based anomaly detection. The concept of anomaly propagation is based on the premise that false alarms do not propagate within the network. Unless anomaly propagation is observed alarms are to be treated as false positives. The rationale behind the concept lies in the fact that the most common feature of worms and viruses is selfreplication. As replication takes place, a malicious code propagating through the network would carry out the same activity resulting in almost identical system call sequences and triggering the same alarm at different hosts. The alarm propagation effect can be used to distinguish “true alarms” from “false positives”. At the host-level, a new anomaly detection mechanism operating that employs non-stationary Markov models is proposed. Many applications or services have different operating modes, which have different dynamics with respect to system call issuance. Therefore, an application or service can be treated as a non-stationary stochastic process and be modeled with a non-stationary Markov chain, which significantly improves model consistency compared to stationary Markov chain. INTRODUCTION The first Intrusion Detection System (IDS) utilizing system calls was proposed in [1]. Today, these systems utilize two main approaches, misuse detection and anomaly detection. Misuse or signature-based detection systems utilize descriptions of known attack expressed in terms of system calls. Although signature-based systems can provide high level of accuracy, they fail to detect previously unknown attacks. Anomaly detection systems utilize models of normal behavior of legitimate processes, especially privileged ones. These systems check the consistency between the invoked system calls and the profile of normality for a given process and have the potential to detect unknown attacks, though they frequently suffer from a high rate of false positives. This research targets anomaly-based IDSs that in spite of their advantages are impractical due to high rate of false positives. The limited success of known research aimed at the alleviation of this problem [2, 3, 6, 8] is due to it being primarily aimed at the improving the accuracy of the normality models (profiles) rather than achieving high confidence in classifying the detected anomaly. Two major contributions of this paper are as follows. First, a novel host-level anomaly detection mechanism is proposed. Second, having efficient hostlevel anomaly detection, the unique but rather simple principle, false positives do not propagate, is suggested as the basis for establishing, with high degree of confidence, whether detected anomaly is a false positive or a true positive. The anomaly detection mechanism utilizes nonstationary Markov models. While many shell codes and exploits (in buffer overflow attack) may use only 20-30 system calls, which would certainly be concealed in a histogram, Markov models are clearly preferable to other order insensitive techniques (such us frequency histograms) used to model normality profiles [1, 3]. However, the common assumption that the source (application or service) is a stationary stochastic process generally may not by true. Any application or services utilize high level functions intended to solve different tasks. When an application realizes several related tasks or group of tasks which