Alert correlation and prediction using data mining and HMM

Intrusion Detection Systems (IDSs) are security tools widely used in computer networks. While they seem to be promising technologies, they pose some serious drawbacks: When utilized in large and high traffic networks, IDSs generate high volumes of low-level alerts which are hardly manageable. Accordingly, there emerged a recent track of security research, focused on alert correlation, which extracts useful and high-level alerts, and helps to make timely decisions when a security breach occurs. In this paper, we propose an alert correlation system consisting of two major components; first, we introduce an Attack Scenario Extraction Algorithm (ASEA), which mines the stream of alerts for attack scenarios. The ASEA has a relatively good performance, both in speed and memory consumption. Contrary to previous approaches, the ASEA combines both prior knowledge as well as statistical relationships. Second, we propose a Hidden Markov Model (HMM)-based correlation method of intrusion alerts, fired from different IDS sensors across an enterprise. We use HMM to predict the next attack class of the intruder, also known as plan recognition. This component has two advantages: Firstly, it does not require any usage or modeling of network topology, system vulnerabilities, and system configurations; Secondly, as we perform high-level prediction, the model is more robust against over-fitting. In contrast, other published plan-recognition methods try to predict exactly the next attacker action. We applied our system to DARPA 2000 intrusion detection scenario dataset. The ASEA experiment shows that it can extract attack strategies efficiently. We evaluated our plan-recognition component both with supervised and unsupervised learning techniques using DARPA 2000 dataset. To the best of our knowledge, this is the first unsupervised method in attack plan recognition.