Traffic anomaly detection and cause identification using flow-level measurements

Measurement and analysis of traffic in IP networks are of great interest for network operators as they provide important information about the utilization of network resources, the user behavior, as well as the deployed applications and services. In particular, flow-level traffic measurements have become more and more ubiquitous in the past years. In this context, a flow is defined as a stream of packets which are observed at a given interface in the network and which share a set of common properties called flow keys. For each flow, a flow record is generated containing the flow keys as well as additional flow attributes and statistical information, such as the observation time of the flow, the number of bytes and packets etc. This dissertation deals with the detection of traffic anomalies and the identification of their causes using flow-level measurement data. Traffic anomalies are significant deviations from the pattern of usual network traffic. Possible reasons for traffic anomalies are changes in the network topology (e.g., newly connected hosts, routing changes) or network usage (e.g., changed customer behavior, new applications). Anomalies may also be caused by failures of network devices as well as by malicious worm or attack traffic. The early detection of such events is of particular interest as they may impair the safe and reliable operation of the network. For the detection of traffic anomalies, we convert the flow records into time series of eight different traffic metrics describing the traffic volume as well as the cardinality of certain flow keys. We investigate various statistical change detection methods which have been originally conceived for quality control in manufacturing processes. In particular, we use control charts to detect shifts in the mean, standard deviation, or correlation of traffic measurement variables. As most traffic measurement variables exhibit nonstationarity and serial correlation, residual generation methods need to be applied in order to reduce the effect of systematic changes, such as seasonal variation. For anomaly detection in a single traffic metric, we examine different time-series analysis methods with special focus on robust forecasting techniques. For multi-metric anomaly detection, we study the application of principal component analysis (PCA) which allows modeling the correlation structure between different measurement variables. The investigated change detection and residual generation methods are evaluated and compared based on flow data which was collected in the network of an Internet service provider (ISP). The detected anomalies are clas-