1 A Classification System for Rule Extraction from Support Vector Machines

Over the last years, a number of studies on rule extraction from support vector machines (SVMs) have been introduced [1-5]. The research strategy in these projects is similar: to explore and develop algorithms for rule extraction based on the perception (or “view”) of the underlying SVM which is either explicitly or implicitly assumed within the rule extraction technique. In the context of rule extraction from artificial neural networks (ANNs) [6, 7] the notion of “translucency” describes the degree to which the internal representation of the ANN is accessible to the rule extraction technique. More broadly, a taxonomy for rule extraction from neural networks has been introduced [6, 7] which includes five evaluation criteria: translucency, rule quality, expressive power, portability and algorithmic complexity. These evaluation criteria are now commonly used for rule extraction from SVMs. At this point, it is important to develop new techniques for rule extraction from support vector machines, including those that are soly based on SVMs and do not require any other machine learning technique. In particular support vector machines that allow the generation of structured outputs [8, 9] can be used to generate rule sets not unlike those extracted from neural networks. This represents a clear advancement since user explanation is realized by an SVM and not by a technique with a different representational bias. In addition, methods for the extraction of high quality rule sets from SVMs trained on high-dimensional data are required. The following briefly describes the first two of the five evaluation criteria for rule extraction from neural networks [6, 7] which are then discussed in the context of rule extraction from SVMs. A new classification schema for rule extraction from SVMs is presented and an approach is outlined which (1) uses SVMs only, including those generating structured output, and (2) works well for high-dimensional data. Translucency and rule quality in the context of rule extraction from ANNs Translucency describes the degree to which the internal representation of the ANN is accessible to the rule extraction technique. At one end of the translucency spectrum are those rule extraction techniques which view the underlying ANN at the maximum level 2 of granularity i.e. as a set of discrete hidden and output units. Craven and Shavlik [10] categorized such techniques as “decompositional”. The basic strategy of such decompositional techniques is to extract rules at the level of each individual hidden and output unit within the trained ANN. In general, decompositional rule extraction techniques incorporate some form of analysis of the weight vector and associated bias (threshold) of each unit in the trained ANN. Then, by treating each unit in the ANN as an isolated entity, decompositional techniques initially generate rules in which the antecedents and consequents are expressed in terms which are local to the unit from which they are derived. A process of aggregation is then required to transform these local rules into a composite rule base for the ANN as a whole [7]. In contrast to the decompositional approaches, the strategy of the pedagogical approaches is to view the trained ANN at the minimum possible level of granularity i.e. as a single entity or alternatively as a “black box”. The focus is on finding rules that map the (ANN) inputs (i.e. the attribute/value pairs from the problem domain) directly to outputs [7]. In addition to these two main categories, Andrews et. al. [6] also proposed a third category which they labeled as “eclectic” to accommodate those rule extraction techniques which incorporate elements of both the decompositional and pedagogical approaches. Rule extraction from neural networks adopted criteria for the quality of the extracted rules. The set of criteria for evaluating rule quality includes [6]: a) accuracy b) fidelity c) consistency, and d) comprehensibility of the extracted rules. A rule set is considered to be accurate if it can correctly classify a set of previously unseen examples from the problem domain [7]. Similarly a rule set is considered to display a high level of fidelity if it can mimic the behavior of neural network from which it was extracted by capturing all of the information represented in the ANN. An extracted rule set is deemed to be consistent if, under differing training sessions, the artificial neural network generates rule sets which produce the same classifications of unseen examples. Finally the comprehensibility of a rule set is determined by measuring the size of the rule set (in terms of the number of rules) and the number of antecedents per rule [7]. Translucency and rule quality applied to rule extraction from SVMs Most current studies on rule extraction from SVMs focus on decompositional extraction; however, learning-based approaches are also available [4]. The idea is simple: learn what the SVM has learned. For this purpose a dataset is divided in two or more parts. The first set is used to train the SVM to completion. The second set does not include targets, the inputs are presented to the SVM and the output is obtained from the SVM. Inputs and outputs combined represent a new data set which is used for a second machine learning episode by use of a machine learning system that produces rules as output.