Genetic algorithms for linear feature extraction

Feature extraction is a commonly used technique applied before classification when a number of measures, or features, have been taken from a set of objects in a typical statistical pattern recognition task. The goal is to define a mapping from the original representation space into a new space where the classes are more easily separable. This will reduce the classifier complexity, increasing in most cases classifier accuracy. Feature extraction methods can be divided into linear and non-linear, depending on the nature of the mapping function (Lerner et al., 1998). They can also be classified as supervised or unsupervised, depending on whether the class information is taken into account or not. Feature extraction can also be used for exploratory data analysis, where the aim is not to improve classification accuracy, but to visualise high dimensional data by mapping it into the plane or the 3dimensional space. The best known linear methods are Principal Component Analysis, or PCA (unsupervised) (Fukunaga, 1990), Linear Discriminant Analysis or LDA (supervised) (Fukunaga, 1990; Aladjem, 1991; Siedlecki et al., 1988), and Independent Component Analysis or ICA (unsupervised) (Cardoso, 1993). Schematically, PCA preserves as much variance of the data as possible, LDA attempts to group patterns of the same class, while separating them from the other classes, and ICA obtains a new set of features by extracting the less correlated (in a broad sense) directions in the data set. On the other hand, well-known non-linear methods are: Sammon’s Mapping (unsupervised) (Sammon, 1969; Siedlecki et al. 1988), non-linear discriminant analysis or NDA (supervised) (Mao & Jain, 1995), Kohonen’s self-organising map (unsupervised) (Kohonen, 1990) and evolutionary extraction (supervised) (Liu & Motoda, 1998). Sammon’s mapping tries to keep the distances among the observations using hill-climbing or neural network methods (Mao & Jain, 1995; Sammon, 1969), NDA obtains new features from the coefficients of the second hidden layers of a multi-layer perceptron (MLP) (Mao & Jain, 1995) and Kohonen Maps project data in an attempt to preserve the topology. Finally, evolutionary extraction uses a genetic algorithm to find combinations of original features in order to improve classifier accuracy. These new features are obtained by multiplying, dividing, adding or subtracting the original features. In the linear methods, the mapping function is known and simple; therefore, the task is reduced to finding the coefficients of the linear transformation by maximising or minimising