A Critical Overview of Neural Network Pattern Classifiers*

A taxonomy of neural network pattern classifiers is presented which includes four major groupings. Global discriminant classifiers use sigmoid or polynoniial computing elements that have “high” non-zero outputs over most of their input space. Local discriminant classifiers use Gaussian or other localized computing elements that have Lbhigli” non-zero outputs over only a small localized region of their input space. Nearest Neighbor classifiers compute the distance to stored exemplar patterns and Rule Forming classifiers use binary threshold-logic computing elements to produce binary outputs. Results of experiments are presented which demonstrate that neural network classifiers provide error rates which are equivalent to and sornetirries lower than those of more conventional Gaussian, Gaussian mixture, and binary tree classifiers using the same amount of training data. Many neural network classifiers also provide outputs which estimate Bayesian a posteriori probabilities. Experiments used low-diniensional (2 to 55 input) phoneme classification tasks, a high-diiiiensioiial (360 pixel input) handwritten digit classification task, and niacliirie learning data bases. They demonstrate that neural network classifiers provide new alternatives to more conventional approaches. They often provide reduced error rates and always allow other classifier characteristics to be traded off to best match the requirements of a particular problem. Characteristics which often differ dramatically across classifiers include classification time, training and adaptation time, ease of implementation, memory requirements, rejection accuracy, and usefulness of outputs a s Bayes probability estimates. INTRODUCTION The table in Figure 1 contains a tasonomy of five major types of neural network and conventsional patt,ern clasifiers that can be used to classify fixedlength patterns. The first row in this table represents conventional proba‘This work was sponsored by the Defense Advanced Research Projects Agency and the Air Force Office of Scientific Research. 0-7803-0 11 8-8/9 1 /OOO9-U266SO1 .MI 0 1 99 1 IEEE I GROUP I DECISION REGION I COMPUTING ELEMENT REPRESENTATIVE CLASIFIERS PROBABILISTIC DISTRIBUTION DEPENDENT Figure 1: A Taxonomy Inclticling Five Types of Conventional and Neural Network Patberii Classifiers. bilistic classifiers \vliicli i iwd(~1 likelihood distributions of pattern classes separately using parametric functions. Probabilistic classifiers used for speech recognition include Gaussian linear discriminant and Gaussian mixture classifiers. This most common approach to pattern classification provides good performance when the assumed functional forin of class distributions matches real-world data distributions and when there is sufficient training data to e s timate parameters. Performance can be poor when class distributions are not modeled well or when training data is limited. The bottom four ro\vs i n Figure 1 include both neural network and conventional classifiers. Global classifiers form output discriminant functions from internal comput,ing elements or nodes that use sigmoid or polynomial functions which have “high” non-zero outputs over a large region of the input space. These classifiers include iiiulti-layer perceptrons trained with backpropagation (back-propagation classifiers), Boltzmann machines, and highorder polynomial networks. Local classifiers form output discriminant functions from internal computing elements that use Gaussian or other radially symmetric functions which have “high” non-zero outputs over only a localized region of the input space. These two types of classifiers make no strong assumptions concerning underlying distributions. They can form complex decision regions with only one or t.wo hidden layers and are typically trained to minimize the mean-squared error between desired and actual network out-