Overfitting and generalization in learning discrete patterns

Understanding and preventing overrtting is a very important issue in artiicial neural network design, implementation, and application. Weigend (1994) reports that the presence and absence of overrtting in neural networks depends on how the testing error is measured, and that there is no overrtting in terms of the classiication error (symbolic-level errors). In this paper, we show that, in terms of the classiication error, overrtting does occur for certain representation used to encode the discrete attributes. We design simple Boolean functions with clear rationale, and present experimental results to support our claims. In addition, we report some interesting results on the best generalization ability of networks in terms of their sizes.