Neural Networks: What Non-linearity to Choose

Neural networks are now one of the most successful learning formalisms. Neurons transform inputs x1, ..., xn into an output f(w1x1 + ... + wnxn), where f is a non-linear function and wi are adjustable weights. What f to choose? Usually the logistic function is chosen, but sometimes the use of different functions improves the practical efficiency of the network. We formulate the problem of choosing f as a mathematical optimization problem and solve it under different optimality criteria. As a result, we get a list of functions f that are optimal under these criteria. This list includes both the functions that were empirically proved to be the best for some problems, and some new functions that may be worth trying. 1. FORMULATION OF THE PROBLEM. Neural networks are now one of the most successful learning formalisms (see, e.g., the recent survey in [Hecht-Nielsen 1991]). After the initial success of linear neural models, in which the output y is equal to the linear combination of the input signals xi, i.e. y = w1x1 + w2x2 + ..., it was shown in [Minsky Papert 1968] that if we only have linear neurons, then we end up with only linear functions and this severely limits the number of problems that we can solve using the network. The next step, then, is to consider non-linear neurons, in which the output signal is equal to f(w1x1 +w2x2 + ...), where f(y) is a given non-linear function. A natural question arises: what function f(y) do we choose? Why is this problem important? It is a very important problem because although neural networks help us to design good learning procedures, these procedures are far from being reliable. Sometimes these procedures do not work; sometimes they work but demand too much time, and too big a sample, to learn. Naturally, we might think that this is because the function f that we used was not the best one. Sometimes the use of different functions can improve the practical efficiency of the network (see, e.g., [Wasserman 1989, pp. 15-16]). If a simple guess can really improve the learning performance, then it is natural to suppose that deep mathematical optimization will lead to even better results. Why is this problem difficult? We want to find a function f for which some characteristics J of learning, such as average learning time or average number of errors, is optimal (in these cases