Flexible Transfer Functions with Ontogenic Neural Networks

Transfer functions play a very important role in learning process of neural systems. This paper presents new functions which are more flexible than other functions commonly used in artificial neural networks. The latest improvement added is the ability to rotate the contours of constant values of transfer functions in multidimensional spaces with only N − 1 adaptive parameters. Rotation using full covariance matrices requires N parameters. These functions have biases and slopes separable in each dimension for each neuron, completely independent in multi-dimensional spaces. Therefore they are dimensionally separable — each dimension may be excluded independently at any time. As the neural network model for testing the performance of these new transfer functions the Incremental Network (IncNet) was chosen. These networks are similar to radial basis function (RBF) networks and resource allocating networks. The architecture of IncNet is the same as the architecture of RBF networks, but the structure (the number of hidden nodes) changes during the learning process according to certain statistical criterion that controls growth and pruning of network connections. Preliminary results show superior performance of the new transfer functions comparing with gaussian functions often used by RBF networks and other models.