Convergence of Batch BP Algorithm with Penalty for FNN Training

Penalty methods have been commonly used to improve the generalization performance of feedforward neural networks and to control the magnitude of the network weights. Weight boundedness and convergence results are presented for the batch BP algorithm with penalty for training feedforward neural networks with a hidden layer. A key point of the proofs is the monotonicity of the error function with the penalty term during the training iteration. A relationship between the learning rate parameter and the penalty parameter is proposed to guarantee the convergence. The algorithm is applied to two classification problems to support our theoretical findings.