Pruning with Minimum Description Length

The number of parameters in a model and its ability to generalize on the underlying data-generating machinery are tightly coupled entities. Neural networks consist usually of a large number of parameters, and pruning (the process of setting single parameters to zero) has been used to reduce the nets complexity in order to increase its generalization ability. Another less obvious approach is to use Minimum Description Length (MDL) to increase generalization. MDL is the only model selection criterion giving a uniform treatment of a) the complexity of the model and b) how well the model ts a speciic data set. This article investigates pruning based on MDL, and it is shown that the derived algorithm results in a scheme identical to the well known Optimal Brain Damage pruning. Furthermore, an example is given on a well known benchmark data set yielding ne results.