Fast Learning with Noise in Deep Neural Nets

Dropout has been raised as an effective and simple trick [1] to combat overfitting in deep neural nets. The idea is to randomly mask out input and internal units during training. Despite its usefulness, there has been very little and scattered understanding on injecting noise to deep learning architectures’ internal units. In this paper, we study the effect of dropout on both input and hidden layers in deep neural nets via explicit formulation of an equivalent marginalization regularizer. We show that training with regularizer from marginalized noise in deep neural nets doesn’t loose much performance compared to dropout, yet in significantly shorter amount of training time and noticeably less sensitivity to hyperparameter tuning, which are main practical concerns of dropout.