Two-Layer Contractive Encodings with Shortcuts for Semi-supervised Learning
نویسندگان
چکیده
Supervised training of multi-layer perceptrons (MLP) with only few labeled examples is prone to overfitting. Pretraining an MLP with unlabeled samples of the input distribution may achieve better generalization. Usually, pretraining is done in a layer-wise, greedy fashion which limits the complexity of the learnable features. To overcome this limitation, two-layer contractive encodings have been proposed recently—which pose a more difficult optimization problem, however. On the other hand, linear transformations of perceptrons have been proposed to make optimization of deep networks easier. In this paper, we propose to combine these two approaches. Experiments on handwritten digit recognition show the benefits of our combined approach to semi-supervised learning.
منابع مشابه
Two-Layer Contractive Encodings with Linear Transformation of Perceptrons for Semi-Supervised Learning
It is difficult to train a multi-layer perceptron (MLP) when there are only a few labeled samples available. However, by pretraining an MLP with vast amount of unlabeled samples available, we may achieve better generalization performance. Schulz et al. (2012) showed that it is possible to pretrain an MLP in a less greedy way by utilizing the two-layer contractive encodings, however, with a cost...
متن کاملTwo-layer contractive encodings for learning stable nonlinear features
Unsupervised learning of feature hierarchies is often a good strategy to initialize deep architectures for supervised learning. Most existing deep learning methods build these feature hierarchies layer by layer in a greedy fashion using either auto-encoders or restricted Boltzmann machines. Both yield encoders which compute linear projections of input followed by a smooth thresholding function....
متن کاملLearning Two-Layer Contractive Encodings
Unsupervised learning of feature hierarchies is often a good initialization for supervised training of deep architectures. In existing deep learning methods, these feature hierarchies are built layer by layer in a greedy fashion using auto-encoders or restricted Boltzmann machines. Both yield encoders, which compute linear projections followed by a smooth thresholding function. In this work, we...
متن کاملEstimation of tangent planes for neighborhood graph correction
Local algorithms for non-linear dimensionality reduction [1], [2], [3], [4], [5] and semi-supervised learning algorithms [6], [7] use spectral decomposition based on a nearest neighborhood graph. In the presence of shortcuts (union of two points whose distance measure along the submanifold is actually large), the resulting embbeding will be unsatisfactory. This paper proposes an algorithm to co...
متن کاملQuery-focused Multi-Document Summarization: Combining a Topic Model with Graph-based Semi-supervised Learning
Graph-based learning algorithms have been shown to be an effective approach for query-focused multi-document summarization (MDS). In this paper, we extend the standard graph ranking algorithm by proposing a two-layer (i.e. sentence layer and topic layer) graph-based semi-supervised learning approach based on topic modeling techniques. Experimental results on TAC datasets show that by considerin...
متن کامل