Regularizers versus Losses for Nonlinear Dimensionality Reduction: A Factored View with New Convex Relaxations
نویسندگان
چکیده
We demonstrate that almost all nonparametric dimensionality reduction methods can be expressed by a simple procedure: regularized loss minimization plus singular value truncation. By distinguishing the role of the loss and regularizer in such a process, we recover a factored perspective that reveals some gaps in the current literature. Beyond identifying a useful new loss for manifold unfolding, a key contribution is to derive new convex regularizers that combine distance maximization with rank reduction. These regularizers can be applied to any loss.
منابع مشابه
Regularizers versus Losses for Nonlinear Dimensionality Reduction
We demonstrate that almost all nonparametric dimensionality reduction methods can be expressed by a simple procedure: regularized loss minimization plus singular value truncation. By distinguishing the role of the loss and regularizer in such a process, we recover a factored perspective that reveals some gaps in the current literature. Beyond identifying a useful new loss for manifold unfolding...
متن کاملConvex Relaxation of Vectorial Problems with Coupled Regularization
We propose convex relaxations for nonconvex energies on vector-valued functions which are tractable yet as tight as possible. In contrast to existing relaxations, we can handle the combination of nonconvex data terms with coupled regularizers such as l-regularizers. The key idea is to consider a collection of hypersurfaces with a relaxation that takes into account the entire functional rather t...
متن کاملCombinatorial Penalties: Which structures are preserved by convex relaxations?
We consider the homogeneous and the non-homogeneous convex relaxations for combinatorial penalty functions defined on support sets. Our study identifies key differences in the tightness of the resulting relaxations through the notion of the lower combinatorial envelope of a set-function along with new necessary conditions for support identification. We then propose a general adaptive estimator ...
متن کاملRobust Discriminative Clustering with Sparse Regularizers
Clustering high-dimensional data often requires some form of dimensionality reduction, where clustered variables are separated from “noise-looking” variables. We cast this problem as finding a low-dimensional projection of the data which is well-clustered. This yields a one-dimensional projection in the simplest situation with two clusters, and extends naturally to a multi-label scenario for mo...
متن کاملCombinatorial Penalties: Structure preserved by convex relaxations
In this paper, we study convex relaxations of combinatorial penalty functions. Specifically, we consider models penalized by the sum of an `p-norm and a set function defined over the support of the unknown parameter vector, which encodes prior knowledge on supports. We consider both homogeneous and non-homogeneous convex relaxations, and highlight the difference in the tightness of each relaxat...
متن کامل