We consider the problem of estimating a sparse linear regression vector β∗ under a gaussian noise model, for the purpose of both prediction and model selection. We assume that prior knowledge is available on the sparsity pattern, namely the set of variables is partitioned into prescribed groups, only few of which are relevant in the estimation process. This group sparsity assumption suggests us...

We show that the two-stage adaptive Lasso procedure (Zou, 2006) is consistent for high-dimensional model selection in linear and Gaussian graphical models. Our conditions for consistency cover more general situations than those accomplished in previous work: we prove that restricted eigenvalue conditions (Bickel et al., 2008) are also sufficient for sparse structure estimation.

Background: Two main issues that challenge model building are number of Events Per Variable and multicollinearity among exploratory variables. Our aim is to review statistical methods that tackle these issues with emphasize on penalized Lasso regression model.  The present study aimed to explain problems of traditional regressions due to small sample size and m...

High-dimensional feature selection arises in many areas of modern sciences. For example, in genomic research we want to find the genes that can be used to separate tissues of different classes (eg. cancer and normal) from tens of thousands of genes that are active (expressed) in certain tissue cells. To this end, we wish to fit regression and classification models with a large number of feature...

SVRG and its variants are among the state of art optimization algorithms for the large scale machine learning problem. It is well known that SVRG converges linearly when the objective function is strongly convex. However this setup does not include several important formulations such as Lasso, group Lasso, logistic regression, among others. In this paper, we prove that, for a class of statistic...

We develop a PAC-Bayesian bound for the convergence rate of a Bayesian variant of Multiple Kernel Learning (MKL) that is an estimation method for the sparse additive model. Standard analyses for MKL require a strong condition on the design analogous to the restricted eigenvalue condition for the analysis of Lasso and Dantzig selector. In this paper, we apply PAC-Bayesian technique to show that ...

The scope of this work is the constraint-based synthesis of termination arguments for the restricted class of programs called linear lasso programs. A termination argument consists of a ranking function as well as a set of supporting invariants. We extend existing methods in several ways. First, we use Motzkin’s Transposition Theorem instead of Farkas’ Lemma. This allows us to consider linear l...

In this paper, we are concerned with regularized regression problems where the prior regularizer is a proper lower semicontinuous and convex function which is also partly smooth relative to a Riemannian submanifold. This encompasses as special cases several known penalties such as the Lasso (`-norm), the group Lasso (`−`-norm), the `∞-norm, and the nuclear norm. This also includes so-called ana...

High-dimensional sparse linear regression is a basic problem in machine learning and statistics. Consider a linear model y = Xθ + w, where y ∈ R is the vector of observations, X ∈ R is the covariate matrix with ith row representing the covariates for the ith observation, and w ∈ R is an unknown noise vector. In many applications, the linear regression model is high-dimensional in nature, meanin...