Computationally efficient sparse clustering

Abstract We study statistical and computational limits of clustering when the means centres are sparse their dimension is possibly much larger than sample size. Our theoretical analysis focuses on model $X_i=z_i \theta +\varepsilon _{i}, \ z_i \in \{-1,1\}, \varepsilon _i \thicksim \mathcal{N}(0, I)$, which has two clusters with $\theta $ $-\theta $. provide a finite new algorithm based Principal Component Analysis (PCA) show that it achieves minimax optimal misclustering rate in regime $\|\theta \| \rightarrow \infty results require sparsity to grow slower square root Using recent framework for lower bounds—the low-degree likelihood ratio—we give evidence this condition necessary any polynomial-time succeed below Baik-Ben Arous-Péché (BBP) threshold. This complements existing reductions query bounds. Compared these results, we cover wider set parameter regimes more precise understanding runtime required error achievable. imply large class tests polynomials fail solve even weak testing task.