Technical Report for Collaborative Research Center Sfb 876 Providing Information by Resource- Constrained Data Analysis Subproject A1 Data Mining for Ubiquitous System Software a New Look at Regularization for Probabilistic Graphical Models

Probabilistic graphical models can simulate and classify high dimensional, heterogeneous data and serve as underlying formalism in the data analysis of biology, physics, computer vision, natural language processing and others. Their parameter dimension is a function of the treewidth of the data's conditional independence graph and the data domain. Even if most dependencies are ignored and a pairwise approximation is considered, the dimension remains large, e.g., it exceeds millions. Hence, the sample complexity is large, models tend to overfit the data and the learned models are hard to communicate. Classic 1-regularization approaches can not be applied without changing the underlying graphical structure, which is not an option if certain dependencies should be handled explicitly by the model. A recent advance in this area has shown, that a combination of regularization and reparametrization can lead to sparse models for spatio-temporal data. This is achieved by explicitly encoding knowledge about the spatio-temporal nature of the data into the model parameters. We discuss related approaches and how these techniques can be generalized to other data domains. Furthermore, we present a first promising experimental result. The exponential family of densities arises naturally as the maximum entropy distribution that reproduces data's empirical marginals. We consider a discrete 1 (multivariate) ran-1 Restricting ourselves to discrete models is just for notational convenience. The ideas presented here do apply to models with continuous variables as well. 2 The-symbol denotes the Cartesian product. d maps realizations x to a d-dimensional feature vector. The map φ G is fully determined by a graph G = (V, E) that encodes conditional independence between the components of X (see [8]). In the most general case, the dimension d is a multivariate polynomial in the variables state space sizes, i.e., d = C * ∈C(G) v ∈C |X v |, where C * (G) is the set of maximal cliques (fully connected subgraphs). In practice, often only the vertices and edges of G are modeled. Nevertheless, large state spaces or large graphs will still lead to millions of parameters. Classic 1-regularization [4] can not be applied directly, since the estimation of the graph and the estimation of the parameters are coupled: regu-larization based graph estimation approaches (e.g. [3]) perform maximum a posteriori parameter estimation, whereas a Laplace prior pushes small parameter values towards 0. The remaining non-zero weights represent the estimated graph G. Obviously, a subsequent estimation of model parameters with 1-regularization can …