Information Theory on Rectifiable Sets

Entropy and—maybe even more so—mutual information are invaluable tools for analyzing properties of probability distributions, especially in coding theory. While there are general definitions for both concepts available for arbitrary probability distributions, these tend to be hard to work with and the literature (e.g., [CT06]) focuses on either discrete, or continuous random variables. In this thesis we extend the theory to singular probability distributions on suitably “smooth” lower-dimensional subsets of Euclidean space, where no p.d.f. or p.m.f. is readily available. We choose those prerequisites carefully in order to retain most properties from the discrete and/or continuous case. The mathematical framework, this work is built upon, is the field of geometric measure theory. In particular, we make extensive use of the material found in [Fed69]. As it is the study of geometric properties of measures, and thereby closely related to probability theory as well, geometric measure theory proves fruitful for analyzing information theoretic properties of probability measures, when geometric restrictions are imposed. We consider a random variable X on Euclidean space. The distribution (i.e. the induced measure) of X is required to be absolutely continuous with respect to the m-dimensional Hausdorff measure and to be concentrated on an m-dimensional rectifiable set E , i.e., the complement of E is a set of measure zero. Under these conditions we obtain expressions for the entropy h(X ) and develop the mutual information I (X ;Y ) for two random variables when the combined random variable (X ,Y ) satisfies similar constraints. We give integral expressions for these quantities and show how to manipulate them using results from geometric measure theory. Another central result is the proof of the relation I (X ;Y ) = h(X )+ h(Y )− h(X ,Y ) between mutual information and entropy for our newly introduced definitions. This work is intended as a theoretical starting point for further investigations. Possible applications are, e.g., the information theoretic treatment of sparse sources in source coding or of the vector interference channel in channel coding. In both examples singular distributions on "smooth" lower-dimensional subsets play a pivotal role. While this work was conducted with applications in coding theory in mind, the presented framework is inherently measure theoretic and might, therefore, be applied in other areas as well.