Differential Entropy on Statistical Spaces

Differential entropy is the entropy of a continuous random variable. It is related to the shortest description length and thus similar to the entropy of a discrete random variable. A basic introduction can be found in the book of Cover and Thomas [1]. In this paper, we are interested in the concept of shortest description length. Indeed, in a recent paper [2], we have investigated the case of spaces where points are in fact localized within a certain volume, i.e. they are statistical in nature. The motivation was the existence of a minimal length in physical theories. It was possible to introduce a concept of distance using Fisher information metric on such spaces. In this paper we show that the reasoning leading to the definition of a distance is analogous to the usual introduction of differential entropy in information theory. In our case also, care must be taken of the precise meaning of minimal distance or shortest description length. In this extended abstract we only present an outline of our method. It is structured as follows. We remind first the introduction of differential entropy. Then, we show that the concept of distance introduced in [2] leads to a mutual information function analogous to the usual one. This is the main and new contribution of this paper.