Entropy Computations via Analytic Depoissonization

Wojciech Szpankowskit Department of Computer Science Purdue University W. Lafayette, IN 47907 UB.A. [email protected] We investigate the basic question of information theory, namely, evaluation of Shannon entropy, and a more general Renyi entropy, for some discrete distributions (e.g., binomial, negative binomial, etc.). We aim at establishing analytic methods (Le., those in which complex analysis plays a pivotal role) for such computations which often yield estimates of unparalleled precision. The main analytic tool used here is that of analytic poissonization and depoissonization. We illustrate our approach on the entropy evaluation of the binomial distribution, that is, we prove that [or Binomial(n, p) distribution the entropy hn becomes h n ::=::: ~ In n+ ~+ln J21rp(1 p)+ L:k>l akn-k where ak are explicitly computable constants. Moreover, we shall argue that analytic methods (e.g., complex asymptotics such as Rice's method and singularity analysis, Mellin transforms, poissonization and depoissoni7.ation) can offer new tools for information theory, especially for studying second-order asymptotics (e.g., for code redundancy). In fact, in recent years there has been a resurgence of interest and a few successful applications of analytic methods to a variety of problems of information theory, therefore, we we coin the term analytic information theory for such investigations.