A Comprehensive Comparison of Shannon Entropy and Smooth Renyi Entropy

We provide a new result that links two crucial entropy notions: Shannon Entropy H1 and collision entropy H2. Our formula gives the worst possible amount of collision entropy in a probability distribution, when its Shannon Entropy is fixed. Our results and techniques used in the proof immediately imply many quantitatively tight separations between Shannon and smooth Renyi entropy, which were previously known as qualitative statements or one-sided bounds. In particular, we precisely calculate the number of bits that can be extracted from a Shannon Entropy source, and calculate how far from the uniform distribution is a distribution with the given amount of Shannon Entropy. To illustrate our results we provide clear numerical examples. In the typical situation, when the gap between Shannon Entropy of a distribution and its length is bigger than 1, the length of the extracted sequence is very small, even if we allow the randomness quality to be poor. In the case of almost full entropy, where the gap is close to 0, the `2-distance to uniform is roughly of the same order as the gap. Therefore, it is actually not possible to decide the strong quality of supposed true randomness, efficiently and at extremely high confidence level, by means of Shannon Entropy estimators, like Maurer’s Universal Test or others. Our approach involves convex optimization techniques, applied to characterize worst case distributions, and the use of the Lambert W function, by which we resolve equations coming from Shannon Entropy constraints. We believe that it may be useful and of independent interests elsewhere, particularly for studying Shannon Entropy with constraints.