Relations between Shannon entropy and Rényi entropies of integer order are discussed. For any N–point discrete probability distribution for which the Rényi entropies of order two and three are known, we provide an lower and an upper bound for the Shannon entropy. The average of both bounds provide an explicit extrapolation for this quantity. These results imply relations between the von Neumann...

Most assertions involving Shannon entropy have their Kolmogorov complexity counterparts. A general theorem of Romashchenko [4] states that every information inequality that is valid in Shannon’s theory is also valid in Kolmogorov’s theory, and vice verse. In this paper we prove that this is no longer true for ∀∃-assertions, exhibiting the first example where the formal analogy between Shannon e...

In these notes we discuss Shannon’s noiseless coding theorem, which is one of the founding results of the field of information theory. Roughly speaking, we want to answer such questions as how much information is contained in some piece of data? One way to approach this question is to say that the data contains n bits of information (in average) if it can be coded by a binary sequence of length...

Quantifying the difference between two distributions is a common problem in many machine learning and data mining tasks. What is also common in many tasks is that we only have empirical data. That is, we do not know the true distributions nor their form, and hence, before we can measure their divergence we first need to assume a distribution or perform estimation. For exploratory purposes this ...

The paper deals with the generalization of both Boltzmann entropy and distribution in the light of most-probable interpretation of statistical equilibrium. The statistical analysis of the generalized entropy and distribution leads to some new interesting results of significant physical importance.

Binary Decision Diagrams (BDDs) are a well-known tool for representing Boolean functions. We show how BDDs can be extended to full rst-order logic by integrating means for representing quantiiers. The resulting structures are called Shannon graphs. A calculus based on these Shannon graphs is set up, and its soundness and completeness proofs are outlined. A comparison of deduction with rst-order...

This paper deals with approximations by the interpolating generalized Shannon sampling series.

This paper suggests an effective method for facial recognition using fuzzy theory and Shannon entropy. Combination of fuzzy theory and Shannon entropy eliminates the complication of other methods. Shannon entropy calculates the ratio of an element between faces, and fuzzy theory calculates the membership of the entropy with 1. More details will be mentioned in Section 3. The learning performanc...

The Tsallis entropy given for a positive parameter α can be considered as a generalization of the classical Shannon entropy. For the latter, corresponding to α = 1, there exist many axiomatic characterizations. One of them based on the well-known Khinchin-Shannon axioms has been simplified several times and adapted to Tsallis entropy, where the axiom of (generalized) Shannon additivity is playi...