The paper examines relationships between the conditional Shannon entropy and the expectation of `α-norm for joint probability distributions. More precisely, we investigate the tight bounds of the expectation of `α-norm with a fixed conditional Shannon entropy, and vice versa. As applications of the results, we derive the tight bounds between the conditional Shannon entropy and several informati...

We propose a novel information theoretic approach for semi-supervised learning of conditional random fields that defines a training objective to combine the conditional likelihood on labeled data and the mutual information on unlabeled data. In contrast to previous minimum conditional entropy semi-supervised discriminative learning methods, our approach is grounded on a more solid foundation, t...

Abstract: Based on the fast kernel entropy optimization independent component analysis and the minimum conditional entropy, this paper proposes a harmonic source localization method which aims at accurately estimating harmonic currents and identifying harmonic sources. The injected harmonic currents are estimated by the fast kernel entropy optimization independent component analysis (FKEO-ICA) ...

The entanglement of formation as well as the conditional entropy can be used to define leaves in the state space, given by the linear superposition of their extremal points. Examples, where these leaves can be found and can be used to calculate the entanglement respectively the conditional entropy are presented. The definition of entanglement is generalized to infinite systems and allows again ...

The minimum error entropy (MEE) criterion has been successfully used in fields such as parameter estimation, system identification and the supervised machine learning. There is in general no explicit expression for the optimal MEE estimate unless some constraints on the conditional distribution are imposed. A recent paper has proved that if the conditional density is conditionally symmetric and...

In this paper, we show that in order to obtain the Tsallis entropy rate for stochastic processes, we can use the limit of conditional entropy, as it was done for the case of Shannon and Renyi entropy rates. Using that we can obtain Tsallis entropy rate for stationary Gaussian processes. Finally, we derive the relation between Renyi, Shannon and Tsallis entropy rates for stationary Gaussian proc...

3 Basics of information theory 3 3.1 Entropy and uncertainty . . . . . . . . . . . . . . . . . . . . . . . 4 3.2 Joint and conditional entropy . . . . . . . . . . . . . . . . . . . . 4 3.3 Mutual information and relative entropy . . . . . . . . . . . . . . 5 3.4 Some inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.5 Entropy of discrete distributions . . . . . . . . . . . ...

Stationarity is investigated in the context of a general multiple-input–multipleoutput (MIMO) fading channel with memory. It is shown that stationary processes behave nicely as one does expect. Concretely, this reports concentrates on three topics with regard to stationarity: firstly it is proven that—under weak conditions on the channel—any stationary channel model will have a capacity-achievi...

The pure quantum entanglement is generalized to the case of mixed compound states on an operator algebra to include the classical and quantum encodings as particular cases. The true quantum entanglements are characterized by quantum couplings which are described as transpose-CP, but not Completely Positive (CP), trace-normalized linear positive maps of the algebra. The entangled (total) informa...