Homework 2: Estimators of Entropy and Mutual Information
نویسنده
چکیده
The argument of the log function is called the information potential. To make comparisons fair, for different kernels, we will use the estimator based on the empirical expectation of the Parzen density estimation. This estimator is given by the average of the summation of the elements of a matrix K whose elements are evaluation of the kernel k function between pairs of points (i, j) in the sample X of size N . In other words
منابع مشابه
An Assessment of Hermite Function Based Approximations of Mutual Information Applied to Independent Component Analysis
At the heart of many ICA techniques is a nonparametric estimate of an information measure, usually via nonparametric density estimation, for example, kernel density estimation. While not as popular as kernel density estimators, orthogonal functions can be used for nonparametric density estimation (via a truncated series expansion whose coefficients are calculated from the observed data). While ...
متن کاملNon-Parametric Estimation of Mutual Information through the Entropy of the Linkage
A new, non–parametric and binless estimator for the mutual information of a d–dimensional random vector is proposed. First of all, an equation that links the mutual information to the entropy of a suitable random vector with uniformly distributed components is deduced. When d = 2 this equation reduces to the well known connection between mutual information and entropy of the copula function ass...
متن کاملEstimation of Renyi Entropy and Mutual Information Based on Generalized Nearest-Neighbor Graphs
We present simple and computationally efficient nonparametric estimators of Rényi entropy and mutual information based on an i.i.d. sample drawn from an unknown, absolutely continuous distribution over R. The estimators are calculated as the sum of p-th powers of the Euclidean lengths of the edges of the ‘generalized nearest-neighbor’ graph of the sample and the empirical copula of the sample r...
متن کاملContribution to Transfer Entropy Estimation via the k-Nearest-Neighbors Approach
This paper deals with the estimation of transfer entropy based on the k-nearest neighbors (k-NN) method. To this end, we first investigate the estimation of Shannon entropy involving a rectangular neighboring region, as suggested in already existing literature, and develop two kinds of entropy estimators. Then, applying the widely-used error cancellation approach to these entropy estimators, we...
متن کاملEstimating Mutual Information for Discrete-Continuous Mixtures
Estimating mutual information from observed samples is a basic primitive, useful in several machine learning tasks including correlation mining, information bottleneck clustering, learning a Chow-Liu tree, and conditional independence testing in (causal) graphical models. While mutual information is a well-defined quantity in general probability spaces, existing estimators can only handle two s...
متن کامل