WEIGHTED k-NN GRAPHS FOR RÉNYI ENTROPY ESTIMATION IN HIGH DIMENSIONS

Rényi entropy is an information-theoretic measure of randomness which is fundamental to several applications. Several estimators of Rényi entropy based on k-nearest neighbor (kNN) based distances have been proposed in literature. For d-dimensional densities f , the variance of these Rényi entropy estimators of f decay as O(M), whereM is the sample size drawn from f . On the other hand, the bias, because of the curse of dimensionality, decays as O(M). As a result the bias dominates the mean square error (MSE) in high dimensions. To address this large bias in high dimensions, we propose a weighted k-NN estimator where the weights serve to lower the bias to O(M), which then ensures convergence of the weighted estimator at the parametric rate of O(M). These weights are determined by solving a convex optimization problem. We subsequently use the weighted estimator to perform anomaly detection in wireless sensor networks.