Statistical Estimation of the Kullback–Leibler Divergence

Asymptotic unbiasedness and L2-consistency are established, under mild conditions, for the estimates of Kullback–Leibler divergence between two probability measures in Rd, absolutely continuous with respect to (w.r.t.) Lebesgue measure. These based on certain k-nearest neighbor statistics pair independent identically distributed (i.i.d.) due vector samples. The novelty results is also treating mixture models. In particular, they cover mixtures nondegenerate Gaussian measures. mentioned asymptotic properties related estimators Shannon entropy cross-entropy strengthened. Some applications indicated.