Consistency Analysis of an Empirical Minimum Error Entropy Algorithm

In this paper we study the consistency of an empirical minimum error entropy (MEE) algorithm in a regression setting. We introduce two types of consistency. The error entropy consistency, which requires the error entropy of the learned function to approximate the minimum error entropy, is shown to be always true if the bandwidth parameter tends to 0 at an appropriate rate. The regression consistency, which requires the learned function to approximate the regression function, however, is a complicated issue. We prove that the error entropy consistency implies the regression consistency for homoskedastic models where the noise is independent of the input variable. But for heteroskedastic models, a counterexample is used to show that the two types of consistency do not coincide. A surprising result is that the regression consistency is always true, provided that the bandwidth parameter tends to infinity at an appropriate rate. Regression consistency of two classes of special models is shown to hold with fixed bandwidth parameter, which further illustrates the complexity of regression consistency of MEE. Fourier transform plays crucial roles in our analysis. ∗ The work described in this paper is supported by National Natural Science Foundation of China under Grants (No. 11201079, 11201348, and 11101403) and by a grant from the Research Grants Council of Hong Kong [Project No. CityU 104012]. Jun Fan ([email protected]) is with the Department of Statistics, University of Wisconsin-Madison, 1300 University Avenue Madison, WI 53706 USA. Ting Hu ([email protected]) is with School of Mathematics and Statistics, Wuhan University, Wuhan 430072, China. Qiang Wu ([email protected]) is with Department of Mathematical Sciences, Middle Tennessee State University, Murfreesboro, TN 37132, USA. Ding-Xuan Zhou ([email protected]) are with Department of Mathematics, City University of Hong Kong, Kowloon, Hong Kong, China. Corresponding author. Tel: +1 615 898 2053; Fax: +1 615 898 5422; Email: [email protected]