Computing MLE Bias Empirically

This note studies the bias arises from the MLE estimate of the rate parameter and the mean parameter of an exponential distribution. 1 Motivation Although maximum likelihood estimation (MLE) methods provide estimates that are useful, the estimates themselves are not guaranteed to be unbiased. Nevertheless, MLE methods are still highly regarded in practice due to several of their properties, notably, the estimates are consistent and asymptotically normal (Casella and Berger, 2002; Panchenko, 2006). The most popular example that illustrates the bias of the MLE methods is the MLE estimate of the variance parameter σ of a normal distribution N(μ, σ), we refer the readers to Liang (2012) for details. Another example that is of interest is that of an exponential distribution. In this case, the MLE estimate of the rate parameter λ of an exponential distribution Exp(λ) is biased, however, the MLE estimate for the mean parameter μ = 1/λ is unbiased. Thus, the exponential distribution makes a good case study for understanding the MLE bias. In this note, we attempt to quantify the bias of the MLE estimates empirically through simulations. For this purpose, we will use the exponential distribution as example. 2 MLE for Exponential Distribution In this section, we provide a brief derivation of the MLE estimate of the rate parameter λ and the mean parameter μ of an exponential distribution. We note that MLE estimates are values that maximise the likelihood (probability density function) or loglikelihood of the observed data. Let {xi} be i.i.d. random variables that are exponentially distributed, written as xi ∼ Exp(λ) . (1) The likelihood function associated with X = {xi} can be written as