A Symmetric Information Divergence Measure of the Csisz r's f -Divergence Class and Its Bounds

K e y w o r d s P a r a m e t r i c measure, Nonparametric measure, Csiszg~r's f-divergence, Information measure. 1. I N T R O D U C T I O N There are several types of information divergence measures s tudied in l i te ra ture which compare two probabi l i ty d is t r ibut ions and have appl icat ions in informat ion theory, s tat is t ics and engineering. A convenient classification to differentiate these measures is to categorize them as parametr ic , nonparamet r ic and ent ropytype measures of informat ion [1]. Pa ramet r i c measures of informat ion measure the amount of information about an unknown pa ramete r 0 supplied by the d a t a and are functions of 0. The best known measure of this t ype is F isher ' s measure of informat ion [2]. Nonparamet r ic measures give the amount of informat ion suppl ied by the da t a for d iscr iminat ing in favor of a probabi l i ty d is t r ibut ion f l against another f2, or for measuring the dis tance or affinity between f l and f2The Kullback-Leibler measure is the bes t known in this class [3]. Measures of ent ropy express the amount of informat ion contained in a dis t r ibut ion, t ha t is, the amount of uncer ta in ty associated with the outcome of an exper iment . The classical measures of this type are Shannon 's [4] and Rdnyi 's measures [5]. The construct ion of measures of informat ion divergence is not an easy task. Methods for deriving pa ramet r i c measures of informat ion from the nonparamet r ic ones and from the informat ion matr ices axe suggested in [1]. This research was partially supported by the first author's Discovery Grant from the Natural Sciences and Engineering Research Council of Canada (NSERC). 0898-1221/05/$ see front matter (~) 2005 Elsevier Ltd. All rights reserved. doi:10.1016/j.camwa.2004.07.017 Typeset by .AA4S-TEX 576 P. KUMAR AND S. CHHINA In this paper, we present a new symmetric nonparametric information divergence measure which belongs to the class of Csiszdr's f-divergences [6,7]. In Section 2, we discuss the Csisz£r's f-divergences and their properties. Information inequMities are presented in Section 3. New symmetric divergence measure and its bounds are obtained in Section 4. In Section 5, it is shown that the suggested measure can be applied to the parametric family of distributions. A numerical illustration for studying the behavior of new measure is shown in Section 6. Section 7 concludes the paper. 2. C S I S Z / k R ' S F D I V E R G E N C E S A N D P R O P E R T I E S Let ~ = {xl, x2, ...} be a set with at least two elements, fi(~), the set of all subsets of and ~, the set of all probability distributions P = (p(x) : x E f~) on ft. A pair (P, Q) e ~2 of probability distributions is called a simple versus simple testing problem. Two probability distributions P and Q are called orthogonal (P _k Q) if there exists an element A E P(~) such that P(A) = Q(A c) = 0, where A c = f~/A. A testing problem (P,Q) E ~2 is called least informative if P = Q and most informative if P _L Q. b-hrther, let F be a set of convex functions f : [0, co) , ~ ( -co , 0o)] continuous at 0, that is, f(0) = lim~10 f(u), F0 = {f E F : f(1) = 0} and let D _ f and D+f denote the left-hand side and right-hand side derivatives of f , respectively. Define f* E F , the *-conjugate (convex) function of f , by f*(u) = uf(1/u) , u 6 (0, co) and ] = f + f*. For a convex function f : [0, co) -~ •, the f-divergence of the probability distributions P and Q is defined [6-8], C / ( P , Q ) = Z q(x) f ( p ( x ) ~ (2.1) \q(x)/ x E ~ It is well known that C/(P, Q) is a versatile functional form which results in a number of popular divergence measures [9,10]. Most common choices of f satisfy f(1) = 0, so that CI(P, P) = O. Convexity ensures that divergence measure Cy(P, Q) is nonnegative. Some examples are f (u) = u l n u provides the Kullback-Leibler measure [3], f (u) = ]u 1] results in the variational distance [11,12], f (u) = (u 1) 2 yields the x2-divergence [13]. The basic general properties of f-divergences including their axiomatic properties and some important classes are given in [9]. For f , f*, f l E F, V (P, Q) E p2, u E (0, co), (i) c (p,Q) = (ii) UNIQUENESS THEOREM. (See [14].) I A (P, Q) = I / (P , Q), iff ~c E R : f l (u) f (u) = c (u 1). (iii) Let c E [D_f(1), D+f(1)]. Then, f l(u) = f(u) c(u 1) satisfies f l(u) >_ f(1), Vu E [0, co) while not changing the f-divergence. Hence, without loss of generality f l (u) >_ f(1), Vu c [0,0o). (iv) SYMMETRY THEOREM. (See [14].) I f . ( P , Q ) = I / ( P , Q ) , i f f 3 c e R : f * ( u ) f ( u ) = c ( u 1 ) . (V) RANGE OF VALUES THEOREM. (See [15].) f (1) _< I f (P, Q) _< f (0) + f* (0). In the first inequality, equality holds iff P = Q. The latter provides f is strictly convex at 1. The difference I f (P, Q) f(1) is a quantity that compares the given testing problem (P, Q) c p2 with the least informative testing problem. Given f C F, by setting f (u) := f (u) f(1), we can have f(1) -0 and hence, without loss of generality, f E F0. Thus, I f (P, Q) serves as an appropriate measure of similarity between two distributions. A Symmetric Information Divergence 577 In the second inequality, equality holds iff P A_ Q. The latter provides ](0) < f(0)+f*(0) < co. The difference Ig(P, Q) := ](0) I f ( P , Q) is a quantity that compares the given testing problem (P, Q) e F2 with the most informative testing problem. Thus, Ig(P, Q) serves as an appropriate measure of orthogonality for the two distributions where the concave function g : [0, co) ~ R is given by g(u) = f(O) + uf*(O) f(u). (vi) CHARACTERIZATION THEOREM. (See [7].) Given a mapping I : ~2 ~ (-oc, co), (a) I is an f-divergence, that is, there exists an f C IF, such that I (P, Q) = cs (P, Q), v (P, Q) e P~. (b) Cf(P, Q) is invariant under permutation of ~. (c) Let A = (A~, i _> 1) be a partition of ~, and PA = (P(Ai), i _> 1) and QA = (Q(Ai), i >_ 1) be the restrictions of the probability distributions P and Q to A. Then, I(P,Q) >_ I(PA, QA) with equality if P(Ai) × p(x) = Q(A~) x p(x), Vx c Ai, i>_1. (d) Let P1, P2 and Q1, Q2 be probability distributions on ft. Then, I (aP1 + (1 a)/)2, aQ1 + (1 a) Q2) _< aI (P1, Q1) + (1 a) I (P2, Q2)By characterization theorem, the *-conjugate of a convex function f is f* (u) uf(1/u). For brevity, in what follows now, we will denote Cf(P, Q), p(x), q(x), and ~-~en by C(P, Q), p, q, and ~ , respectively. Some popularly practised information divergence measures are as follows. X ~-Divergences. VARIATIONAL DISTANCE. (See [11,12].) V (P, Q) = ~ [p q]. x2-DIVERGENCE. (See [13].) x2(p,Q) E ( p q)2 E p 2 q q SYMMETRIC x2-DIVERGENCE. (P, Q) = x 2 (p, Q) + x 2 (Q, P) = ~ (P + q) (p q)~ Pq KULLBACK AND LEIBLER. (See [3].) KULLBACK-LEIBLER SYMMETRIC DIVERGENCE. J ( P , Q ) = K ( P , Q ) + K ( Q , P ) = E ( p q)ln ( P ) . TRIANGULAR DISCRIMINATION. (See [16,17].) Ip ql: A(p,Q) : ? ; q . (2.2)