On Discriminative Semi-Supervised Classification

The recent years have witnessed a surge of interests in semi-supervised learning methods. A common strategy for these algorithms is to require that the predicted data labels should be sufficiently smooth with respect to the intrinsic data manifold. In this paper, we argue that rather than penalizing the label smoothness, we can directly punish the discriminality of the classification function to achieve a more powerful predictor, and we derive two specific algorithms: SemiSupervised Discriminative Regularization (SSDR) and Semi-parametric Discriminative Semi-supervised Classification (SDSC). Finally many experimental results are presented to show the effectiveness of our method. Introduction In many practical pattern classification and data mining problems, the aquisition of sufficient labeled data is often expensive and/or time consuming. However, in many cases, large numbers of unlabeled data are far easier to obtain. For example, in text classification, one may have an easy access to a large database of documents (e.g. by crawling the web), but only a small part of them are classified by hand. Consequently, semi-supervised learning methods, which aim to learn from both labeled and unlabeled data points, are proposed (Chapelle et al., 2006). Many semi-supervised learning algorithms have been proposed in the last decades, for example, the generative model based methods (Shahshahani & Landgrebe, 1994; Miller & Uyar, 1997; Nigam et al, 2000), co-training (Blum & Mitchell, 1998), transductive SVM (TSVM) (Joachims, 1999), and the approaches based on statistical physics theories (Getz, et al, 2005; Wang et al, 2007). One basic assumption behind semi-supervised learning is the cluster assumption (Chapelle et al., 2006), which states that two points are likely to have the same class label if there is a path connecting them passing through the regions of high density only. Zhou et al. (Zhou et al, 2004) further explored the geometric intuition behind this assumption: (1) nearby points are likely to have the same label; (2) points on the same structure (such as a cluster or a submanifold) are Copyright c © 2008, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. The work is supported by NSFC (Grant No. 60721003, 60675009). likely to have the same label. Therefore, the learned classification function should be sufficiently smooth with to respect the intrinsic data manifold. Belkin et al. (Belkin et al., 2006) further proposed an elegant framework for semi-supervised learning, called manifold regularization, which learns a specific classification function by minimizing a cost composed of two terms, one is the structure loss and the other is a smoothness measure estimated from both labeled and unlabeled data. In such a way, the learned function would be sufficiently smooth while simultaneously hold a good generalization ability. However, as the final goal of classification is to discriminate the data points from different classes, it is reasonable that the predicted labels vary smoothly with respect to the intrinsic geometric structure within the same class, but there should definitely be some discontinuities on the decision boundaries, i.e., on the boundaries where the different classes are connected. Moreover, it would be more encouraged if the predicted labels of the data in different classes are more distinct. Based on the above considerations, in this paper, we propose a novel strategy for semi-supervised classification, which does not require the classification function to be smooth, but to be discriminative. We first construct an unsupervised discriminative kernel based on discriminant analysis (Fukunaga, 1990), and then use it to derive two specific algorithms, Semi-Supervised Discriminative Regularization (SSDR) and Semi-parametric Discriminative Semisupervised Classification (SDSC) to realize our strategy. Finally the experimental results on several benchmark data sets are presented to show the effectiveness of our methods. The rest of this paper is organized as follows. In section 2 we will derive an unsupervised discriminative kernel. The main procedure of the SSDR and SDSC algorithms will be introduced in detail in section 3 and section 4. The experimental results will be presented in section 5, followed by the conclusions in section 6. A Discriminative Kernel Given a set of data objectsX = {x1,x2, · · · ,xn}with xi ∈ R , the goal of discriminant analysis (Fukunaga, 1990) is to learn a projection matrix P ∈ R with d ¿ D, such that after projection the data set have a high within class similarity and between-class dissimilarity. Without the loss of Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (2008)