Cluster Stability Analysis using Sub-sampling

Cluster stability research is involved with the validity of clusters generated by a clustering algorithm. In other words, it answers whether generated clusters are true clusters or due to chance. Estimating true numbers of clusters is related to this problem, since often the cluster validity is based on this estimate. In the literature, there are a number of methods available for both purposes. In most of the cases, assessing validity turns out to be determining the best parameter of clustering algorithm. The confidence estimation is addressed in relatively less number of research papers. In those, confidence is given in terms of the proportion of cases clustering together. Our motivation is making confidence estimation about the clusters itself, i.e. not specifically addressing specific cases. Here we propose three meta-methods from this perspective for cluster stability problem. To the best of the our knowledge, these methods are novel. The methods are all based on sub-sampling of the dataset. The methods are general and can be used with evaluation of clustering generated by wide range of clustering algorithms available. The first method, first makes a clustering using given clustering algorithm and cluster count. Next, it randomly samples from the labelled clusters, then it builds a supervised classifier on the selected subset, the induced classifier evaluates the non-selected portion. Random sub-sampling and evaluation steps are repeated many times, finally the overall accuracy gives the stability of the clustering. To find the best stable clustering for the given algorithm, overall steps are repeated for all possible number of clusters and best stable clustering is chosen for confidence estimation. Instead of random sub-sampling, 10-fold cross-validation is also employed. The second method is based on the subset selection of original clusters. First of all given clustering algorithm finds clusters. For each subset of these clusters, an algorithm that estimates the true number of clusters is used. The argument here is that, if initial clustering is stable, then for each subset of it we expect number of clusters estimated is the same as cardinality of selected subset. The above single step is for assessing the reliability of cluster itself. If the reliability of randomized algorithm like k-means is the concern, the overall steps are repeated for averaging. The confidence is computed as the ratio of correct estimations. It may be the case that, clustering has given large number of clusters (e.g. say 20 clusters). In this case, trying all subsets become computationalyintractable so we resort to subset sampling instead. The third method uses the idea that if a cluster is stable, further clustering the cases in the cluster will reveal one cluster. For each of the clusters, an estimator algorithm is run and expected to give that there is one cluster. The whole step is repeated many times with sub-sampling of dataset, i.e. a bootstrapping approach. Confidence is computed similar to the second method. Bootstrapping approach is employed for confidence estimation. The second and third method can also be used for selecting the best number of clusters in the sense that give highest confidence.