Guided Clustering for Social Media Nowcasting

The last several years have seen a growth in social media “nowcasting” applications—the use of social media data to predict real-world phenomena such as flu activity [5], unemployment behavior [1], and more. Generally these projects pick a target phenomenon that has some sort of official data (e.g. U.S. weekly UI claims), train a prediction model with this data and some features derived from a social media corpus, and finally generate a prediction with that model. Unfortunately, there is a large class of phenomena that lack the required training data to build an accurate prediction model. For instance, the economists we worked with in 2014 [1] shared an interest the U.S. Census Bureau has with indicating human migration patterns via social media— something that is traditionally only collected every ten years. It is arguable that most real-world phenomena lack the training data needed to build a supervised prediction model, as traditional survey-driven data collection methods are very expensive and thus cover very few phenomena. One important challenge of these projects is with how to pick the features derived from the social media corpus. For example, if the features are signals representing the weekly frequency of different phrases, “I lost my job”might be a good indicator for unemployment behavior. Past projects have either used data-intensive (i.e. correlation with the target) or labor-intensive (i.e. hand-filtering) processes for selecting these features. For our phenomena that lack sufficient training data, a data-intensive process is not feasible—since there is no data to test correlations against—and a labor-intensive process is still not great—for the same reasons outlined in previous work [2] (time-consuming, prone to error, etc.). Thus, choosing features for these low data phenomena is a difficult and important challenge. One possible solution is based on clustering. The features could be clustered together based on relatedness with each other. A user would then choose a cluster that is both interesting and high-quality. An interesting cluster would be one that matches what the user believes is descriptive of the target phenomena. A high-quality cluster is one where all the elements are related with each other and which appear