Distributed Analytics over Web Archives

Evolving content of the Web is being accumulated over time into Web archival collections. This creates the need for time travel search to explore the dynamics of the content. Text analytics has also a key role in exploring interesting information in text collections. Moreover, frequent phrase mining, a special case of text analytics, is an important analytical task that is motivated by the need of knowledge on frequent phrases in various areas of computer science, such as information retrieval and machine translation etc. However, time travel search and frequent phrase mining have to be conducted on increasingly large-scale data. Distributed approaches such as MapReduce, which is mainly designed to work on vast amount of text, can be utilized in this case. We address two separate problems in this thesis. The first problem is that time travel inverted index, which enables searching on the time dimension, is proposed in centralized setting. In our work, we parallelize the construction of time travel inverted index using MapReduce, having a distributed index as an end product. The second problem is that finding frequent phrases through näıve counting, even in MapReduce, is a time consuming task, because data to be processed gets much larger in size, when phrases are considered. As our work, we present partitioned approximate phrase counting, a very fast way to retrieve most of the frequent phrases together with their counts out of a collection to enable interactive analysis of the content. Included in this, we propose and a novel technique, partitioned in-mapper combining, which enables us to aggregate data in memory correctly, even though the data to be aggregated is larger than the available memory. Evaluation of experiments on New York Times Annotated Corpus, which contains roughly 2 million documents, show that our approach works at least 2 times faster as compared to näıve approach. It finds more than 90% of frequent phrases with high precision. Moreover, it is able to find all highly frequent phrases exactly, along with their accurate counts. Furthermore, by a quick second pass on the data, we precisely provide most of the frequent phrases with their corresponding true counts, still being significantly faster than näıve approach. Figure 1: Tag cloud of phrases in New York Times Annotated Corpus (1987 2007)