Profiling Topics on the Web for Knowledge

This paper has been written with the aim of presenting motivations for my dissertation research, based on gaps in current research in text/web mining, as well as to provide an outline of the research I propose to do for my Ph.D. dissertation. The overall goal is to explore methods for knowledge discovery from the web. Text mining also known as Text Data Mining (TDM) [27], Knowledge Discovery in Textual Databases (KDT) [18] and Literature-based Discovery (LBD) [60] can be described as the process of identifying novel ideas from a collection of texts (also known as a corpus). By novel we mean information that is not explicitly present in the text source being analyzed. The kinds of ideas of interest are those indicating associations, hypotheses, trends, etc. This view of text mining is consistent with the definition proposed by Hearst in her highly cited paper [27]. To illustrate, consider the research of Swanson [59] with Raynauds Disease and Fish Oils. Swanson was interested in Raynauds Disease and read a number of research papers on the subject. He observed that Raynauds was exacerbated by certain factors such as platelet aggregability, vasoconstriction, and blood viscosity. From independent literature he also observed that these factors were mitigated by fish oils. Putting the two together he postulated that fish oils may be beneficial for Raynauds. This association was unknown at the time and was later confirmed by bioscientists. In our research we agree with Hearst's view that novelty with respect to the text collection is a requirement in text mining. However, like many others [66, 32] we adopt a more flexible definition of what constitutes " novelty ". Specifically, we see a subjective dimension in what is or is not perceived to be novel. Although not necessary, text mining efforts tend to adopt a multi-document perspective, with novel associations inferred by combining evidence from more than one document. Given the large amount of information available in text form today, we believe that tools that automatically find interesting relationships, hypotheses or ideas, or assist the user in finding these are extremely useful. Interestingly, most of the existing research in text mining has been limited to the context of biomedicine, part of which can be attributed to the early efforts of Swanson and Our focus in this thesis is on Web Mining, which can be thought of as an extension of text mining. As with text mining, …